Security-review-bot persona caught HTTPS enforcement inconsistency in write-path
methods (PostReview, DeleteReview, RequestReviewer) that generalist reviewers missed.
Issue fixed within 30 minutes, all reviewers re-approved. Validates specialized
security persona value in multi-model pipeline.
New analytical lens: security boundary analysis — identifying where trust
assumptions cross component boundaries in exploitable ways.
Document: gargoyle system-overview.md (323 lines)
Models: Claude Opus (15 findings), Claude Sonnet 4 (10 findings)
Key finding: Opus identified that transient signals (a performance design
choice) create a structural security vulnerability — malicious strategies
can probe risk limits without leaving any audit trail.
This experiment establishes security boundary analysis as a distinct,
viable analytical task type for architecture review.
New analytical lens tested on gargoyle order-state-machine.md:
- GPT-5: 15 findings (most CRITICAL issues, exhaustive field analysis)
- Opus: 14 findings (state lifecycle focus, implementation mechanisms)
- Sonnet: 10 findings (fast but shallow)
Key insight: "unstated constraints" finds what's IMPLIED but not stated,
distinct from gaps, race conditions, or ambiguities. GPT-5 is best for
catching CRITICAL data integrity constraints; Opus for state machine
implementation details.
New analytical lens: where data propagation creates stale, contradictory,
or misleading views for different consumers.
Key result: highest model convergence (45% common ground) due to document's
explicit failure mode table. GPT-5 finds event-level provenance gaps; Opus
identifies strategy attribution dimension. Sonnet adds zero unique value.
Two-model stack (GPT-5 + Opus) optimal.
New analytical lens: observability gap analysis — asking 'when something
goes wrong, can you SEE it?' rather than 'what can go wrong?'
Results on aggregation.md (239 lines):
- GPT-5: 23 findings (12 unique), exhaustive telemetry architecture
- Opus: 14 findings (6 unique), operator-behavioral insights
- Sonnet: 11 findings (0 unique), no added value
Key insight: GPT-5 designs the instrumentation; Opus identifies where
available signals mislead operators toward wrong remediations.
Two-model (GPT-5 + Opus) optimal for this task type.
Tested GPT-5, Opus, Sonnet on wash-sale-tracking.md spec.
Opus found a genuine spec bug (trigger logic described backwards).
Confirms pattern: GPT-5 for breadth, Opus for logic contradictions,
Sonnet adds no value for systematic analytical tasks.
New task type: specification gap/completeness analysis (vs adversarial gaming).
GPT-5 dominates count (25 findings), Opus produces best single insight
(realized P&L non-reversibility violates de-escalation model assumption).
Sonnet adds no unique value for this task type — skip for completeness audits.
REPORT.md — full analysis of 29 experiments: model strengths, task-type
mappings, meta-findings, cost-effectiveness, and open questions.
LESSONS.md — distilled operational playbook: which model for which task,
anti-patterns, decision framework, and the three core rules.
Break the monolithic 3249-line findings file into 29 individual files,
one per experiment. Each file is named YYYY-MM-DD-NN-slug.md for easy
chronological sorting and discovery.
No content changes — purely structural reorganization.
Full comparative analysis of GPT-5, Claude Opus 4.6, Claude Sonnet 4.6,
GPT-4.1, and GPT-4.1 Mini on analytical tasks (not coding).
Contents:
- findings/ALL-FINDINGS.md — complete 3,249-line research log with all
29 findings, methodology notes, and open questions
- prompts/ — 6 exact prompts used across experiments
- methodology.md — experimental setup and evaluation criteria
- open-questions.md — unanswered questions for future work
- README.md — overview and summary table
Key findings:
- Cross-document consistency: Opus is 2.4x faster with more findings
- Gap-finding: GPT-5 reasoning tokens find domain-specific gaps
- Race conditions: Opus excels at temporal interaction reasoning
- Bias detection: Signal-to-noise ratio > model capability
- Adversarial analysis: GPT-5 exhaustive, Opus qualitatively different
Signed-off-by: Rodin
Rodin