model-research/REPORT.md

# Model Research Report: AI Models for Analytical Work

_29 experiments across 11 days (2026-04-26 to 2026-05-06). Five models tested on architecture document analysis — not coding._

## Executive Summary

We tested GPT-5, Claude Opus, Claude Sonnet 4.6, Claude Sonnet 4.5, GPT-4.1, and GPT-4.1 Mini on analytical tasks: finding hidden assumptions, race conditions, design contradictions, adversarial attack paths, regulatory gaps, and cross-document inconsistencies in real architecture documents.

**The central finding:** Different models don't just find more or fewer things — they find *qualitatively different kinds* of things. Model choice is task-dependent, and no single model dominates all analytical work.

---

## Part 1: What Each Model Is Good At

### GPT-5
**Strength:** Exhaustive enumeration + domain-specific reasoning about the real world.

GPT-5's reasoning tokens change the *kind* of analysis, not just the depth. Non-reasoning models identify risks within a document's own frame of reference. GPT-5 reasons about the document's relationship to the external world: broker semantics, deployment topology, OTP runtime behavior under load, timing correlations across independent subsystems.

- Best at: gap-finding, hidden assumptions, adversarial enumeration, temporal boundary analysis
- Unique ability: finds multi-component interaction failures that require domain knowledge
- Weakness: slow (2-4x longer than Opus), expensive (5-10x more tokens per finding), sometimes over-verifies
- Finding count: typically 15-35 depending on document complexity

### Claude Opus
**Strength:** Design tensions, logical argumentation, creative adversarial thinking.

Opus consistently identifies where one part of a design undermines another part. It doesn't enumerate failure modes — it finds the deeper question of *why* the design's own principles conflict.

- Best at: contradiction detection, cross-document consistency, race conditions (design-level), adversarial creativity
- Unique ability: self-corrects mid-analysis, finds "your safety mechanism IS your vulnerability" patterns
- Most efficient model: 6-9x fewer tokens per finding than GPT-5 on shared task types
- Weakness: produces fewer findings on pure enumeration tasks (10-13 vs GPT-5's 20-35)

### Claude Sonnet 4.6
**Strength:** Speed, structural issues, assumption-finding. Best precision-per-dollar.

- Best at: quick first-pass screening, structural review, specification gap identification
- Zero false positives on most tasks — every finding is actionable
- Weakness: struggles with concurrency reasoning, contradiction detection, and tasks requiring formal logical reasoning
- Produces false positives on verification-heavy tasks (contradiction, race conditions)

### Claude Sonnet 4.5
**Strength:** Exhaustive coverage. More findings than 4.6, at the cost of some noise.

- Best at: specification completeness (25 findings vs 4.6's 13)
- Catches operational gaps that 4.6 filters out
- Tradeoff: severity inflation, more verbose output

### GPT-4.1
**Strength:** Structured, thorough, good middle ground. Generic but competent.

- Stays within the document's own framing — finds assumptions the document *almost* states
- Best unique contribution: meta-observations about design structure (e.g., "all failure modes treated as isolated")
- Good enough for first-pass review where GPT-5's cost isn't justified

### GPT-4.1 Mini
**Strength:** Cheapest. Formulaic but catches the obvious things.

- Every finding maps cleanly to a section of the document
- Fine for quick sanity checks, not for architectural insight
- Scales with document size (6 findings on 459 lines → 21 on 1,110 lines)

---

## Part 2: What We Learned About Task Types

Not all analytical tasks are the same. Models that excel at one struggle at another.

| Task Type | Best Model | Runner-up | Avoid |
|-----------|-----------|-----------|-------|
| Hidden assumptions | GPT-5 | Opus | Mini (formulaic) |
| Gap-finding | GPT-5 | GPT-4.1 | Mini (surface-level) |
| Race conditions | GPT-5 + Opus | — | Sonnet (errors) |
| Contradiction detection | **Opus** | GPT-5 | Sonnet (false positives) |
| Cross-document consistency | **Opus** | GPT-5 | — |
| Adversarial attack paths | GPT-5 (enumeration) + Opus (creativity) | — | — |
| Bias detection | Any model | — | — |
| Design coherence | Document-dependent | — | — |
| Specification completeness | Sonnet 4.5 (breadth) or GPT-5 (self-contradictions) | — | — |
| Missing feature identification | All (with right prompt) | — | — |
| Invariant violation paths | GPT-5 (precision) | Opus (breadth) | Sonnet (imprecise) |

**Key pattern:** Tasks requiring *identification* (what's missing? what's assumed?) are accessible to all models. Tasks requiring *verification* (is this sequence legal? does this contradict that?) favor reasoning models (GPT-5, Opus) and exclude Sonnet.

---

## Part 3: Meta-Findings About How to Use Models

### 1. Signal-to-noise ratio matters more than model capability (Finding #8)

When biased text is the ONLY input, even GPT-4.1 Mini catches it. When the same bias is buried inside a full PR review with diffs, issues, and project context, expensive models miss it. The issue isn't model intelligence — it's attention dilution.

**Implication:** For important analytical checks, isolate the signal. Extract the relevant text and ask about it specifically. Don't bury important questions inside broad review mandates.

### 2. Prompt framing dominates model personality (Finding #26)

Opus's "finds design tensions" and GPT-5's "exhaustive enumeration" are DEFAULT tendencies, not capabilities. With structured prompts that explicitly ask for breadth, Opus produces MORE findings than GPT-5. With structured prompts asking for contradictions, GPT-5 becomes highly selective.

**Implication:** Model choice matters less than you think for any single task. Prompt structure is the primary lever. Model personality matters for OPEN-ENDED tasks where you want emergent analytical behavior.

### 3. Task type predicts model performance better than "model X is better" (Finding #13)

Sonnet scores 85% of GPT-5's performance on assumption-finding but drops to ~50% on concurrency reasoning. Don't extrapolate across task types.

### 4. The union of models finds the most (Finding #19)

GPT-5 Mini + Sonnet covers ~71% of GPT-5's findings at 31% of the cost. But the missing 29% contains the domain-specific interaction-level findings most likely to prevent production incidents. Each model also finds things the others miss — the total unique finding space is larger than any single model's output.

### 5. Reasoning tokens change the KIND of analysis, not just the amount (Finding #10)

Non-reasoning models ask "what could this mechanism fail at?" Reasoning models ask "what must be true about the world for this mechanism to work?" This is a qualitative difference in analytical mode, not just thoroughness.

### 6. Reasoning effort parameter is a no-op for analytical work (Finding #21)

Low/medium/high reasoning effort had negligible effect on GPT-5's output for open-ended analysis. Task type is a far stronger predictor of reasoning behavior. Don't waste time tuning this parameter for document review.

### 7. Output length kills, input length doesn't (Finding #6)

Single agents die trying to generate 1000+ line documents. Rich input context is fine — it's the output length that causes OOM/timeout. Break output into sections, keep input context rich, draft in parallel, assemble.

### 8. Document complexity shifts model rankings (Finding #27)

Opus beat GPT-5 on coherence analysis for one document but lost on another (more complex) document. Rankings are not universal — they interact with document complexity, domain specificity, and prompt structure.

### 9. Token budget matters more than model size (Finding #7b)

When output is truncated by token limits, even GPT-5 produces shallow findings. Ensure sufficient max_completion_tokens (≥16K for GPT-5). A cheap model with enough tokens beats an expensive model that runs out of space.

---

## Part 4: Cost-Effectiveness

| Model | Typical tokens/finding | Relative cost | Best use case |
|-------|----------------------|---------------|---------------|
| Opus | 179-336 | 1x (baseline) | Cross-doc consistency, contradictions, design tensions |
| Sonnet 4.6 | 194-312 | 0.3x | Quick screening, structural review, assumption-finding |
| GPT-5 | 993-2,967 | 5-9x | High-stakes analysis where missing something has real cost |
| GPT-4.1 | ~500 | 0.5x | Middle-ground first pass |
| GPT-4.1 Mini | ~300 | 0.1x | Bulk screening, sanity checks |

**For financial/safety-critical systems:** Run all three (Opus + GPT-5 + Sonnet). The ~$1 total cost per document is irrelevant vs the value of comprehensive coverage.

**For routine review:** Opus alone or Sonnet + Opus pair. Skip GPT-5 unless the document is complex and the stakes justify it.

---

## Part 5: What's Still Unknown

1. **Would running models sequentially (feed Model A's output to Model B) outperform parallel runs?** Hypothesized for adversarial analysis but untested.
2. **Are these findings corpus-specific?** All 29 experiments used gargoyle architecture docs. Different domains may shift rankings.
3. **How much do results vary across runs?** All findings are single-run. Stochastic variation is unquantified.
4. **Does Sonnet's narrow-framing weakness go away with explicit concurrency prompts?** Untested — the hypothesis that Sonnet's "structural reviewer" tendency is a framing artifact.
5. **What happens on 2000+ line documents?** Largest tested is 1,110 lines. Unknown if model rankings shift at extreme scale.