model-research/findings/2026-05-03-16-specification-completeness-sonnet-45-produces.md

Finding 16: Specification completeness: Sonnet 4.5 produces 2x the findings of Sonnet 4.6 on implementation-gap analysis; quality vs volume tradeoff

Date: 2026-05-03 Task: Identify specification gaps in gargoyle's kill-switch.md (185 lines) — places where an implementer would be forced to guess or decide on their own because the spec doesn't clearly specify behavior. New analytical lens not previously tested. How we used them: Same document (full text) + same focused analytical question to all 3 models via HAI proxy. Highly structured prompt specifying 5 categories of underspecification (behavioral ambiguity, missing edge cases, ordering/sequencing gaps, interface contracts undefined, concurrency semantics omitted). Required specific output format per finding (gap, section, what implementer must decide, risk if wrong, severity). No tools, no project context beyond the document itself.

Model	Time	Output tokens	Reasoning tokens	Gaps found	Critical	High	Medium	Low
Claude Sonnet 4.6	73s	3,403	(internal)	13	8	4	0	1
Claude Sonnet 4.5	102s	5,191	(internal)	25	14	6	4	1
GPT-5	109s	10,140	7,872	19	8	7	3	0

What they found — common ground (all 3 identified):

• Pipeline process identification ambiguity (which processes are "pipeline processes")
• Per-user process scope mapping (how to terminate only one user's processes)
• ETS table ownership and lifecycle (who owns it, what happens on crash)
• Concurrent engage operations (what happens when two sources engage simultaneously)
• Liquidation order tagging mechanism (what the tag is, how verified)
• Process restart prevention (how "must not restart" is enforced)
• Engage sequence atomicity (partial failure between DB write and termination)
• Startup ordering and ETS readiness (pipeline starting before ETS populated)
• Disengage sequence ordering (what happens and in what order)

Sonnet 4.5 unique findings (not in either other model):

• ETS table schema/structure (set vs ordered_set, key format, value schema)
• Missing ETS detection mechanism (catch :badarg vs table existence check)
• Database write atomicity with ETS (transaction boundaries, rollback semantics)
• Per-user engage while global is already engaged (is it a no-op or error?)
• Broker rejection semantics ("already filled" vs "invalid cancel" distinction)
• Cold-start gate interaction (independence vs dependency of the two gates)
• User deletion with active kill switch (orphaned rows, cascade semantics)
• Global disengage effect on per-user states (independent or auto-clear?)
• Audit log write failure during engage (critical-path vs best-effort)
• Dashboard control ambiguity in LIQUIDATE mode (contradictory disable/enable)
• Cancel timeout duration (operational parameter not specified)
• Manual order source code path during LIQUIDATE (how orders bypass the dead pipeline)

GPT-5 unique findings (not in either other model):

• Combined global/per-user mode semantics (what happens when global=RESTRICT, user=LIQUIDATE — can user's liquidation proceed?)
• Scope of "all" in cancel_all and liquidation (system-wide vs per-user)
• Gate behavior when ETS missing but liquidation needed (conflicting requirements: fail-closed says block, but liquidation needs to pass)
• Disengage during in-flight cancellations (what happens to racing tasks)
• Gate placement relative to broker submission (exact point in the flow)
• Engage latency expectations (no quantified SLA)
• Mode change while already engaged (RESTRICT → LIQUIDATE without disengage)
• Dashboard vs backend scope for manual liquidation (individual vs bulk only)

Sonnet 4.6 unique findings (not in either other model):

• ETS sequencing relative to process termination (ETS before or after kill?)
• Concurrent disengage + re-engage race (specific interleaving scenario)
• Close-only enforcement mechanism (UI-only vs backend validation)
• Order-in-flight past ETS gate during termination (already-checked orders)

Quality assessment:

• Claude Sonnet 4.5 was the most EXHAUSTIVE (25 gaps) but with notable quality variance. Several findings were highly specific and implementation- relevant (ETS schema, missing-table detection, broker rejection semantics). Others were relatively obvious or lower-impact (user deletion, audit log failure, cancel timeout duration). The 14 Critical ratings feel somewhat generous — some would be more accurately rated as High in practice. Output was well-structured with clear per-finding format.
• GPT-5 found 19 gaps with consistent high quality. Its unique findings show cross-cutting reasoning: the combined mode semantics finding (global vs per-user mode interaction) identifies a genuine specification gap that neither Sonnet version noticed. The "ETS missing but liquidation needed" finding is architecturally significant — it identifies a CONTRADICTION in the spec's own rules (fail-closed blocks everything, but liquidation must pass). Every finding was actionable. More selective severity ratings (8 Critical vs Sonnet 4.5's 14).
• Claude Sonnet 4.6 was the most SELECTIVE (13 gaps) but with the highest precision. Every finding was genuinely a specification gap that an implementer would face. The ETS sequencing finding (#4) is particularly well-reasoned — it identifies a specific ordering dependency that creates a race window. Sonnet 4.6 appears to self-filter aggressively, producing only findings it's confident about. Higher signal-to-noise than 4.5.

Key insight — Sonnet 4.5 vs 4.6 on analytical tasks: This is the first direct comparison between Claude model versions on the same analytical task. Key differences:

• Volume: 4.5 produced almost 2x the findings (25 vs 13)
• Tokens: 4.5 used ~1.5x the output tokens (5,191 vs 3,403)
• Time: 4.5 took ~1.4x longer (102s vs 73s)
• Severity distribution: 4.5 had more Critical findings (14 vs 8) but with more generous severity ratings
• Quality per finding: 4.6 had higher average quality; fewer "obvious" or lower-impact findings

The 4.6 model appears to have been trained toward higher precision/selectivity. It finds fewer things but each finding is more reliably a genuine gap. The 4.5 model is more exhaustive but includes findings that a reviewer might triage as "yes, technically, but not really a spec gap." This mirrors a known training direction in Claude models: later versions tend to be more concise and selective.

For practical use: If you want completeness (cast a wide net, accept some noise): use 4.5. If you want precision (every finding is actionable, no triage needed): use 4.6. For architecture review where missing a gap has cost, 4.5's exhaustiveness is probably worth the noise. For review where false positives cost attention (e.g., PR review comments), 4.6's selectivity is preferred.

GPT-5 vs Sonnet comparison on this task: GPT-5 (19 findings) sits between the two Sonnets in volume but has the highest consistency — no obvious misses or inflated severities. Its unique strength here: finding CONTRADICTIONS within the spec's own rules (ETS-missing blocking conflicts with liquidation needing to pass). This is consistent with Finding #15 where GPT-5 was unusually selective but precise on coherence checking.

Specification completeness analysis appears to be a task where:

1. Sonnet 4.5 is strongest for breadth (25 findings, catches operational gaps)
2. GPT-5 is strongest for detecting spec self-contradictions (19 findings, high precision)
3. Sonnet 4.6 is strongest for precision (13 findings, zero noise)

Updated model version comparison:

• Claude 4.6 → higher precision, more selective, concise
• Claude 4.5 → more exhaustive, more verbose, occasional severity inflation
• This is a genuine tradeoff, not a simple regression or improvement

Practical implication: Run BOTH Sonnet versions? 4.5 catches things 4.6 filters out (ETS schema, broker rejection semantics, cold-start gate interaction). 4.6 catches things with more specificity (sequencing gaps, exact race windows). For a one-shot budget: 4.5 if you want coverage, 4.6 if you want actionability. GPT-5 if you want to find where the spec contradicts itself.