model-research/findings/2026-05-02-12-sonnet-46-outperforms-expectations-on.md

Finding 12: Sonnet 4.6 outperforms expectations on assumption-finding; competes with reasoning models on complex docs

Date: 2026-05-02 Task: Identify hidden assumptions in gargoyle's order-execution.md (785 lines) — a complex, multi-component document covering OrderManager, BrokerAdapter, TradeStream, and PositionReconciler. How we used them: Same document (full text, no truncation) + same focused analytical question to all 3 models. GPT-5 via HAI OpenAI endpoint; Opus 4.6 and Sonnet 4.6 via HAI Anthropic endpoint. No tools, no project context beyond the document itself. Single prompt, no conversation history.

Model	Time	Output tokens	Reasoning tokens	Assumptions found
GPT-5	93s	8,485	6,016	20
Claude Sonnet 4.6	106s	4,637	(internal)	17
Claude Opus 4.6	105s	4,615	(internal)	12

What they found — common ground (all 3 identified):

• Synchronous broker REST calls blocking OrderManager GenServer (mailbox growth)
• TradeStream event ordering assumptions (out-of-order fills/status)
• Fill deduplication gap (no explicit fill-level idempotency)
• cancel_all/1 with timeout: :infinity blocking GenServer during FLATTEN
• Recovery/restart races with TradeStream fill delivery (fills queued during handle_continue/2)
• Lot operation idempotency under crash recovery (partial execution)
• Replace race: fills for new broker_order_id arriving before replaced event
• Database write latency impact on GenServer throughput under burst fills
• ETS table scope assumptions (single-node, access mode)

GPT-5 unique findings (not in either Claude model):

• Rate-limit retry blocking OrderManager inline (no async retry path specified)
• Single TradeStream connection per user not enforced (duplicate detection gap)
• Kill switch FLATTEN vs degraded state interaction (OM drops cancels while degraded, but FLATTEN calls cancel_all through OM)
• ClOrdID uniqueness scope/retention at broker across sessions and days
• after: datetime filter semantics (clock skew, timezone, inclusive/exclusive)
• Reconciliation responses may exceed single-response size (no pagination)
• Event broadcasting blocking model (synchronous vs fire-and-forget)
• Credential rotation during TradeStream connection lifetime
• market_closed semantics varying across brokers (reject vs queue)
• Dropped Alpaca statuses (stopped/suspended/calculated) may affect accounting

Claude Sonnet 4.6 unique findings (not in either other model):

• Single fill per fill event assumption (broker batching multiple fills into one WebSocket message)
• Lot operations (Lots.open/2, Lots.close/4) assumed to never fail — no {:error, _} handling shown, crash propagation risk
• Task.async_stream inside GenServer creating linked tasks whose crash signals propagate to OrderManager during critical cancel_all
• Broker cancel semantics during in-flight replace at the broker level (cancel targets old broker_order_id which broker already replaced away)
• Database operations in fill processing assumed transactional (no explicit Ecto.Multi/transaction mention)
• Broker position reflects only Gargoyle's activity (external trades cause false-positive reconciliation halts)

Claude Opus 4.6 unique findings (not in either other model):

• {:ok, broker_order_id} from REST place conflated with durable OMS acceptance vs mere HTTP acknowledgment (no timeout on submitted state)
• Concurrent apply_corrections/2 from periodic reconciler running in separate process conflicts with OrderManager's single-writer invariant (corrections write to same tables outside GenServer serialization)
• Reconciliation gate initialized state after :rest_for_one restart — ETS table EXISTS but freshly initialized vs table MISSING are different conditions with different safety properties
• Escalation state reset after crash creating double-exposure window (systematic issue persists but escalation timer resets to zero)
• replace/3 error semantics: non-atomic replace (cancel + re-submit) where cancel succeeds but re-submit fails leaves original order cancelled at broker while OrderManager reverts to "working" locally

Quality assessment:

• GPT-5 maintained its pattern from previous findings: broadest coverage (20 assumptions), most technically specific about implementation details. Found cross-cutting operational concerns (clock skew, credential rotation, pagination) that the Claude models didn't surface. However, several of its findings were medium-severity operational concerns rather than architectural assumptions.
• Claude Sonnet 4.6 was the surprise performer. Found 17 assumptions — close to GPT-5's count (85%) — and several of its unique findings were genuinely insightful. The cancel_all race with broker-side replace state (finding #16) and the lot operation failure propagation (finding #6) show deep reasoning about component interaction despite Sonnet not being positioned as a "reasoning" model. More importantly, Sonnet's findings were consistently well-structured with clear "how it could break" scenarios.
• Claude Opus 4.6 found the fewest assumptions (12) but — consistent with Finding #11 — its unique findings were qualitatively different. The concurrent apply_corrections write conflict, the gate initialization state distinction, and the non-atomic replace error semantics all reveal design tensions that neither GPT-5 nor Sonnet identified. Opus continues to reason about the boundaries between components rather than within-component mechanics.

Key insight — Sonnet 4.6 is NOT just a faster GPT-4.1: In previous findings (#9, #10, #11), non-reasoning models (GPT-4.1, GPT-4.1 Mini) performed significantly below reasoning models on assumption-finding. GPT-4.1 found ~14 assumptions where GPT-5 found 24-26. Here, Sonnet 4.6 finds 17 where GPT-5 finds 20 — a much smaller gap (~85% vs ~58% previously).

Sonnet's findings also included several that showed genuine reasoning about component interactions (not just within-frame risks). This suggests Sonnet 4.6 is qualitatively different from GPT-4.1 for analytical work — it occupies a middle ground between GPT-4.1's "competent but surface-level" and GPT-5's "exhaustive and deep." The severity distribution was also similar to GPT-5 (multiple critical/high findings), whereas GPT-4.1 in previous experiments tended toward medium-severity generic concerns.

Updated model hierarchy for assumption-finding:

1. GPT-5 — broadest coverage, most operational-level findings (20)
2. Sonnet 4.6 — strong analytical depth, good component interaction reasoning (17)
3. Opus 4.6 — fewest but most architecturally insightful, finds design tensions (12)
4. GPT-4.1 — competent within-frame, generic (~14 from previous experiments)
5. GPT-4.1 Mini — formulaic, surface-level (~10-12)

Practical implication: For architecture review, Sonnet 4.6 is now a strong candidate for volume analytical work. It's fast enough to run alongside GPT-5 and catches different things (lot operation failures, broker-side replace races). The ideal three-model review stack for architecture docs appears to be:

• GPT-5 for breadth + operational concerns
• Sonnet 4.6 for component interaction analysis
• Opus 4.6 for design-tension identification

Each consistently finds things the others miss. The cost-efficiency argument for Sonnet is strong: ~85% of GPT-5's count with more actionable findings per token generated (4,637 vs 8,485 tokens for 17 vs 20 assumptions).