model-research/findings/2026-05-03-14-crosscomponent-interaction-analysis-gpt5-mini.md

Finding 14: Cross-component interaction analysis: GPT-5 Mini enters as viable mid-tier; task framing strongly shapes output quality

Date: 2026-05-03 Task: Identify cross-component interaction failures in gargoyle's continuous-risk-monitoring.md (459 lines) — a document specifying PortfolioMonitor's interactions with OrderManager, PortfolioRisk, MarketData, KillSwitch, ETS tables, and the pipeline supervision tree. How we used them: Same document (full text) + same focused analytical question to all 3 models via HAI proxy. Prompt was highly structured: specified 5 categories of cross-component failures to look for (semantic mismatches, ordering violations, feedback loops, partial visibility, supervision boundary effects) and required specific output format (components, sequence, gap, impact). No tools, no project context beyond the document itself.

Model	Time	Output tokens	Reasoning tokens	Findings
GPT-5 Mini	68s	5,445	2,240	6 (+1 cut off)
GPT-5	116s	10,604	8,128	10
Claude Sonnet 4.6	38s	1,868	(internal)	8

What they found — common ground (all 3 identified):

• Fill-to-position query race (fill event triggers evaluation but position store hasn't yet reflected the fill)
• Restrict flag ETS table destruction on PM crash → permissive window
• Kill switch check vs liquidation submission race
• Ticker subscription timing gap (new position opened but ticks not yet subscribed → breach goes undetected)

GPT-5 unique findings (not in either other model):

• Stale prices are NOT fail-safe for drawdown (higher stale price → inflated portfolio value → understated drawdown). The document claims "fail-safe" but this only holds for exposure metrics, not drawdown. This is the most architecturally significant finding across all three models.
• Price definition mismatch between PM (last_trade from ETS) and OrderManager/ broker (bid/ask/mid) causing mis-sized liquidation and oscillation
• Cross-component oscillation: PM hysteresis internal vs PRisk's immediate binary restrict gate clearing (no cross-component cooldown)
• Liquidation stuck after OM restart (terminal events lost; liquidation_in_ flight stays true indefinitely with no timeout/rehydration)
• "Minimal risk checks" not enforced — PM goes through same OM gates as strategy orders but MarketHours/StalePrice controls may reject after-hours or stale-price liquidation attempts
• FLATTEN mode semantics gap — PM refrains from liquidating when kill switch engaged, but FLATTEN cancels open orders without actually CLOSING positions. No component left to close positions.

Claude Sonnet 4.6 unique findings (not in either other model):

• Liquidation feedback loop with PortfolioRisk — buy-to-cover for short positions could INCREASE net long exposure at portfolio level, paradoxically worsening concentration while fixing position-level metrics
• High water mark reset on pipeline restart masks true intraday drawdown (restart → HWM resets to lower current value → drawdown calculated from false baseline → larger losses permitted than intended)
• Multi-metric breach with single boolean flag — concentration liquidation for AAPL sets liquidation_in_flight, blocking simultaneous daily P&L liquidation for different positions
• Market close/open vs after-hours fills — claims to evaluate after-hours fills but uses stale market-close prices

GPT-5 Mini unique findings (not in either other model):

• OrderManager order splitting/remapping causing liquidation_in_flight correlation failure (parent/child order ID mapping breaks terminal-event detection). Well-reasoned but highly implementation-specific.
• Restrict/clear oscillation loop with strategy behavior (strategies react to rejects → back off → restrict clears → strategies re-enter aggressively → re-breach). Good systems-thinking about emergent feedback.

Quality assessment:

• GPT-5 produced the most findings (10) and the highest-quality architectural insight: the stale-price/drawdown contradiction is a genuine design flaw that contradicts the document's own safety claim. Multiple findings showed cross-boundary reasoning about semantic mismatches (price definition, FLATTEN semantics, gate bypass). Every finding named specific components and described precise event sequences.
• Claude Sonnet 4.6 was fast (38s, only 1,868 tokens) and produced 8 solid findings. The HWM reset finding and the multi-metric/single-flag finding show genuine architectural reasoning. The liquidation feedback loop (buy-to-cover worsening portfolio concentration) is subtle and shows cross-position reasoning. However, some findings overlapped significantly with the common-ground set and added less unique depth. Sonnet performed MUCH better here than on race condition identification (Finding #13) — 8/10 ratio vs 7/12 previously.
• GPT-5 Mini produced 6 findings in 68s with 2,240 reasoning tokens. Quality was genuinely good — the order-splitting/correlation finding and the oscillation feedback loop both show real reasoning depth. It's clearly NOT GPT-4.1 Mini — it reasons about component interactions, not just within-frame risks. However, it found fewer issues and one response was cut off (token limit or response truncation).

Key insight — task framing as the dominant variable: This experiment used a much more structured prompt than previous ones: specified 5 categories, required specific output format, explicitly excluded single-component failures. The result: ALL models produced higher-quality, more focused output than in earlier experiments with broader prompts. Even Sonnet — which struggled on race conditions (Finding #13) — performed well here. The structured categories likely helped models organize their reasoning without losing track of what they were looking for.

The prompt explicitly asked for "cross-component interaction failures" rather than general analysis. This is the narrow-lens effect from Finding #2, but applied to a complex multi-component document. The lens is narrow (only inter-component gaps) but the scope is broad (459 lines, many interactions). This combination — narrow analytical lens + broad document scope — appears to be the sweet spot for getting quality from all model tiers.

GPT-5 Mini positioning: First time testing GPT-5 Mini. Results: 6 findings in 68s vs GPT-5's 10 in 116s. That's 60% of the findings in 59% of the time, with 28% of the reasoning tokens (2,240 vs 8,128). Quality-per-finding was solid — the order correlation finding especially showed genuine systems reasoning. GPT-5 Mini appears to be a legitimate mid-tier: more capable than GPT-4.1 (which can't do this kind of cross-boundary reasoning) but less exhaustive than GPT-5. Viable for: first-pass screening, bulk document review where you'd run many docs and can't afford full GPT-5 on each.

Sonnet recovery from Finding #13: Sonnet went from 7 findings (with errors) on race conditions to 8 solid findings here. The difference: this prompt was more structured, the document was larger with more explicit interaction descriptions, and the task didn't require pure temporal/sequential reasoning. "Cross-component interaction failures" is closer to assumption-finding (Sonnet's strength) than race condition identification (Sonnet's weakness). Task taxonomy continues to matter more than raw model capability.

Updated model assignment for cross-component analysis:

1. GPT-5 — broadest coverage, finds semantic contradictions in the doc's own claims (10 findings)
2. Sonnet 4.6 — fast and efficient, good at portfolio-level reasoning and feedback loops (8 findings in 38s)
3. GPT-5 Mini — viable first-pass with genuine reasoning depth (6 findings)
4. (Opus untested for this task type — likely strong on design tensions)