model-research/findings/2026-05-02-09-gapfinding-in-architecture-docs-gpt5.md

Finding 9: Gap-finding in architecture docs: GPT-5 finds domain-specific gaps, GPT-4.1 is generic, Mini is formulaic

Date: 2026-05-02 Task: Identify missing failure scenarios in gargoyle's failure-modes.md (383 lines) How we used them: Same document (full text, no truncation) + same focused analytical question to all 3 models via HAI proxy (OpenAI-compatible endpoint). No tools, no project context beyond the document itself. Single prompt, no conversation history. Temperature 0.3 for GPT-4.1/Mini, default (1.0) for GPT-5 (required by the model).

Model	Time	Output tokens	Reasoning tokens	Scenarios found
GPT-4.1 Mini	16s	2,003	0	10
GPT-4.1	24s	2,575	0	15
GPT-5	45s	8,565	6,656	14

What they found — common ground (all 3 identified):

• ETS table corruption/loss affecting gates
• BEAM scheduler starvation / GC pauses
• WebSocket message duplication/reordering
• Postgres connection pool exhaustion / deadlocks
• Clock skew / time drift
• Process registry inconsistency

GPT-5 unique findings (not in either other model):

• Broker rate limiting (429s) — not "connection lost" so existing logic doesn't trigger, but can't flatten during kill switch
• Broker auth failure / credential rotation — distinct from connection loss
• Corporate actions (splits, symbol changes) — position drift without triggering staleness detection
• Duplicate pipeline instances for same user (DynamicSupervisor race)
• DB "commit unknown outcome" causing restart loops (Ecto commit succeeds at Postgres but client times out → retry → unique constraint → crash loop)
• Cross-symbol strategies with partial staleness — multi-leg signals computed from mix of fresh and stale data
• Partial cancel_all during kill switch masked by process restarts

GPT-4.1 unique findings (not in GPT-5 or Mini):

• Zombie processes after halt (supervisor misconfiguration)
• Unsupervised Task crashes going unnoticed
• Audit log writes failing silently (not in same transaction as state change)
• ClOrdID unique constraint violation from race in sequence generation
• Broker API semantic changes (silent breaking changes)

GPT-4.1 Mini unique findings:

• Race between kill switch engagement and reconciliation completion (timing coordination gap) — this was more explicitly called out than in the other models, though GPT-5 touches it implicitly
• Strategy.Worker / Aggregator partial crash inconsistency

Quality assessment:

• GPT-5 had the most domain-relevant and actionable gaps. Broker rate limiting, auth failures, corporate actions, and the DB commit unknown-outcome scenario are all realistic production issues specific to THIS system. The cross-symbol partial staleness finding shows deeper architectural reasoning about component interactions.
• GPT-4.1 was thorough and well-structured but more generic/defensive. Many of its unique findings (zombie processes, unsupervised Tasks, audit log loss) are general Elixir concerns rather than specific to the document's architecture. Good for a completeness checklist.
• GPT-4.1 Mini was formulaic — each finding followed the same template and several were somewhat surface-level or restated things the document partially covers. Still found the most scenarios per dollar.

Takeaway: For gap-finding in architecture documents, GPT-5's reasoning tokens pay off. It doesn't just list "things that could go wrong" — it identifies specific interactions that the document's existing mechanisms don't cover (e.g., rate limiting bypasses the "connection lost" detection, corporate actions bypass staleness detection). GPT-4.1 is a solid middle-ground: more thorough than Mini, less insightful than GPT-5. Mini is fine for a quick sanity check but won't find the subtle gaps.

Cost-effectiveness: Mini found 10 scenarios in 16s for ~7K tokens. GPT-5 found 14 scenarios (with 7 genuinely unique insights) in 45s for ~13.5K tokens (including 6.6K reasoning). For architecture review where missing a gap could mean financial loss, the GPT-5 cost is justified. For routine doc review, Mini + human judgment is probably sufficient.