model-research/findings/2026-05-08-46-operational-blind-spot-analysis-observability.md

Finding #46: Operational blind spot analysis — a new task type revealing model divergence on "what's invisible"

Date: 2026-05-08 Task: Identify operational blind spots in gargoyle's observability.md (563 lines) — scenarios where the observability design would fail to surface problems, actively mislead operators, provide signal too late, or create diagnostic dead-ends. Novel aspect: This is a new analytical task type distinct from assumption-finding (#10-12), race condition identification (#13), cross-component interaction (#14), or gap-finding (#9). It requires reasoning about the ABSENCE of signal — what can't you see? — and how instrumentation choices create false confidence.

Setup

Same document (full text, no truncation) + same focused analytical question to all 3 models via HAI proxy. Prompt specified 4 categories of blind spot (fail to surface, actively mislead, signal too late, diagnostic dead-ends) and required specific output format (Scenario, Why invisible/misleading, Impact, Detection gap). Explicitly asked for trading-specific scenarios and references to specific mechanisms in the document.

Model	Time	Output tokens	Reasoning tokens	Blind spots found
GPT-5	~87s	7,969	5,056	18
Claude Opus 4.6	~120s	4,591	(internal)	12
Claude Sonnet 4.6	~38s	1,930	(internal)	10

What they found — common ground (all 3 identified)

• Order state post-submission is invisible (traces terminate at order_manager.submit; no metrics for fills/cancels/rejects from broker)
• Position reconciliation gap (no metric for internal-vs-broker position drift)
• Market data staleness per-symbol undetectable at scale (cardinality exclusion of symbol from quote_feed.tick metadata means per-symbol staleness invisible in Prometheus)
• Risk controls only measure rejections, not false approvals (stale data causing incorrect approvals is invisible)
• LogStore GenServer mailbox/write failure silent due to async cast pattern
• Aggregator signal buffering without emission has no observability

GPT-5 unique findings (not in either Claude model)

• Sampling + span links break end-to-end traceability: When signal evaluation sampling is reduced, Aggregator span links reference unsampled trace IDs → Tempo shows broken links, operators underestimate true latency
• Webhook correlation dead-end: opentelemetry_phoenix "does not capture HTTP headers"; inbound broker webhooks have no body-parsed attributes linking back to decision_id
• 15s Prometheus scrape + 10s VM poller too coarse for trading incidents: 5-10 second broker hiccups vanish between scrapes; micro-incidents permanently invisible
• Price integrity/slippage invisible: Span attribute allow-list forbids price_target; no metric for filled_avg_price vs intended → systematic slippage undetectable
• Duplicate decisions undetectable from current labels: Counter tags (ticker/action/strategy) can't distinguish legitimate trades from duplicates; decision_id isn't a Prometheus label
• End-to-end reconstruction depends on both correlation keys always present: If a code path omits signal_id or decision_id, the three-way join breaks silently
• Live logs UI shares Phoenix endpoint: Under load, trading continues but LogsLive becomes unreachable — operators lose situational awareness exactly when needed
• Per-symbol staleness metric contradicts cardinality rules: feed.stale with ticker tag will be removed in practice, leaving only global signals
• Metric/trace/log pillars can disagree without alerting: Sampling/exporter failures make metrics look green while traces vanish; no cross-pillar consistency check
• Broker rate-limiting/HTTP failures at call-site: req not in deps means no outgoing HTTP instrumentation; retries/429s produce no trace or metric
• Tempo/OTLP exporter outage: No metric for exporter drop counts; operators see "no traces" and may assume "nothing happening" vs "monitoring broken"
• Order submission counter "green" while broker later rejects: Post-submit broker-side rejections have no counter or lifecycle span

Claude Opus unique findings (not in either other model)

• Aggregator swallowing signals without emission: Signals entering the buffer that never trigger aggregation produce ZERO observability signal — no error, no metric, no timeout event. Distinguished from the common "aggregator gap" finding by reasoning about the specific mechanism: the aggregator.aggregate span only fires on aggregation, not on receipt. Buffered-and-forgotten signals are completely invisible.
• OTel context propagation failure is silent: OpenTelemetry.Ctx.attach(nil) returns a no-op token; with_span creates a ROOT span (different trace_id). Traces look structurally valid but are fragmented. No validation exists for context propagation success.
• Strategy logic bug with correct observability: All pillars report success when a strategy emits the wrong signal (inverted buy/sell). No metric for financial correctness (P&L, win rate, signal-vs-price-movement) — only operational correctness.
• Decision latency histogram is survivor-biased: Only records COMPLETED journeys. Stuck decisions never appear. Histogram shows healthy p99 while decisions are lost. Distinguished from generic "mailbox" finding by identifying the specific statistical bias in the metric design.
• Broker WebSocket reconnect succeeds but misses fill events during gap: Reconnect counter says "recovered"; fills during disconnect are permanently lost. Orders stay in "submitted" state forever with no timeout metric.
• @keep_keys allowlist silently drops new metadata: Developer adds critical metadata key, forgets to update allowlist. Logs appear in viewer (message intact) but structured queries return nothing. Diagnostic dead-end that ACTIVELY misleads.

Claude Sonnet findings (unique aspects)

• Partial fill accumulation without detection: Systematic 90% fills create gradual position drift — no metric for unfilled quantity accumulation
• Aggregator timing bias creating systematic directional bias: Varying processing latencies cause buy signals to consistently arrive before sells
• Signal ID collision corrupting audit trail: UUID weakness creating impossible timelines in correlation queries
• Telemetry emission timing creating phantom latencies: Mailbox congestion in telemetry handlers makes latency metrics unreliable (showing delays that don't reflect actual performance)

Quality assessment

GPT-5 (18 findings): Most exhaustive as usual. Strong on operational/infrastructure-level blind spots (scrape granularity, exporter health, HTTP instrumentation gaps, cross-pillar disagreement). Several findings showed careful reading of the document's specific configuration choices (15s scrape, @keep_keys, span attribute allow-list). However, some findings were variations on the same theme (multiple findings about post-submission order lifecycle gaps). Every finding referenced specific document mechanisms and explained the causal chain clearly.

Claude Opus (12 findings): Highest insight-per-finding density. Two standout findings:

1. The Ctx.attach(nil) silent fragmentation — this is genuinely subtle and requires understanding Erlang OTel SDK internals to recognize that nil context creates valid-looking but uncorrelated traces. No other model caught this.
2. The @keep_keys metadata stripping creating a diagnostic dead-end that ACTIVELY misleads (investigator sees log, queries by key, gets nothing, concludes event never happened). This is the only finding across all models that describes observability creating a FALSE NEGATIVE in investigation rather than just a gap.

Opus also continued its pattern of identifying design tensions: the strategy-correctness finding (#7) explicitly names the gap between "operational correctness" (system works) and "financial correctness" (system works RIGHT) — a fundamental architectural blind spot that the other models only touched peripherally.

Claude Sonnet (10 findings): Weakest performance in this experiment. Several findings were plausible but somewhat generic or low-specificity compared to the other models:

• "Signal ID collision" assumes a UUID weakness that isn't evidenced in the document
• "Telemetry emission timing creating phantom latencies" is theoretically possible but doesn't reference specific document mechanisms that would cause it
• "Aggregator timing bias" is an interesting idea but doesn't explain WHY this specific observability design would miss it

Sonnet's best finding was the partial-fill accumulation (no metric for systematic underfill), which is genuinely trading-specific. But overall, it produced fewer findings, with less document-grounding and more speculation.

Key insight — "what's invisible" requires reasoning about negation

This task type is fundamentally about NEGATION: "given what IS instrumented, what ISN'T?" This is harder than assumption-finding (which can work from what's stated) or race condition analysis (which works from what's specified about concurrency). Here, the model must:

1. Build a mental model of what the observability design CAN see
2. Enumerate production scenarios (requiring domain knowledge)
3. Check each scenario against the coverage model
4. Identify scenarios that fall in the gaps

GPT-5 excelled at step 1 (thorough coverage mapping) and step 3 (systematic checking). Opus excelled at step 2 (finding subtle scenarios like context propagation failure) and identifying findings that ACTIVELY mislead rather than passively miss. Sonnet struggled with step 3 — some of its scenarios were valid but its explanations of WHY the observability design specifically misses them were weaker.

Comparison to previous task types

Task type	GPT-5	Opus	Sonnet
Assumption-finding (#10-12)	20-26	12-13	17
Race conditions (#13)	12	10	7 (with errors)
Cross-component (#14)	10	—	8
Blind spot analysis	18	12	10

The ratios are consistent: GPT-5 ~1.5x Opus, Opus ~1.2x Sonnet. But quality-per-finding continues to favor Opus for finding the most architecturally insightful issues. GPT-5's breadth advantage is real but includes more operational/infrastructure findings vs Opus's focus on design-level blind spots.

Practical implications

1. New analytical task for architecture review: "What can't you see?" is a distinct and valuable question to ask of any observability or monitoring design. It's not covered by assumption-finding, gap-finding, or consistency checking.

2. Model assignment for blind spot analysis:

   • GPT-5: Operational blind spots (infrastructure interactions, configuration gaps, cross-system dependencies)
   • Opus: Design-level blind spots (false confidence, active misdirection, semantic gaps between what's measured and what matters)
   • Sonnet: Not recommended for this task type — insufficient document grounding

3. The "actively misleads" category is highest-value: Of all findings across 3 models, the ones that describe observability CREATING false confidence (rather than just missing signal) are the most dangerous and actionable. Opus found 3 of these; GPT-5 found 2; Sonnet found 0. This suggests Opus should be specifically tasked with: "Where does this design create false confidence?"

Updated task-model matrix

Task	Best model(s)	Why
Assumption-finding	GPT-5 + Opus	Breadth + design tensions
Race conditions	GPT-5 + Opus	Sonnet unreliable for concurrency
Cross-component	GPT-5 + Sonnet	Both good; Sonnet recovers with structure
Cross-document consistency	Opus + GPT-5	Opus dominates boundary reasoning
Operational blind spots	GPT-5 + Opus	GPT-5 for coverage mapping; Opus for false confidence
Bias detection	Any (with narrow framing)	Signal-to-noise matters more than model

Source

• Document: gargoyle/docs/impl/observability.md (563 lines)
• Models: GPT-5 (via HAI OpenAI endpoint), Claude Opus 4.6, Claude Sonnet 4.6 (via HAI Anthropic endpoint)
• No tools, no project context beyond the document itself