model-research/findings/2026-05-09-56-operational-burden-analysis.md

Operational Burden Analysis: A New Analytical Lens

Finding ID: 56 Date: 2026-05-09 Document: gargoyle/docs/domain/contexts/risk/escalation-policy.md (~11.8KB, ~240 lines) Task type: Operational burden analysis — a NEW analytical lens Prompt: "Analyze this document for operational burden: manual steps (toil), cognitive load, implicit expertise, decision points lacking guidance, and recovery scenarios without procedures. Focus on what an on-call engineer at 3am would struggle with." Models compared: GPT-5, Claude Sonnet 4.6, Claude Opus 4.6

Experiment Design

This experiment tests a novel analytical lens: operational burden analysis. Unlike gap-finding or contradiction detection, this lens asks: "What cognitive and procedural load does this design place on operators?" This is particularly relevant for systems with manual intervention points, fail-safes, and recovery procedures.

The hypothesis: models with strong real-world reasoning (GPT-5) might excel at enumerating operational scenarios, while models with strong logical structure (Opus) might better identify where documented automation creates undocumented manual work.

Performance Metrics

Model	Time	Input Tokens	Output Tokens	Reasoning Tokens	Findings
GPT-5	74s	2,416	5,843	4,224	21 findings + 3 observations
Claude Sonnet 4.6	55s	2,726	2,586	(internal)	12 findings (excerpt)
Claude Opus 4.6	52s	2,726	2,397	(internal)	12 findings (excerpt)

Common Ground (All Three Models)

All three models identified these CRITICAL issues — the core operational gaps:

1. Manual liquidate→restrict de-escalation has no procedure — The document says "confirm the portfolio is safe" but provides no acceptance criteria, tooling, or checklist. Every model flagged this as CRITICAL.

2. Persistent restrict flag with monitor down requires manual clear without procedure — The failure mode table says "Manual clear" but doesn't specify how, when, or what to verify first.

3. Kill switch recovery is absent — The policy escalates to kill switch automatically but provides no return path. Cross-references exist but no in-document guidance.

Model-Specific Strengths

GPT-5: Exhaustive Enumeration + Operational Context

GPT-5 produced the most comprehensive analysis (21+ findings) with specific operational details:

Unique catches:

• Unspecified "N" in "liquidation attempted + N more breaches" before kill switch — operator can't discover configured value under pressure
• Multi-metric conflicts: no guidance on prioritizing remediation when metrics are at different levels
• Evaluation frequency changes require proportional debounce/cooldown adjustment — no formula provided
• Event stream reconstruction burden: minimal event fields require mental correlation
• Mid-incident threshold adjustments: no safe-change procedure or rollback guidance

Characteristic: GPT-5 thinks like an on-call engineer. It imagines specific 3am scenarios and asks "what would I need to know?" It found configuration gaps (value of N) that the other models missed.

Claude Opus 4.6: Systemic Vulnerability Identification

Opus produced concise findings but identified deeper systemic issues:

Unique catches:

• Monitor crash restarts from Clear is worse than it appears: the "Recovery: Automatic" label is misleading because the system resets to an unsafe state while risk conditions may persist. The debounce window (3-5s) allows the decision engine to open new positions during unmonitored risk.
• Liquidation death spiral: cascading liquidation in illiquid markets could worsen losses, but no guidance on when to override autonomous liquidation.
• Evaluation frequency is referenced but never authoritatively defined — operators can't derive debounce/cooldown timing without it.

Characteristic: Opus identifies where documented behavior creates undocumented risk. Its "silent loss of escalation context" analysis goes beyond "no procedure" to explain why this is dangerous operationally.

Claude Sonnet 4.6: Structured, Complete, Efficient

Sonnet produced well-organized findings with clear severity stratification:

Unique catches:

• Authorization gap: who is authorized to perform manual de-escalation confirmations?
• Timeout uncertainty: can the liquidate state persist indefinitely?
• "Broker unavailable" definition: is it total outage or partial fill failure?

Characteristic: Sonnet thinks like a runbook author. It asks practical questions about authorization, timeouts, and edge definitions. Its analysis is more procedural than the others.

Overlap Analysis

Finding	GPT-5	Sonnet	Opus
Liquidate de-escalation has no procedure	✅ CRITICAL	✅ CRITICAL	✅ CRITICAL
Restrict flag manual clear undefined	✅ CRITICAL	✅ CRITICAL	✅ HIGH
Kill switch recovery absent	✅ HIGH	✅ CRITICAL	✅ HIGH
Broker unavailability response undefined	✅ HIGH	✅ HIGH	❌
Monitor crash state loss danger	❌	❌	✅ CRITICAL
Multi-metric state cognitive load	✅ HIGH	✅ HIGH	❌
Unspecified N before kill switch	✅ CRITICAL	❌	✅ HIGH
Liquidation death spiral risk	❌	❌	✅ HIGH
Evaluation frequency undefined	✅ HIGH	❌	✅ HIGH
Mid-incident configuration changes	✅ MEDIUM	❌	❌
Authorization for manual actions	❌	✅ (implicit)	❌

Union: 19 unique findings across all three models Intersection: 3 findings all agreed on GPT-5 unique: 4 findings Opus unique: 2 findings (but deeper systemic analysis) Sonnet unique: 2 findings

Key Insight: Operational Burden Analysis Is Model-Sensitive

This lens reveals clear model personalities:

1. GPT-5 enumerates exhaustively. It walks through every section and asks "what would an operator need here?" This catches configuration gaps and parameter questions.

2. Opus identifies systemic vulnerabilities. It asks "where does documented automation create hidden risk?" This catches the monitor-crash-to-clear danger that the others missed.

3. Sonnet structures procedurally. It asks "what would a runbook need?" This catches authorization and timeout questions.

For comprehensive operational burden analysis: run all three. GPT-5 finds the parameters, Opus finds the systemic risks, Sonnet fills in the procedural gaps.

Actionable Recommendations for Escalation Policy Document

Based on this analysis, the escalation policy needs:

1. Liquidate de-escalation runbook (CRITICAL)

   • Explicit acceptance criteria for "portfolio is safe"
   • Step-by-step procedure with tooling/interface
   • Authorization requirements

2. Recovery procedures for each failure mode (CRITICAL)

   • Monitor crash: reconciliation procedure, re-escalation timeline
   • Restrict flag stuck: detection, clearance procedure, verification

3. Kill switch recovery section (HIGH)

   • Either in this document or explicit pointer with summary

4. Configuration reference (HIGH)

   • Define N for "liquidation + N breaches"
   • Define default evaluation frequency
   • Document how to discover configured values

5. Operator intervention guidance (HIGH)

   • When to override autonomous liquidation
   • How to handle multi-metric conflicts

Conclusion

Operational burden analysis is a valuable lens that complements gap-finding and contradiction detection. It asks a fundamentally different question: not "what's missing from the spec" but "what will the human need when this runs?"

Model recommendation for this lens:

• Run GPT-5 for exhaustive enumeration
• Run Opus for systemic vulnerability identification
• Optionally run Sonnet for procedural structure
• Union the findings — they don't overlap much

Cost-effectiveness: Opus (52s, 2,726+2,397 tokens) was fastest and found the most dangerous systemic issue (monitor crash → silent unsafe state). For time-constrained analysis, Opus alone catches the critical issues. For thoroughness, GPT-5 + Opus union is optimal.