model-research/findings/2026-05-10-64-specification-gap-analysis.md

Specification Gap Analysis — Finding #64

Date: 2026-05-10
Lens: Implementation Specification Gaps (NEW)
Document: gargoyle's specid-lot-selection.md (125 lines)
Models: GPT-5, Claude Opus 4, Claude Sonnet 4

Summary

New analytical lens examining implementation specification gaps — areas where an implementer would need to make decisions the specification doesn't explicitly address. Unlike hidden assumptions (what must be true) or race conditions (temporal interactions), this lens focuses on ambiguity that leads to divergent implementations.

The prompt explicitly asked for:

1. Underspecified algorithms
2. Edge case silence
3. Ordering ambiguity
4. Implicit prerequisites
5. Consistency gaps
6. Semantic ambiguity

Results

Model	Time	Output tokens	Reasoning tokens	Gaps found	High impact	Medium
GPT-5	126s	9,561	6,592	17	13	4
Claude Opus 4	63s	3,951	(internal)	18	12	4
Claude Sonnet 4	28s	1,639	(internal)	11	5	6

Key Findings by Category

Tie-Breaking Ambiguity (all models found)

• HIFO tie-breaking — Two lots with identical cost basis: sort by opened_at? lot_id? quantity?
• FIFO/LIFO tie-breaking — Two lots with identical timestamps: deterministic secondary key?
• All three models correctly identified this as HIGH impact (affects tax classification)

Regulatory Compliance Gaps (all models found)

• Settlement timing — Treasury Regulation §1.1012-1(c) requires designation "at or before settlement" but spec doesn't define when settlement occurs or when designation locks
• Holding period calculation — "Lots held > 1 year" but spec doesn't define whether this is trade date, settlement date, timestamp-precise, or date-only

Partial Fill Handling (GPT-5 + Opus, not Sonnet)

• Manual selection + partial fills — User selects Lot A:600, Lot B:400 for 1000-share sell. Only 300 fills. Pro-rata? Sequential? Reject?
• Sonnet mentioned partial lot closure mechanics but missed the manual selection + partial fill interaction
• HIGH impact: partial fills are common; ambiguity affects every partial fill with Manual selection

GPT-5 Unique Findings (6 not in either Claude model)

1. Fees/commissions in gain formula — gain = (sell_price - cost_basis) × quantity but no mention of fees. Different tax systems handle fees differently.
2. Multi-execution price handling — Fill at 50@$10.00 + 50@$9.90. Use VWAP? Per-execution closures? Pro-rate fees?
3. Sell exceeds available long — Algorithm assumes consuming open lots but doesn't specify shorting or rejection
4. Account/portfolio scoping — "Correct instrument" but no mention of cross-account lot selection
5. Partial metadata availability — What if opened_at is missing but basis exists? All-or-nothing failure?
6. Basis at selection time vs post-adjustment — HIFO uses current basis but wash sales can adjust basis retroactively. Idempotent replay could choose different lots.

Opus Unique Findings (5 not in either other model)

1. Strategy evaluation timing — User changes HIFO→FIFO between order submission and fill arrival. Which applies?
2. Corporate action during consumption — Stock splits while algorithm is mid-walk through lots
3. Lot ledger invariants — Assumes re-derivation works but doesn't state invariants (lots sum to position, no negative shares)
4. "Last fill wins" vs audit completeness — Serialized processing drops fills but audit trail requires completeness
5. Same-day open/close — Buy at 9:00, sell at 15:00 same day. Lot opening and closing overlap in time. Consumption algorithm handles this?

Sonnet Unique Findings (1 not in other models)

1. Instrument delisting during processing — Order placed, stock delisted before fill. Proceed or fail?

• Note: This is a valid edge case but lower priority than the implementation ambiguities GPT-5 and Opus found

Model Analytical Style Comparison

GPT-5: Exhaustive enumeration with concrete divergence examples

GPT-5 produced the most detailed divergence scenarios, often with three different implementation approaches spelled out. Its fee/commission finding (#5) is particularly valuable — a real-world tax preparation concern that neither Claude model mentioned. The multi-execution price handling (#6) shows deep domain reasoning about how fills actually arrive.

Strength: Real-world operational considerations (fees, multi-price fills, cross-account) Weakness: Sometimes conflates "things the spec doesn't mention" with "things the implementer must guess"

Opus: Regulatory and system-boundary focused

Opus found fewer gaps but they were more architecturally significant. The "strategy evaluation timing" gap reveals a fundamental question about when configuration binding occurs. The "lot ledger invariants" gap identifies that the spec assumes self-consistency without stating the rules. The "last fill wins vs audit completeness" finding shows Opus's characteristic strength at finding consistency tensions.

Strength: Design-level contradictions and boundary ambiguities Weakness: Less operational detail than GPT-5

Sonnet: Structural scan with lower depth

Sonnet found 11 gaps vs 17-18 for the other models. Its findings were valid but shallower — it identified tie-breaking ambiguity but didn't explore cross-account, multi-execution, or fee implications. The "instrument delisting" finding was unique but low-priority.

Strength: Fast (28s vs 63s/126s), correct on core issues Weakness: Misses operational nuance and doesn't reason about component interactions

Novel Insight: Implementation Spec Gaps as Lens

This analytical lens differs from previous experiments:

Previous lens	Focus	Example
Hidden assumptions	What must be true for this to work	"Assumes broker API returns all fills"
Race conditions	Temporal interleavings that cause bugs	"Fill arrives before lot state updates"
Specification gaps (NEW)	What implementer must decide that spec doesn't	"HIFO tie-breaking undefined"

Key distinction: Specification gap analysis is about ambiguity of intent, not risks or assumptions. A spec can be internally consistent, assume correct inputs, have no race conditions, and STILL be underspecified — leading to divergent implementations that all "follow the spec."

This is particularly valuable for:

• Design review before implementation
• Documentation quality assessment
• Identifying where tests should specify behavior

Overlap Analysis

Gap type	GPT-5	Opus	Sonnet	All 3
Tie-breaking ambiguity	✓	✓	✓	✓
Settlement/holding period	✓	✓	✓	✓
Partial closure mechanics	✓	✓	✓	✓
Manual + partial fills	✓	✓	—	—
Fees/commissions	✓	—	—	—
Multi-execution pricing	✓	—	—	—
Strategy evaluation timing	—	✓	—	—
Cross-account scoping	✓	—	—	—
Lot ledger invariants	—	✓	—	—
Instrument delisting	—	—	✓	—

Overlap rate: 5 gaps found by all 3 models, 12+ gaps unique to one model.

Practical Implication

For specification quality review (not architecture review), run:

1. GPT-5 — Catches operational/financial edge cases (fees, multi-fills, cross-account)
2. Opus — Catches design-level binding ambiguities and consistency gaps
3. Optional Sonnet for speed — Catches structural issues but misses depth

The union of GPT-5 + Opus findings would produce 23+ unique gaps vs 11-18 from any single model.

Cost-Efficiency

Model	Gaps	Time	Tokens	Gaps/minute	Gaps/1K tokens
GPT-5	17	126s	9,561	8.1	1.8
Opus	18	63s	3,951	17.1	4.6
Sonnet	11	28s	1,639	23.6	6.7

Opus is the most cost-effective for this task type — 2.5x more efficient than GPT-5 (gaps per token) while finding comparable depth. Sonnet is fast but misses too much for serious specification review.

Recommendations

1. Spec quality gate: Before implementation starts, run Opus + GPT-5 on the spec with this prompt. Address HIGH-impact gaps before coding.
2. Different from architecture review: This is a documentation quality check, not a safety review. Different skill for different purpose.
3. Domain expertise matters: Several GPT-5 findings (fees, multi-execution pricing) reflect financial domain knowledge. For domain-specific specs, GPT-5's breadth may be worth the extra cost.