model-research/findings/2026-05-09-54-wash-sale-multi-model-analysis.md

# Multi-Model Analysis: Wash Sale Design Document Review

**Finding ID:** 54
**Date:** 2026-05-09
**Document:** gargoyle/docs/domain/contexts/ledger/wash-sale-tracking.md
**Task type:** Design doc edge case analysis
**Prompt:** "What edge cases, ambiguities, or potential bugs might this design miss?"
**Models compared:** Claude Sonnet 4, GPT-5, Claude Opus 4.6

## Experiment Design

This experiment compares how different frontier models analyze a real production design document for a trading platform's wash sale tracking feature. The document involves:
- Tax law compliance (IRC §1091)
- Event-driven architecture
- Domain modeling with immutable events
- Edge cases in financial calculations

## Key Findings by Model

### Claude Sonnet 4 (5.5K tokens, ~5s latency)

**Strengths:**
- Fast, structured response with clear categories
- Strong on IRS compliance gaps (IRA permanent disallowance, cross-account rules, retirement accounts)
- Good on "substantially identical" scope limitations
- Identified trade date vs timestamp issue

**Unique catches:**
- Explicitly called out retirement account interactions (IRA/401k)
- Noted constructive sale provisions (Section 1259)
- Annual wash sale carryover across tax years

**Weaknesses:**
- Less depth on event ordering/concurrency
- Formula correctness analysis less thorough

### GPT-5 (16K tokens with 4K reasoning, ~3min latency)

**Strengths:**
- Most comprehensive coverage
- Exceptional detail on implementation ambiguities
- Strong on ordering/allocation algorithm gaps
- Specific bug-prone scenarios with concrete examples
- Excellent actionable recommendations section

**Unique catches:**
- Fractional shares/rounding policy missing
- Fees/commissions treatment undefined
- Short sale handling completely absent
- Most detailed on multiple-lot allocation problem
- Concrete numeric examples (double-counting across replacements)

**Weaknesses:**
- Required 16K token budget (4K for reasoning alone)
- 3+ minute latency
- Verbose — some redundancy across sections

### Claude Opus 4.6 (16K tokens, ~2min latency)

**Strengths:**
- Deepest reasoning about edge cases
- Best on chain wash sales (daisy-chaining) scenario
- Excellent on concurrent detection race conditions
- Strong "holding period tacking" correctness analysis (add vs backdate)
- Clear "highest-risk issues" prioritization

**Unique catches:**
- Backdating vs adding holding period — IRS requires adding, design does backdating (different results in edge cases)
- FIFO ordering per Rev. Rul. 85-4 for multiple replacements
- Gain-then-loss sequence scenario
- Forward detection on purchase arriving before backward detection on same-day sale
- Adjustment event interleaving with P&L queries

**Weaknesses:**
- Slower than Sonnet for similar category coverage
- Some overlap with GPT-5 on cross-account gaps

## Synthesis: What Each Model Catches

| Finding | Sonnet | GPT-5 | Opus |
|---------|--------|-------|------|
| Cross-account wash sales missing | ✅ | ✅ | ✅ |
| IRA permanent disallowance | ✅ | ✅ | ✅ |
| Multiple replacement lot allocation | Partial | ✅ | ✅ |
| Short sale handling absent | ✅ | ✅ | ✅ |
| Trade date vs timestamp | ✅ | ✅ | ✅ |
| Substantially identical too narrow | ✅ | ✅ | ✅ |
| Chain wash sales (daisy-chaining) | ❌ | Partial | ✅ |
| Holding period add vs backdate | ❌ | ❌ | ✅ |
| FIFO ordering per IRS rules | ❌ | Partial | ✅ |
| Concurrent detection race | ❌ | ✅ | ✅ |
| Rounding/fractional shares | ❌ | ✅ | ✅ |
| Fees/commissions treatment | ❌ | ✅ | ❌ |
| Corporate action edge cases | ✅ | ❌ | ✅ |
| Year-end boundary handling | ✅ | ❌ | ✅ |
| Section 1259 constructive sales | ✅ | ❌ | ❌ |
| Concrete numeric examples | ❌ | ✅ | ❌ |
| Actionable recommendations | Partial | ✅ | Partial |

## Model Selection Guidance

**For quick design review (time-critical):** Sonnet — catches most high-severity compliance gaps, fast enough for interactive use.

**For comprehensive pre-implementation review:** GPT-5 — exhaustive coverage, actionable recommendations, but budget time and tokens (16K+ completion tokens needed).

**For deep edge case analysis:** Opus — best at chain scenarios, ordering/concurrency, subtle correctness issues. Good for final review before production.

**Optimal pipeline:**
1. Sonnet for initial triage (identifies categories)
2. GPT-5 or Opus for deep dive on specific high-risk areas Sonnet flagged
3. Both GPT-5 and Opus if the domain is critical (tax compliance, financial calculations)

## Surprising Results

1. **Opus caught holding period semantics that others missed** — The IRS requires *adding* holding periods, not *backdating*. This produces different results when loss lot open date ≠ (replacement open date - loss holding period). Neither GPT-5 nor Sonnet caught this.

2. **GPT-5's reasoning tokens consumed 4K before any output** — At 4K max_completion_tokens, GPT-5 returned empty content (all tokens went to reasoning). This is a critical operational consideration.

3. **Sonnet caught Section 1259 (constructive sales)** — A relatively obscure IRS provision that neither Opus nor GPT-5 mentioned. Suggests Sonnet may have fresher/broader tax law training data.

4. **All three missed the same thing** — None explicitly addressed what happens when a user has *multiple brokers* reporting different 1099-Bs with different wash sale treatments. The reconciliation problem is real and untreated.

## Cost Analysis (estimated)

| Model | Input tokens | Output tokens | Latency | Relative cost |
|-------|--------------|---------------|---------|---------------|
| Sonnet | ~1.3K | ~5.5K | ~5s | 1x |
| GPT-5 | ~1.3K | ~16K (4K reasoning) | ~180s | ~8x |
| Opus | ~1.3K | ~16K | ~120s | ~6x |

## Lessons Learned

1. **GPT-5 token budget is critical** — Must use `max_completion_tokens` ≥16K for reasoning-heavy tasks. 4K produces empty output because reasoning consumes all tokens.

2. **Model blind spots are complementary** — Each model caught things the others missed. Multi-model review is justified for high-stakes domains.

3. **Opus excels at subtle correctness issues** — The holding period add-vs-backdate distinction is exactly the kind of thing that causes production bugs months later.

4. **Sonnet's speed enables iteration** — At ~5s latency, you can run Sonnet multiple times with different prompts in the time it takes for one GPT-5 response.

## Conclusion

For design document review of financial/regulatory domains:
- **Single model:** GPT-5 with ≥16K completion tokens is most comprehensive
- **Speed/cost constrained:** Sonnet catches ~70% of critical issues at ~10x lower cost
- **Multi-model pipeline:** Sonnet → Opus catches the most unique issues (complementary blind spots)
- **GPT-5 + Opus overlap** is high (~80%) but each has unique catches

The multi-model approach is justified for high-stakes domains where the cost of a missed edge case exceeds the cost of running 2-3 models.