model-research/findings/2026-05-05-23-regulatory-compliance-analysis-gpt5-finds.md

Finding 23: Regulatory compliance analysis: GPT-5 finds broadest scope of IRS issues; Opus self-corrects mid-analysis; all models converge on cross-account gap

Date: 2026-05-05 Task: Identify where gargoyle's wash-sale-tracking.md (391 lines) could produce incorrect tax reporting compared to IRS wash sale regulations (IRC 1091). NEW analytical lens: regulatory compliance verification — asking models to reason about a code implementation's correctness against EXTERNAL regulatory requirements (not internal system assumptions or race conditions). How we used them: Same document (full text) + same focused analytical question to all 3 models via HAI proxy. Prompt structured around 5 categories: regulatory gaps, interpretation errors, edge cases at regulatory boundaries, cross-account/entity concerns, and interaction with other IRC sections. Required specific regulatory citations, implementation analysis, concrete tax errors, and audit risk levels. No tools, no project context beyond the document.

Model	Time	Output tokens	Reasoning tokens	Findings
GPT-5	178s	12,525	9,536	16
Claude Opus 4.6	155s	7,326	(internal)	16 (with 2 self-corrections/withdrawals)
Claude Sonnet 4.6	40s	1,818	(internal)	12

What they found — common ground (all 3 identified):

• Cross-account/IRA/external broker wash sales not tracked (IRS applies at taxpayer level)
• Options/contracts to acquire stock not triggering wash sales (explicit in IRC 1091(a) text)
• "Substantially identical" definition too narrow (same index ETFs, share classes, ADRs)
• Trade date vs settlement date ambiguity in opened_at/closed_at
• Short sale wash sales not addressed
• Section 475 mark-to-market traders incorrectly subjected to wash sale tracking
• IRC 1092 straddle rules interaction not addressed
• Related party / spousal transactions not considered
• Corporate action identity changes breaking matching

GPT-5 unique findings (not in either other model):

• Per-share vs lot-level basis tacking (#1): The system applies disallowed_loss and tacked_opened_at at the LOT level, but IRS requires per-share treatment when only partial shares are matched. A lot of 100 shares where only 60 trigger wash sale should have per-share basis segregation — the system inflates basis for all 100 shares. Most architecturally significant finding — a fundamental design-level error, not a missing feature.
• IRA permanent disallowance (#2): When replacement purchase is in an IRA, the loss is PERMANENTLY lost (no basis adjustment possible in tax-deferred accounts). System either incorrectly applies basis adjustment inside IRA or misses it entirely.
• Instruments not subject to §1091 (#4): §1256 contracts (futures, index options), cryptocurrency, and §475 elections are all exempt — system may over-disallow.
• Average-cost mutual fund basis (#11): Wash sale adjustments for funds using average-cost method require different math than discrete lot-level adjustments.
• ADRs vs local shares (#14): ADRs and underlying foreign ordinaries are substantially identical but have different instrument_ids.
• RSU vestings/ESPP purchases (#15): Equity compensation creating lots via corporate action paths may not trigger check_replacement/2.
• Ordering priority between pre/post sale purchases (#10): Industry convention (post-sale first, then pre-sale) may differ from system's strict chronological ordering, causing 1099-B mismatches.

Claude Opus unique findings (not in either other model):

• Year-end boundary timing (#5): Loss in December + replacement in January means tax reports generated between Dec 31 and the replacement purchase date are incorrect. Forward detection fires retroactively but users may have already filed. System needs a "30-day pending window" for year-end reports.
• Form 8949 reporting format (#6): IRS requires code "W" in column (f) and specific adjustment amounts in column (g). System doesn't describe how tax_summary/3 produces Form 8949-compatible output — potential CP2000 notice triggers from automated IRS matching against broker 1099-B.
• "Open lots" query in backward detection (#10): If backward detection only queries currently-open lots, it misses replacements that were acquired AND SOLD within the window. IRS looks at acquisition regardless of current holding status. (Rev. Rul. 56-602)
• Forward detection loss ordering unspecified (#7): When multiple prior losses compete for the same replacement shares, ordering matters — different allocation produces different basis amounts on the replacement lot.
• DRIP reinvestments triggering wash sales (#9): Dividend reinvestment creates new lots that should trigger forward detection but may not if only buy fills produce LotOpened events.
• Self-correcting analytical style (CONFIRMED): Opus withdrew Finding #4 entirely mid-analysis ("Revised assessment: holding period logic appears correct. I withdraw the claim of error"). Spent ~500 words reasoning through the holding period tacking logic, found it correct, and explicitly retracted. This is now confirmed across Findings #15, #20, and #23 as a consistent Opus behavior for verification-heavy regulatory analysis.

Claude Sonnet unique findings (not in either other model):

• Entity-level tracking for partnerships/S-Corps (#4.2): Tax-transparent entities trading through the platform need K-1 reporting to partners — user-scoped model doesn't address pass-through entity wash sale reporting.
• Constructive sale integration (IRC 1259) (#4.1): Short positions or derivatives creating constructive ownership interact with wash sale determination in ways not addressed.
• NOL carryforward interaction (#5.3): Wash sale deferrals affect character and timing of losses contributing to NOL calculations across tax years.

Quality assessment:

• GPT-5 produced the broadest regulatory scope (16 findings) with the most specific IRS citations (Rev. Rul. 2008-5, Pub. 550, IRC §§267, 1091, 1092, 1222, 1223, 1256, 475). Its per-share vs lot-level finding (#1) is the only one that identifies a FUNDAMENTAL DESIGN ERROR (not a missing feature). Most other models' findings are "you don't handle X" — GPT-5's #1 says "what you DO handle is handled INCORRECTLY." This distinction matters: missing features are known scope limitations; incorrect logic is a bug.
• Claude Opus matched GPT-5's count (16 with 2 self-corrections = 14 net confirmed) but with different character. Opus excelled at identifying OPERATIONAL implications (year-end boundary timing, Form 8949 format requirements, forward detection ordering) rather than just statutory gaps. Its findings tend to describe HOW the gap manifests in practice ("user files taxes, then January purchase retroactively invalidates the filing") vs GPT-5's approach of citing the statute and describing the theoretical violation.
• Claude Sonnet was fast (40s) and produced 12 competent findings but with less regulatory precision. Findings lacked specific IRS citations (no Rev. Rul. references, no Treas. Reg. citations). Several findings overlapped heavily with common ground items without adding unique depth. The entity-level and constructive sale findings show awareness of tax complexity but are relatively generic ("this is complex and not addressed").

Key insight — regulatory compliance as a distinct task type:

This experiment tests a fundamentally different cognitive demand than previous ones: previous tasks asked "what could go wrong with this system?" (internal reasoning). This task asks "does this system correctly implement external rules?" (external reasoning). The model must hold TWO bodies of knowledge simultaneously: the implementation spec AND the regulatory framework, then find mismatches.

All three models had strong tax law knowledge — they cited IRC sections, Revenue Rulings, and Treasury Regulations correctly. The differentiation wasn't in legal knowledge but in HOW they applied it:

• GPT-5: Exhaustive statutory mapping ("here's every IRC section that touches wash sales; here's where the implementation falls short on each"). Breadth-first coverage. Found the most issues by sheer scope of regulatory awareness.
• Opus: Operational consequence reasoning ("here's how this gap manifests as a real-world problem for the user/auditor"). Found issues by reasoning about the implementation's interaction with real-world workflows (filing deadlines, form formats, broker reconciliation).
• Sonnet: Category-based analysis ("here are cross-account issues, here are entity issues, here are interaction issues"). Followed the prompt structure closely but didn't go deep within each category.

The per-share vs lot-level finding (GPT-5 #1) — why it matters:

This is the experiment's most important result. Every model found missing features (options, cross-account, short sales) — those are SCOPE limitations that the document itself acknowledges or defers. GPT-5 uniquely found a correctness bug in the IMPLEMENTED logic: the system's lot-level basis adjustment is mathematically wrong for partial wash sales.

Example: Loss lot of 100 shares, replacement lot of 60 shares. Only 60 shares trigger wash sale. System adds full 60% of disallowed loss to the entire replacement lot's basis. If the replacement lot later sells 30 shares, the per-share basis is inflated (reflects 60 shares of adjustment spread across 60 shares). This is actually correct for the replacement lot specifically — but the tacked_opened_at is applied to ALL 60 shares when only the matched shares should have tacked holding periods. For lots where adjusted_quantity < replacement_quantity, the non-matched shares have incorrect holding period characterization.

Actually, on closer inspection: if adjusted_quantity = min(loss_quantity, replacement_quantity), and the system matches 60 shares of a 60-share replacement lot, ALL shares of that lot are matched. The edge case GPT-5 identifies would require a replacement lot larger than the loss — e.g., loss of 60 shares matched against a replacement lot of 100 shares where only 60 are affected. In that case, the tacked_opened_at is set on the entire lot (100 shares) when only 60 should be affected. This IS a genuine bug: 40 shares get incorrect holding period classification.

Updated task-type taxonomy:

Task type	Primary cognitive demand	Best model
Hidden assumptions	Breadth identification (what's not stated?)	GPT-5 (exhaustive)
Race conditions	Sequential temporal reasoning	GPT-5 + Opus
Cross-component interactions	Component boundary reasoning	GPT-5 + Sonnet
Design coherence	Internal consistency checking	Opus
Invariant violation paths	Construction + verification	GPT-5 (precision)
Silent correctness	External requirement matching	Opus
Regulatory compliance	Dual-knowledge-base comparison	GPT-5 (breadth) + Opus (operations)

Regulatory compliance is closest to "silent correctness" (Finding #22) in that both require reasoning about external requirements. The key difference:

• Silent correctness asks "does this produce correct outputs for all inputs?"
• Regulatory compliance asks "does this implement the law correctly?"

Both favor models that reason about the system's relationship to the outside world (Opus's strength), but regulatory compliance also rewards breadth of statutory knowledge (GPT-5's strength). The combination produces the most complete picture.

Practical implication: For regulatory compliance review of financial systems:

• Run GPT-5 for exhaustive statutory coverage (finds the most gaps)
• Run Opus for operational impact analysis (finds how gaps manifest in practice)
• Sonnet adds marginal value — use only if budget allows
• GPT-5's unique strength: identifying correctness bugs in implemented logic (not just missing features)
• Opus's unique strength: identifying timing/workflow issues (year-end, form reporting, reconciliation with broker)