model-research/findings/2026-05-07-38-regulatory-compliance-gap-analysis.md

# Finding #38: Regulatory Compliance Gap Analysis

**Date:** 2026-05-07
**Document:** `docs/impl/dtbp-margin-call.md` (363 lines)
**Task type:** Domain-specific regulatory knowledge test (FINRA/SEC PDT rules)
**Models:** GPT-5, Claude Opus 4.6, Claude Sonnet 4.6

## Experiment Design

First experiment testing **domain-specific regulatory knowledge** rather than pure
architectural reasoning. Asked models to identify where the implementation design
might violate or inadequately handle actual FINRA/SEC regulatory requirements
around Pattern Day Trader (PDT) rules and margin calls.

Prompt specified 5 categories:
1. Regulatory gaps (FINRA/SEC PDT rules, Reg T requirements)
2. Broker semantic mismatches (API field meanings under real conditions)
3. Temporal edge cases (market boundaries, holidays, early closes)
4. State machine incompleteness (missing states/transitions)
5. Calculation correctness (DTBP arithmetic under specific order patterns)

## Results

| Model | Time | Output tokens | Reasoning tokens | Findings |
|---|---|---|---|---|
| GPT-5 | 155s | 11,734 | 9,024 | 13+ (cut off by content filter) |
| Claude Opus 4.6 | 117s | 5,049 | (internal) | 15 |
| Claude Sonnet 4.6 | ~39s | 1,938 | (internal) | 12 |

**NOTE:** GPT-5's response was terminated by the SAP HAI proxy's content safety
filter (financial/trading content triggered it), cutting off mid-finding #13.

## Common Ground (all 3 identified)

- Short sale DTBP consumption not tracked (buy-only accumulation)
- Options assignment creating untracked DTBP consumption
- Market close/open boundary timing issues
- Margin call detection relying solely on DTBP numeric comparison
- 5-day cure period calendar computation edge cases

## GPT-5 Unique Findings

- `account.buying_power` already being 2× from broker → system double-multiplies
  to 4× in overnight mode (concrete implementation bug)
- After-hours trades consuming DTBP that resets at 4pm (dtbp_used_today reset too
  early for same-day extended session)
- Premarket DTBP enforcement gap (broker enforces DTBP in extended hours but system
  uses 2× overnight mode pre-open)
- House/concentration surcharges consuming DTBP faster than notional cost
- GTC orders executing after-hours at 4× sizing while system is in 2× overnight
- FIFO/LIFO matching ambiguity for partial sell DTBP release

## Claude Opus Unique Findings

- **PDT designation trigger gap:** System passively reads PDT status but doesn't
  preemptively gate the 4th day trade that CAUSES designation; $25k equity not
  verified before triggering trade
- **90-day freeze allows day trades:** Design restricts to 1× buying power but
  FINRA actually PROHIBITS the activity entirely during escalation (not just
  restricts leverage) — a genuine regulatory violation
- **Margin call issuance date recovery:** If pipeline is down when call is issued,
  system sets issued_at to detection time, not actual issuance → extends cure
  period beyond regulatory 5 days
- **Time-and-tick accounting requirement:** FINRA requires tracking maximum open
  commitment (high water mark) for DTBP, not net basis — the release logic may
  violate this
- **Multiple concurrent margin calls:** Second call upserts over first, losing the
  earlier deadline (single-state-per-user model inadequate)
- **dtbp_used_today NOT reset in margin call mode:** Close sequence guard
  (`bp_mode != :margin_dtbp`) skips reset, causing stale accumulation
- **Cash account free-riding 90-day freeze:** Broader Reg T scope not modeled
- **Broker re-query race on rapid fills:** Response ordering creates stale DTBP
  window between consecutive fills

## Claude Sonnet Unique Findings

- PDT designation timing mismatch (Gargoyle vs broker overnight batch)
- Wash sale impact on maintenance requirements affecting DTBP (IRS interaction)

## Key Insights

### 1. Regulatory domain expertise varies significantly across models

- **Opus has deepest regulatory knowledge.** Cited specific FINRA Rule 4210
  subsections, understood the distinction between restricting leverage vs
  prohibiting activity, and knew about time-and-tick DTBP accounting.
- **GPT-5 has deepest broker-API semantic knowledge.** Reasoned about what
  specific broker API fields actually mean vs what the design assumes
  (buying_power already being 2×, DTBP in extended hours, house surcharges).
- **Sonnet is competent but surface-level.** Good coverage for a first pass
  but doesn't match regulatory depth of Opus or semantic precision of GPT-5.

### 2. Domain-specific lens changes model ranking

In general assumption-finding (previous experiments):
- GPT-5 > Sonnet > Opus (by count)
- Opus > GPT-5 > Sonnet (by insight per finding)

In regulatory compliance analysis:
- Opus > GPT-5 > Sonnet (by regulatory significance)
- GPT-5 > Opus > Sonnet (by broker-semantic precision)

The regulatory lens ELEVATED Opus because it triggered domain-specific
knowledge that Opus possesses more deeply than the other models.

### 3. Content filters as a new failure mode

Enterprise AI proxies may filter financial/regulatory analytical content.
GPT-5's response was cut off by content safety — a failure mode not seen
in architectural analysis. For production regulatory compliance review,
use direct API access or configure filters for analytical discourse.

## Practical Implications

For systems with regulatory requirements (finance, healthcare, legal):
- **Run Opus for regulatory compliance analysis** — its domain knowledge
  produces findings other models won't surface
- **Combine with GPT-5 for implementation semantics** — what does this API
  field actually mean in practice?
- **Sonnet for fast first-pass** but not sole reviewer for regulatory matters
- **Direct API access for financial domain** — enterprise proxy content
  filters may interfere

## Comparison to Previous Experiments

This extends the finding from #11 and #13 that task type changes model
performance. Here we show that task DOMAIN also matters. A model's strength
on architectural reasoning doesn't predict its strength on regulatory
reasoning. The optimal model assignment depends on both:
- Task type (assumptions vs races vs compliance)
- Task domain (architecture vs regulation vs security)