# Finding: Audit Log Data Integrity Analysis — GPT-5 excels at distributed systems reasoning; Sonnet identifies core issues but lacks depth

**Date:** 2026-05-11
**Task:** Identify data integrity violations in gargoyle's `audit-log.md` (170 lines) — scenarios where the audit log could become inconsistent, lose entries, or fail to be the authoritative record it claims to be.
**New task type:** Data integrity analysis — focused on distributed systems concerns (write ordering, referential integrity, consistency windows, recovery correctness, concurrent access hazards).

**How we used them:** Same document (full text) + same focused analytical question to both models via HAI proxy. Structured prompt with 5 categories and required output format. No tools, no project context beyond the document itself.

| Model | Time | Output tokens | Reasoning tokens | Findings |
|---|---|---|---|---|
| GPT-5 | 134s | 7,274 | 4,416 | 18 (+ 1 design note) |
| Sonnet 4.6 | 26s | 1,792 | (internal) | 13 |

## What they found — common ground (both identified):

- **Portfolio Risk outcome visible before decision record** — replication lag or write ordering can cause PR outcome to appear before the decision exists
- **Multi-aggregator signal processing race** — same signal appears under multiple decisions in arbitrary timestamp order
- **Orphaned decision references** — atomic write fails partway, leaving partial or phantom decision records
- **Missing signal risk rejections** — write failure leaves gap where signal appears to bypass controls
- **Signal expiration race window** — signals marked as "expired" that actually contributed
- **Portfolio Risk evaluation gap** — trades execute but decision shows no risk evaluation for a window
- **Duplicate signal processing** — network retries cause same signal to contribute to multiple decisions
- **Portfolio Risk duplicate evaluation** — message duplication causes conflicting outcomes (approved AND rejected)

## GPT-5 unique findings (not in Sonnet):

1. **Cross-service clock skew** — queries "ordered by time" mix stages incorrectly when clocks drift; SLA measurements and timelines become meaningless
2. **Large atomic batches not truly atomic across partitions** — distributed storage with sharding breaks atomicity guarantee; partial batches visible
3. **Decision_id collisions across aggregators** — without globally unique scheme, different decisions can share IDs; referential integrity collapses
4. **Duplicate PR outcomes from retries** — at-least-once delivery without idempotency creates conflicting terminal states
5. **Correction entries referencing missing entries** — correction chain breaks if original was lost
6. **Expired signal entries with bad signal_ids** — orphan "expired" rows with no other entries
7. **Read-replica lag windows** — transiently incomplete views depending on which replica is hit
8. **Corrections append-only interim truth problem** — queries between error and correction see wrong state
9. **Permanent holes when store unavailable** — no backfill = permanent gaps in "authoritative" record
10. **Approved logged but no order sent** — crash between PR write and OM handoff = factually wrong audit
11. **Aggregator duplicates decision after crash** — input replay creates duplicate or mutated decisions
12. **Conflicting terminal outcomes from concurrent PR paths** — multiple controls race, no finality rule
13. **At-least-once writers without idempotency keys** — duplicates inflate counts and confuse traces
14. **Two aggregators both own same decision** — split-brain creates conflicting decisions for same opportunity
15. **Mixed writers without transactional boundaries** — external Risk writes interleave with DE writes

## Sonnet unique findings (not in GPT-5):

1. **Partial recovery with ID sequence reset** — crash during write + checkpoint restart can cause ID reuse, creating duplicate IDs for different decisions (GPT-5 addressed this via different framing in #14)
2. **Inconsistent recovery state** — signal both rejected AND contributing after replay (GPT-5's #14 is similar but framed differently)
3. **Concurrent decision ID assignment** — ID service race returns same ID to multiple aggregators (GPT-5's #6 is similar)

## Quality assessment:

**GPT-5** was significantly more thorough and demonstrated deeper distributed systems expertise. Key observations:

- Found **18 distinct issues** with detailed sequences and precise impact analysis
- Identified issues Sonnet missed entirely: clock skew, replica lag windows, correction chain integrity, permanent holes semantics, the "approved but not sent" crash window
- Each finding named specific components and described exact interleaving scenarios
- Added a "modeling gap" note about signal_id/decision_id query asymmetry that isn't strictly a violation but creates incomplete narratives
- Output was 4x longer (7,274 vs 1,792 tokens) with substantially more depth per finding

**Sonnet 4.6** identified the core issues but with less depth:

- Found **13 issues** — 72% of GPT-5's count
- Many findings overlapped with GPT-5 but with less precise sequences
- Some findings were near-duplicates of each other under different category headings
- Missed the clock skew, replica lag, correction chain, and permanent-hole issues
- Completed in 26s (5x faster) — useful for quick first-pass but not comprehensive

## Key insight — distributed systems reasoning benefits significantly from reasoning tokens:

This experiment tested a new task type: analyzing an architecture document for distributed systems consistency issues. This requires reasoning about:
- Message ordering across services
- Crash-recovery semantics
- Replication lag visibility windows
- Idempotency and exactly-once delivery
- Atomic write guarantees across storage boundaries

GPT-5's 4,416 reasoning tokens enabled it to trace through complex multi-step scenarios (e.g., "aggregator writes, PR evaluates, replica lags, query hits stale replica" as a 4-step sequence). Sonnet's findings were shallower — it identified the category of issue but often didn't trace through the full causal chain.

This is consistent with Finding #13 (race condition identification) where Sonnet struggled with temporal/sequential reasoning. Distributed systems integrity analysis is essentially "race conditions at architecture scale" — the same cognitive skill that Sonnet lacks at the code level also shows up at the system design level.

## Comparison to previous findings:

| Task type | GPT-5 | Sonnet | Ratio | Notes |
|---|---|---|---|---|
| Assumption-finding (#12) | 20 | 17 | 85% | Sonnet's best relative performance |
| Cross-component interaction (#14) | 10 | 8 | 80% | Structured prompt helped Sonnet |
| Race condition identification (#13) | 12 | 7 | 58% | Sonnet struggled with concurrency |
| **Data integrity analysis** | 18 | 13 | 72% | New task type, between extremes |

Data integrity analysis is between "cross-component interaction" (where Sonnet does well) and "race condition identification" (where Sonnet struggles). The task combines both: understanding component interactions (Sonnet's strength) but also reasoning through temporal/ordering scenarios (Sonnet's weakness).

## Practical implications:

1. **For distributed systems design review:** Use GPT-5. The depth of analysis on issues like "permanent holes," "correction chain integrity," and "replica lag windows" provides genuine value that Sonnet misses.

2. **For quick sanity checks:** Sonnet is viable — it catches the obvious issues (orphaned references, duplicate processing) in 1/5 the time. But don't rely on it for thoroughness.

3. **Task framing helps but doesn't close the gap:** The structured prompt (5 categories, required output format) helped both models produce organized output. But unlike Finding #14 where structure helped Sonnet recover to 80% of GPT-5's count, here structure only got Sonnet to 72%. The task itself is inherently harder for non-reasoning models.

4. **New task type confirmed:** "Data integrity analysis" is a distinct analytical lens useful for architecture review. It complements assumption-finding (what must be true) and race condition analysis (what can interleave) with a focus on what can become inconsistent.

## Cost-effectiveness:

- GPT-5: 134s, ~8.5K total tokens (1.3K prompt + 7.2K completion including 4.4K reasoning)
- Sonnet: 26s, ~3.2K total tokens (1.4K prompt + 1.8K completion)

GPT-5 found 5 issues Sonnet missed entirely. At ~2.7x token cost and 5x time cost, this is justified for architecture review of data-critical systems where consistency violations have financial/regulatory impact.

## Document analyzed:

gargoyle's `docs/domain/contexts/decision-engine/audit-log.md` (170 lines) — describes the Decision Engine's append-only audit trail for signals, decisions, and risk evaluations. Claims to be "authoritative record" with "immutability" guarantees.