finding 59: convention rule gap analysis

New task type: analyzing prescriptive/specification documents for completeness.

- GPT-5 dominates with exhaustive enumeration (34 findings)
- Opus traces gaps to consequences (routing failures, compiler issues)
- Sonnet surface-level (not recommended for thorough analysis)

Key insight: GPT-5 found internal contradiction (telemetry verb rule vs example)
that neither Claude model caught. Opus unique in tracing PubSub collision
to actual routing failure scenario.

Task taxonomy: convention gap analysis follows same pattern as architecture
docs - GPT-5 for coverage, Opus for consequences.

findings/2026-05-09-59-convention-rule-gap-analysis.md

@@ -0,0 +1,117 @@
+# Finding 59: Convention Rule Gap Analysis — GPT-5 Dominates with Exhaustive Enumeration; Opus Finds Design Contradictions; Sonnet Is Surface-Level
+
+**Date:** 2026-05-09
+**Task:** Identify gaps, ambiguities, and internal inconsistencies in gargoyle's `naming-conventions.md` (196 lines) — a prescriptive document defining mechanical naming rules for modules, topics, and metrics.
+**Document type:** Convention/specification document (rules rather than architecture), testing whether models can analyze prescriptive text for completeness.
+
+## Methodology
+
+Same document (full text) + same focused analytical question to all 3 models via HAI AI Core proxy. Prompt specified 5 categories: ambiguous decision points, missing scenarios, internal contradictions, implicit assumptions, and edge cases. Required specific output format with concrete scenarios and severity ratings. No tools, no project context beyond the document.
+
+| Model | Time | Output tokens | Reasoning tokens | Findings |
+|---|---|---|---|---|
+| GPT-5 | 122s | 10,563 | 7,232 | 34 |
+| Claude Opus 4.6 | 69s | 3,898 | (internal) | 18 (12 fully written, 6 cut off) |
+| Claude Sonnet 4.6 | 28s | 1,415 | (internal) | 14 |
+
+## What they found — common ground (all 3 identified)
+
+- `Gargoyle.Accounts` namespace exists "outside bounded contexts" but module rules say "every module belongs to exactly one of seven contexts" — creates ambiguity
+- Behaviour vs struct/schema naming can collide (both use `Gargoyle.<Context>.<Name>`)
+- No guidance for behaviour *implementations* (adapter naming patterns)
+- Sub-namespace creation criteria are subjective ("different lifecycles, failure modes, or team ownership")
+- Process naming "role not action" rule is ambiguous (where's the boundary?)
+- Schema qualification ("when ambiguous, qualify it") lacks canonical qualifier terms
+- Registry naming undefined for multiple registries in same context
+- PubSub per-entity pattern conflicts with per-user pattern (topic collision risk)
+- Telemetry sub-namespace mapping from modules is unclear
+
+## GPT-5 unique findings (not in either Claude model)
+
+- **PubSub subtopic casing/separators undefined** — snake_case vs kebab-case vs plain not specified
+- **PubSub entity key encoding not specified** — colons in keys (BTC-PERP), case sensitivity, composite keys
+- **PubSub tenant/org scope missing** — only user/system/entity defined; no per-org or multi-tenant
+- **Telemetry verb rule contradicts examples** — rule requires past-tense but `stale` appears in examples (not `became_stale`)
+- **Telemetry tag naming unspecified** — key/value casing and naming patterns missing
+- **`_all_users` assumes user is only cross-scope concept** — no guidance for `_all_orgs`, `_all_tenants`
+- **`_all_users` for non-list actions undefined** — what about `count_all_users`, `purge_all_users!`?
+- **Test fake namespace collision risk** — fakes use production namespace (`Gargoyle.*`) in test files
+- **Exceptions/error modules lack conventions** — `OrderRejectedError` vs `OrderRejected` vs `Trading.Error.OrderRejected`
+- **Protocols and Ecto types not covered** — custom `Ecto.Type`, embedded schemas, defimpl placement
+- **Shared struct vs Ecto schema collision** — non-persisted struct named `Order` conflicts with schema
+- **Acronym normalization for topics/events** — `IB` → `ib` vs `i_b` not explicit
+- **Pluralization guidance incomplete** — schemas singular, but sub-namespace pluralization varies (Strategies vs Positioning)
+- **Web component naming gaps** — no guidance for LiveComponents, functional components, .JSON modules
+- **Short-name disambiguation left to reader inference** — "Supervisor" in prose without context is ambiguous
+- **Telemetry noun vs sub-namespace boundary ambiguous** — when is something a sub-namespace segment vs noun?
+- **Telemetry depth/mapping from module nesting unclear** — how many segments for `Trading.BrokerAdapters.IB.OrderRouter`?
+
+## Claude Opus unique findings (not in either other model)
+
+- **PubSub topic pattern collision is CRITICAL, not just ambiguous** — Opus specifically worked through how `"market_data:AAPL"` (per-entity) is indistinguishable from `"market_data:42:status"` (per-user) without the subtopic segment. GPT-5 noted the conflict but didn't trace through the routing failure scenario.
+- **Cross-context data structures have no guidance** — what happens to the `Signal` struct that Engine produces and Trading consumes? GPT-5 focused on module placement; Opus focused on the DDD principle that "same concept has different representations" and whether shared DTOs are allowed.
+- **Test fake location vs namespace mismatch is a compiler issue** — fakes in `test/support/fakes/` with `Gargoyle.*` namespace causes `mix compile` scoping ambiguity and accidental import risk. GPT-5 mentioned collision; Opus traced the consequence.
+- **No rules for implementation modules** — pure helper modules that aren't schemas, processes, or behaviours have no placement guidance. Is `Aggregator.Calculator` a valid name? What about `Internal.AggregatorHelpers`?
+
+## Claude Sonnet findings (less depth than others)
+
+- **Module spanning contexts lacks split guidance** — mentioned but didn't work through specific scenarios
+- **Third-party integration modules unaddressed** — Bloomberg, broker APIs: belong in relevant context or special treatment?
+- **Validation modules at context boundaries** — data validation between Market Data and Decision Engine: which context?
+- **Utility/helper module placement missing** — date helpers, math functions used across contexts
+
+Sonnet's findings were valid but less specific. Several overlapped with GPT-5/Opus without adding the concrete scenario detail that makes findings actionable.
+
+## Quality assessment
+
+- **GPT-5** produced the most exhaustive analysis (34 findings) with deep coverage of every section of the document. It systematically worked through PubSub rules, telemetry rules, `_all_users` functions, web naming, and identified edge cases in each. The telemetry verb/example contradiction finding is a genuine internal inconsistency in the document. The acronym, pluralization, and encoding findings show attention to implementation-level detail. GPT-5 also provided a concrete "Suggestions to clarify" section with remediation proposals.
+
+- **Claude Opus** produced fewer findings (18, with 6 cut off by token limit) but several were qualitatively superior. The PubSub collision finding traced through the routing failure consequence that GPT-5 only hinted at. The cross-context data structure finding engaged with DDD principles ("same concept, different representations") rather than just flagging the gap. Opus consistently reasoned about *why* the gap matters at the architectural level, not just *that* it exists.
+
+- **Claude Sonnet** produced the fewest findings (14) with the least depth. Several findings were restatements of the prompt categories ("no guidance for X") without working through specific collision scenarios. The third-party integration finding and validation module finding were valid but generic — they identify gaps without demonstrating concrete breakage.
+
+## Key insight — convention analysis requires exhaustive enumeration + constraint reasoning
+
+This task had two components:
+1. **Enumeration**: Systematically cover every rule in the document and ask "what's missing?"
+2. **Constraint reasoning**: For each gap, reason about whether two developers following the rules could produce incompatible results
+
+GPT-5 excelled at (1) — its 7,232 reasoning tokens enabled methodical section-by-section coverage. Every major topic (modules, PubSub, telemetry, `_all_users`, fakes, web) got explicit attention.
+
+Opus excelled at (2) — when it identified a gap, it reasoned through the *consequence* (routing failure, compiler ambiguity, architectural drift). Fewer findings, but each finding included the "why it breaks" reasoning that makes it actionable.
+
+Sonnet did neither particularly well — it identified gaps but didn't enumerate exhaustively or reason through consequences deeply.
+
+## Task taxonomy update
+
+**Convention/specification gap analysis** is a new task type with distinct model performance:
+- **GPT-5**: Best for exhaustive coverage — will find the acronym rules, the tag naming, the entity key encoding that others miss
+- **Opus**: Best for consequence reasoning — will trace gaps to architectural or operational failures
+- **Sonnet**: Adequate for quick sanity check; not recommended for thorough analysis
+
+This pattern is consistent with previous findings: GPT-5's reasoning enables systematic enumeration; Opus finds tensions the document can't see about itself; Sonnet lacks depth for analytical work.
+
+## Unique GPT-5 insight worth highlighting
+
+The telemetry verb contradiction (rule says past-tense, example shows `stale`) is a genuine internal inconsistency the document contains. This is different from "gap" or "ambiguity" — it's a self-contradiction where following the rule would produce `became_stale` but the example shows `stale`. GPT-5 caught this; neither Claude model did.
+
+## Unique Opus insight worth highlighting
+
+The PubSub topic collision finding traced through *why* the current rules break: `"market_data:AAPL"` (per-entity) is pattern-indistinguishable from `"market_data:42:status"` (per-user) for a subscriber matching `"market_data:" <> _`. This shows the gap doesn't just allow inconsistency — it allows *silent routing failures*. Opus consistently elevates gaps to their operational consequences.
+
+## Practical implications
+
+For convention/specification review:
+- Run GPT-5 for exhaustive coverage (will catch encoding rules, acronyms, pluralization — the details)
+- Run Opus to identify which gaps have architectural consequences (routing failures, compiler issues)
+- Sonnet is not recommended for this task type
+
+The ideal workflow: GPT-5 first for comprehensive gap list, then Opus to prioritize which gaps matter. This is the same pattern as state machine analysis (Finding #58) and assumption-finding (Finding #10-12).
+
+## Meta-observation: Document type affects model performance less than expected
+
+This is the first experiment on a *prescriptive rules* document rather than an *architecture* document. The model performance ranking (GPT-5 > Opus > Sonnet) held. The task skills (enumeration for GPT-5, consequence reasoning for Opus) also held. This suggests the analytical framework from previous findings generalizes beyond architecture docs.
+
+## Raw data
+
+Full model outputs preserved in `/tmp/gpt5-conventions.json`, `/tmp/opus-conventions.json`, `/tmp/sonnet-conventions.json`.