claw
0c632c255a
Tested open question from Finding #5: does narrow framing give Sonnet GPT-5-level semantic analysis? Result: NO. Narrow framing changes WHAT Sonnet looks for (redirects from gaps to contradictions) but not HOW WELL it reasons. Sonnet narrow found 3 contradictions but only 1 was genuine (2 were analytical errors/misreads). GPT-5 found 4 all-genuine findings with precise reasoning. Key insight: framing controls scope, not reasoning depth. For tasks requiring logical verification (contradictions, race conditions, invariant violations), reasoning tokens are necessary — framing alone is insufficient. Updated open-questions.md: marked Sonnet+narrow as answered, added new question about Opus+narrow for contradiction detection.
77 lines
3.8 KiB
Markdown
77 lines
3.8 KiB
Markdown
# Open Questions
|
|
|
|
Unanswered questions from experiments, ordered by potential impact.
|
|
|
|
## High Priority
|
|
|
|
### Signal-to-noise confirmation (from Finding #8)
|
|
Give a model the FULL PR review context (diff, files, issue, AC) but add
|
|
the narrow bias question as an explicit review checklist item. If the model
|
|
catches bias despite the rich context, it confirms the signal-to-noise
|
|
hypothesis. If it misses, it suggests something else (attention allocation,
|
|
task switching cost).
|
|
|
|
### Cross-document consistency as maintenance tool (from Finding #28)
|
|
Does running cross-doc analysis across MORE document pairs (domain readmes
|
|
vs implementation docs, design docs vs plan docs) yield additional real
|
|
inconsistencies? Could become a systematic documentation maintenance tool.
|
|
|
|
### Why Opus dominates cross-doc consistency (from Finding #28)
|
|
Opus was 2.4x faster AND found more issues than GPT-5. Is this because
|
|
cross-doc contradictions are easy to verify once spotted (reducing GPT-5's
|
|
verification advantage)? Or because boundary reasoning (Opus's strength)
|
|
is the primary skill needed?
|
|
|
|
### Opus + narrow framing for contradiction detection (from Finding #39)
|
|
Would Opus + narrow framing match GPT-5 for self-contradiction detection?
|
|
Finding #39 showed Sonnet can't do it even with narrow framing (reasoning
|
|
depth issue). Opus has strong cross-boundary reasoning — does its internal
|
|
reasoning depth suffice for the verification step that Sonnet lacks?
|
|
|
|
### ~~Sonnet + narrow framing = GPT-5 level? (from Finding #5)~~ → ANSWERED (Finding #39)
|
|
~~Would Sonnet catch semantic issues if given a narrower "check for logical
|
|
consistency" framing instead of broad review? The hypothesis: Sonnet's
|
|
"structural reviewer" tendency is a framing artifact, not a capability limit.~~
|
|
|
|
**NO.** Narrow framing changes WHAT Sonnet looks for (redirects from gaps to
|
|
contradictions) but not HOW WELL it reasons. Sonnet narrow found 3 contradictions
|
|
but only 1 was genuine (2 were misreadings). GPT-5 found 4 all-genuine findings.
|
|
The gap is reasoning depth, not framing — Sonnet can't reliably verify whether
|
|
two statements actually contradict each other.
|
|
|
|
## Medium Priority
|
|
|
|
### ~~Adversarial analysis ensemble (from Finding #29)~~ → ANSWERED (Finding #35)
|
|
~~Run GPT-5 and Opus sequentially — give Opus access to GPT-5's findings
|
|
and ask it to critique and extend. Does the ensemble find more than either
|
|
alone? Does Opus's system-level thinking complement GPT-5's exhaustiveness?~~
|
|
|
|
**YES.** Ensemble produces 56 findings vs 43 (GPT-5) or 28 (Opus) alone (30%
|
|
improvement). Zero full disagreements — critique phase calibrates severity
|
|
without discarding. Extension phase adds 13 genuinely new findings (4 High).
|
|
The critique's structured assessment is more valuable than raw extensions.
|
|
Cost: ~28% more tokens for 30% more coverage + prioritization.
|
|
|
|
### Reasoning effort parameter (from Finding #21)
|
|
Reasoning effort (low/medium/high) had negligible effect on GPT-5's
|
|
analytical output. Is this because the parameter doesn't work for open-ended
|
|
analysis? Or because the task was already within GPT-5's "easy" threshold?
|
|
Test with a harder document.
|
|
|
|
### Model personality vs prompt (from Finding #26)
|
|
Missing-feature identification IS promptable across all models — prompt
|
|
framing eliminates Opus's historical advantage. How many other "model
|
|
personality" observations are actually just prompt framing effects?
|
|
|
|
## Answered Questions
|
|
|
|
- ~~Opus's "missing feature identification" mode — is it promptable?~~
|
|
**YES** (Finding #26): all models find regulatory gaps when explicitly
|
|
prompted. Opus's behavior was an emergent DEFAULT tendency, not a unique
|
|
capability.
|
|
|
|
- ~~Is Opus > GPT-5 for coherence tasks universal?~~
|
|
**NO** (Finding #27): Opus's advantage from Finding #15 was document-
|
|
specific. On risk-controls.md (992 lines, more complex), GPT-5 regained
|
|
top position. Document complexity and domain specialization affect ranking.
|