Rodin
6af8a6ee10
Break the monolithic 3249-line findings file into 29 individual files, one per experiment. Each file is named YYYY-MM-DD-NN-slug.md for easy chronological sorting and discovery. No content changes — purely structural reorganization.
19 lines
1.0 KiB
Markdown
19 lines
1.0 KiB
Markdown
# Finding 2: Cheap model + narrow lens > expensive model + broad review (one data point)
|
|
|
|
**Date:** 2026-04-26
|
|
**Task:** Check 12 rewritten hypotheses for directional bias
|
|
**How we used them:**
|
|
- Sonnet & GPT-5: full PR review context (diff, file content, issue, AC).
|
|
Broad mandate: "review this PR." Rich context but unfocused task.
|
|
- GPT-4.1 Mini: given ONLY the 12 hypothesis texts + one focused question:
|
|
"Do any of these hypotheses lead toward a predetermined conclusion?"
|
|
Minimal context, laser-focused task. No diff, no project docs, no issue.
|
|
|
|
- Both Sonnet and GPT-5 approved the hypotheses as reviewers
|
|
- GPT-4.1 Mini found ALL 12 pushed toward predetermined conclusions
|
|
- Words like "requires," "necessary," "must be" were flagged as directional
|
|
- **Takeaway:** Task framing mattered more than model size. Rich context +
|
|
broad mandate = missed the forest for the trees. Minimal context + precise
|
|
question = found exactly what mattered. This needs more testing — was it
|
|
the narrow framing, the lack of surrounding context, or both?
|