rodin/model-research
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

finding #48: defense-in-depth gap analysis on auth-and-credentials.md

Browse Source 
New analytical lens: where systems rely on single mechanisms rather than
layered defenses. GPT-5 finds exploitable SSRF; Opus identifies trust-root
collapse (session+sudo share SECRET_KEY_BASE); Sonnet is surface-level.
This commit is contained in:
claw
2026-05-08 03:47:09 -07:00
parent f3266ccc13
commit 8f9e87415e
1 changed files with 86 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
findings/2026-05-08-48-defense-in-depth-gap-analysis.md
+86
View File
@@ -0,0 +1,86 @@
# Finding #48: Defense-in-Depth Gap Analysis
**Date:** 2026-05-08
**Document:** gargoyle's `auth-and-credentials.md` (209 lines)
**Analytical lens:** Defense-in-depth gaps — where the system relies on a SINGLE mechanism to prevent catastrophic outcomes rather than layered independent defenses.
**Models:** GPT-5, Claude Opus 4.6, Claude 4 Sonnet
## Setup
Same document (full text, 8KB) + same focused analytical prompt to all 3 models via HAI proxy. Structured prompt specifying 5 focus areas:
1. Single points of failure where one component crash/bug exposes secrets or grants unauthorized access
2. Missing rate limiting, monitoring, or alerting that would detect exploitation
3. Single-check authorization without defense-in-depth
4. Encryption with single-key dependency (no key escrow, HSM, or rotation safety net)
5. Session/token security relying on one mechanism with no revocation fallback
Required structured output per finding (protected asset, single mechanism, bypass scenario, missing layers, severity).
## Results
| Model | Time | Output tokens | Reasoning tokens | Findings |
|---|---|---|---|---|
| GPT-5 | 87.9s | 8,077 | 5,952 | 10 |
| Claude Opus 4.6 | 59.4s | 2,371 | (internal) | 7 |
| Claude 4 Sonnet | 26.2s | 1,161 | (internal) | 6 |
## Common Ground (all 3 identified)
- Single encryption key as catastrophic single point of failure
- Session token lacks revocation on password change
- Scope-based credential access with no secondary authorization check
- Admin role enforcement relying on a single role field
- Invite token with no rate limiting or brute-force detection
## GPT-5 Unique Findings
- **SSRF via user-controlled base_url/data_url:** "Test connection" makes server-side HTTP requests to user-supplied URLs with no allowlist. Genuine exploitable vulnerability.
- **Audit/telemetry integrity gap:** No tamper protection, no external sink, no hash chains.
- **Session token storage format:** Document doesn't confirm tokens are hashed at rest.
- **Fragile key rotation procedure:** Reliance on manual operator discipline.
- **Bearer session with no posture checks:** No device binding, geo-velocity, or reuse detection.
## Claude Opus Unique Findings
- **Trust-root collapse in sudo + session:** Both session token integrity AND sudo timestamp depend on the SAME trust root (SECRET_KEY_BASE). What appears to be defense-in-depth is actually a single mechanism dressed as two. **Most architecturally insightful finding across all models.**
- **No credential kill switch:** No bulk revocation, no Vault "seal" operation, no mechanism to halt decryption during incident response.
- **Automatic Cloak Ecto decryption as hazard:** Any code path returning the struct exposes plaintext — no decrypt-on-demand pattern.
## Claude 4 Sonnet Unique Findings
- **Test connection credential exposure:** Focused on transit/logging risk during credential testing (different angle than GPT-5's SSRF — Sonnet sees credential exposure while GPT-5 sees network probing).
## Key Insights
### Defense-in-depth as a distinct cognitive task
This lens requires: identifying what APPEARS to be protected → asking "what if the ONE mechanism fails?" → identifying where layers COLLAPSE into single points. It's fundamentally about **architectural trust analysis**.
| Analytical lens | Cognitive mode |
|---|---|
| Assumption-finding | "What must be true?" (identification) |
| Race conditions | "What ordering can break?" (temporal reasoning) |
| Invariant violation | "What legal sequence violates?" (construction + verification) |
| **Defense-in-depth** | "Where do layers collapse?" (trust relationship analysis) |
### Opus excels at trust-root analysis
Opus's trust-root collapse finding is the most architecturally significant because it identifies that apparent defense-in-depth is illusory. Session + sudo LOOK like two layers but share SECRET_KEY_BASE — compromise one, compromise both. This is exactly the kind of "design's relationship to itself" reasoning Opus consistently excels at.
### GPT-5's security breadth
GPT-5 found the only genuine exploitable vulnerability (SSRF) and covered the broadest attack surface: crypto, session, SSRF, audit, storage format, and operational procedure. Its remediation suggestions are operationally mature (KMS, egress proxy, refresh-token families, geovelocity).
### Claude 4 Sonnet positioning
Adequate but surface-level. Catches obvious gaps but won't surprise a security reviewer. Similar positioning to GPT-4.1 in earlier experiments — a quick sanity check, not deep analysis.
## Practical Implications
For security architecture review:
- **GPT-5** for breadth — finds exploitable vulnerabilities and operational gaps
- **Opus** for trust analysis — finds where apparent layering is illusory
- **Sonnet** for quick sanity check — catches obvious gaps cheaply
The defense-in-depth lens is particularly well-suited to Opus's analytical style because it's fundamentally about structural relationships between protection mechanisms.