I Reverse-Engineered Five AI Vulnerability Harnesses. Here’s What Actually Ships in Production.
A senior developer’s read on Project Glasswing’s harness architectures — what I’d steal, what I’d argue about in code review, and where I’d push back.
I’ve spent the last stretch pulling apart five systems built around the same premise: point a frontier model at a codebase, get it to find real vulnerabilities, and don’t drown in false positives doing it. Four are named harnesses I could read source for; one — Cloudflare’s internal pipeline — is documented in enough technical depth across two blog posts that it counts as a sixth data point even without the repo [5][6]. This is the write-up I wish I’d had before starting: what these things actually look like under the hood, where the five teams agree, where they genuinely fight about tradeoffs, and what I’d change if I were shipping one.
Why this exists: Project Glasswing, briefly
Skip this if you already know it. Anthropic announced Project Glasswing in April 2026 after an unreleased model — Claude Mythos Preview — found a 27-year-old OpenBSD crash, a 16-year-old FFmpeg bug that five million fuzzer executions had missed, and a chained Linux kernel privilege escalation, mostly without human steering [7][8]. The response was to get the model into defenders’ hands (AWS, Apple, Cisco, Google, Microsoft, and 40+ more) before offense caught up [7]. That’s the “why now.” The interesting engineering problem starts one layer down: what do you actually build so that “the model found a bug” turns into “the bug got fixed” without burning your whole security team’s week re-verifying every claim?
The thing nobody tells you: a raw agent loop doesn’t work here
If you’ve built anything with an agent SDK, you already know the failure mode. One long-running session, one context window, one hypothesis at a time. Cloudflare’s writeup names it exactly: an hour into an unstructured run, “the context window fills up and the model will cannibalize its own memory, instantly forgetting the bugs it spent all morning tracking down” [6]. I’d add the corollary every one of these five teams hit: a crash mid-run with no external state means losing hours of billable model calls to a rate-limit blip. That’s not a UX complaint, it’s an architecture bug.
The fix, independently arrived at five times, is boring in the best way: externalize state, break the work into narrow single-purpose stages, persist everything to disk or a database keyed by run, and let any stage resume without redoing completed work. Cloudflare keys their SQLite-equivalent state store by (run_id, repo, stage) [5]; Visa’s Vulnerability Agentic Harness (VVAH) does the same with run_manifest.json and a SQLite store outside the scanned repo [1]; Anthropic’s Defending Code Reference Harness (DCRH) resumes by re-checking each result.json’s terminal status [4]. If you’re building one of these and skip persistence to “ship the demo faster,” you will regret it inside a week.
The architecture that actually recurs
Strip the stage names and every one of these five collapses to the same three-part shape:
Finder(s) → adversarial validator → tightly-leashed fixer.
- VVAH: 11 stages (S0–S11) across four phases, config-selectable backend per role (
cli,sdk,openai,deepagents) [1]. - Capital One’s VulnHunter: three Claude Code skills —
/vulnhunt,/vulnhunter-fix,/vulnhunt-fix-verify— no multi-agent framework anywhere in the stack (the one agent dependency, the Claude Agent SDK, lives in a dev-only batch runner) [2]. - STRIX (an independent, open-source autonomous penetration-testing agent): root agent spawns child agents in a shared Docker sandbox, hard-dependent on the OpenAI Agents SDK for the loop primitives [3].
- DCRH:
claude -pinvoked directly as a subprocess, explicitly not through the Agent SDK, because “the Agent SDK is itself a subprocess wrapper around the same CLI; going direct keeps the argv shape under our control” [4] — a design opinion I happen to agree with if you need tight control over resume/retry semantics. - Cloudflare’s Vulnerability Discovery Harness / Vulnerability Validation System (VDH/VVS): an 8-stage producer-consumer loop feeding a separate, shared triage system that other harnesses can also write into [5][6].
If you’re deciding how to structure your own: the finder stage should be tuned to over-report. Every team says this explicitly or implicitly — recall is the finder’s job, precision is the validator’s. Don’t try to make one agent do both; you’ll get a system that’s mediocre at everything.
The design decisions worth stealing
Fresh-container / cross-model validation. DCRH’s grading agent runs in a clean container that only ever sees PoC bytes, never the finder’s chain of reasoning [4]. Cloudflare takes this further and runs discovery and validation on different models entirely, arguing this is the only way to get a genuinely adversarial second opinion instead of a rubber stamp [6]. If you’ve ever had a code reviewer who also wrote the PR, you understand why this matters. Do not let your validator inherit context from your finder.
No shell for post-discovery agents, full stop. VVAH’s remediation and validation stages get Read/Glob/Grep(/Edit/Write) and never Bash — their own reasoning is that a prompt-injected agent with a host shell is RCE on your own scanner [1]. VulnHunter’s verifier has no Bash and no network [2]. This is the single most copy-pasteable security control across all five reviews. If your harness’s job is to process untrusted, attacker-controlled input (which a vulnerability scanner’s target absolutely is), don’t hand the model doing the processing a general-purpose shell.
PoC-or-it-didn’t-happen. Every team independently landed on requiring executable proof before a finding counts — a compiled crash reproduced 3/3 for DCRH [4], a runnable exploit script for STRIX [3], a RED-then-GREEN test pair for VulnHunter’s fixer [2]. Cloudflare’s line on why is the sharpest one I read: “If a Hunter is allowed to grade its own homework, it will confidently validate everything it outputs” [6]. This isn’t paranoia — it’s the same reason you don’t let a PR author self-approve.
Model-agnostic by config, not by accident. VVAH’s models.<role> keys and STRIX’s STRIX_LLM env var both let you swap providers without touching pipeline code [1][3]. Worth building this in from day one even if you’re only using one provider right now — Cloudflare’s stated reason is blunt: providers change temperature, caching, and inference budgets underneath you even within one model version, and you don’t want your whole system’s behavior to silently drift when that happens [6].
Where I’d push back in review
Framework choice is genuinely unsettled, and I don’t think there’s a clean right answer. VulnHunter and DCRH both went zero-framework by design — direct CLI/SDK calls, full control over retry and resume [2][4]. STRIX went the opposite direction and hard-committed to the OpenAI Agents SDK for its multi-agent primitives [3]. VVAH hedges by making the framework itself config-selectable (via: deepagents is opt-in, not mandatory) [1]. If I were starting today I’d lean toward VulnHunter/DCRH’s position for anything where I need exact control over what gets retried and how — but I’d concede the SDK route saves real implementation time on multi-agent coordination if that’s not your team’s core competency.
“Cost per finding” is still a soft metric everywhere, including Cloudflare’s numbers. Their funnel math is genuinely impressive — 20,799 raw candidates down to 7,245 actionable findings [6] — but note what’s not published anywhere across all five: a false-negative rate. Cloudflare says this outright: “there’s no labeled set of every real bug in a codebase, so any claimed recall number is entirely speculative” [6]. VulnHunter’s benchmark harness is the closest thing to an answer — it scores detection accuracy against a known-vulnerable ground-truth corpus [2] — but that’s measuring against known bugs, which tells you something different from “how many bugs did we miss.”
None of these handle the fix-lives-somewhere-else problem, and it’s a real gap, not a nitpick. Every harness reviewed assumes the operator owns the merge button. VVAH scans Visa’s own repos. VulnHunter’s fixer opens a PR against a branch the operator controls [2]. DCRH’s targets are its own demo apps [4]. That’s fine until the harness correctly flags a bug in a transitive open-source dependency three hops upstream — at which point there’s no PR to open, because you don’t own that repo, and possibly no maintainer actively watching it either. Roughly a quarter of npm’s transitively-affected library versions stay vulnerable after a fix exists upstream just because the fix hasn’t propagated down the graph yet [19]. If you’re building one of these for anything beyond first-party code, budget real design time for “found it, can’t merge it” as a first-class outcome, not an edge case.
There’s no interchange format, and everyone is reinventing evidence packages. SARIF (Static Analysis Results Interchange Format) is the OASIS standard that lets traditional static-analysis tools hand results to CI pipelines and ticketing systems in one common shape — GitHub code scanning ingests it natively [25]. Nothing like that exists for agentic findings: DCRH ships PoC bytes plus a 3-of-3 crash reproduction [4], VulnHunter ships a PoC plus a RED→GREEN test pair [2], STRIX ships an exploit script plus captured evidence [3], Cloudflare ships a validation verdict into its own shared triage pool [6] — four incompatible evidence formats for what’s structurally the same claim. A SARIF-for-agentic-findings would need attached PoC artifacts, model-provenance metadata, falsification history, and stable dedup keys, at minimum. Build one of these harnesses today and you’re not just solving discovery and validation — you’re inventing your own private wire format too, and that’s the same underlying gap that leaves the fix-lives-somewhere-else problem above unsolved: there’s no shared substrate for findings to travel across a boundary the harness itself doesn’t control.
You’re building a system that runs untrusted code and stores exploits — threat-model it like one. This is the part that bites teams who copy the architecture without the authors’ security instincts. Walk the modules: the finder eats attacker-controlled input (the target repo), so target content must never hit a system-prompt slot or pick your model endpoint — VVAH refusing an in-repo config.yaml [1] and DCRH’s <untrusted_data> wrapping [4] are the pattern, and DCRH still calls the wrapping a mitigation, not a guarantee. The validator/sandbox runs hostile code by design: gVisor + allowlist-proxy egress (DCRH [4]) is the bar; plain Docker is a softer boundary and a shared sandbox (STRIX [3]) lets one poisoned target touch sibling runs — go per-run-disposable, default-deny egress. The fixer is why nobody grants it Bash: a prompt-injected agent with a shell is RCE on your scanner [1], so branch-only writes + human merge gate, always. And the module nobody writes up: the findings store is a database of working PoCs for unpatched bugs in your own systems — VVAH keeps SQLite state outside the scanned repo [1], but none of the docs specify encrypting it or scoping access, and if it leaks you’ve handed someone a pre-validated target list. Encrypt it, lock it down, set retention. Same for the credential/routing config: least-privilege short-lived repo tokens, pinned model endpoints, secrets a compromised stage can’t read. Copy the architecture and you inherit all of this attack surface for free.
The takeaway for anyone building one of these
Five teams, independently: separate the finder from the grader, demand executable proof, never give the fixer a shell, make the model swappable. That’s not five coincidences — it’s what the failure modes force you toward once you actually run this at scale instead of in a demo. Build for that from the start and you’ll save yourself the six-week rebuild cycle at least two of these teams describe going through to get there.
References
- Visa, Visa Vulnerability Agentic Harness (VVAH), github.com/visa/visa-vulnerability-agentic-harness — open-sourced June 10, 2026 under the Apache License 2.0 (© 2026 Visa, Inc.). Reviewed design documentation.
- Capital One, VulnHunter, github.com/capitalone/VulnHunter. Reviewed design documentation.
- STRIX, open-source autonomous penetration-testing agent — architecture and design documentation reviewed for this piece. Reviewed design documentation.
- Anthropic, Defending Code Reference Harness (DCRH), github.com/anthropics/defending-code-reference-harness. Reviewed design documentation.
- Cloudflare, “Project Glasswing: what Mythos showed us,” The Cloudflare Blog, May 18, 2026.
- Cloudflare, “Build your own vulnerability harness,” The Cloudflare Blog, June 18, 2026.
- Anthropic, “Project Glasswing: Securing critical software for the AI era,” April 7, 2026.
- Anthropic Frontier Red Team, “Mythos Preview,” cited via [7].
- “Original Sin of npm: A Study on Vulnerability Propagation in JavaScript Dependency Networks,” arXiv.
- OASIS, Static Analysis Results Interchange Format (SARIF) v2.1.0, docs.oasis-open.org/sarif/sarif/v2.1.0.
Full source list, including workforce and open-source-remediation data, in the companion piece: “Inside Project Glasswing: How Five Independent Teams Built the Same Machine.”