← Back to Vulnerability Identification Harness Reviews indexSr. Developer voice

I Reverse-Engineered Five AI Vulnerability Harnesses. Here’s What Actually Ships in Production.

A senior developer’s read on Project Glasswing’s harness architectures — what I’d steal, what I’d argue about in code review, and where I’d push back.

The shared pattern behind five independent harnesses

I’ve spent the last stretch pulling apart five systems built around the same premise: point a frontier model at a codebase, get it to find real vulnerabilities, and don’t drown in false positives doing it. Four are named harnesses I could read source for; one — Cloudflare’s internal pipeline — is documented in enough technical depth across two blog posts that it counts as a sixth data point even without the repo [5][6]. This is the write-up I wish I’d had before starting: what these things actually look like under the hood, where the five teams agree, where they genuinely fight about tradeoffs, and what I’d change if I were shipping one.

Why this exists: Project Glasswing, briefly

Skip this if you already know it. Anthropic announced Project Glasswing in April 2026 after an unreleased model — Claude Mythos Preview — found a 27-year-old OpenBSD crash, a 16-year-old FFmpeg bug that five million fuzzer executions had missed, and a chained Linux kernel privilege escalation, mostly without human steering [7][8]. The response was to get the model into defenders’ hands (AWS, Apple, Cisco, Google, Microsoft, and 40+ more) before offense caught up [7]. That’s the “why now.” The interesting engineering problem starts one layer down: what do you actually build so that “the model found a bug” turns into “the bug got fixed” without burning your whole security team’s week re-verifying every claim?

The thing nobody tells you: a raw agent loop doesn’t work here

If you’ve built anything with an agent SDK, you already know the failure mode. One long-running session, one context window, one hypothesis at a time. Cloudflare’s writeup names it exactly: an hour into an unstructured run, “the context window fills up and the model will cannibalize its own memory, instantly forgetting the bugs it spent all morning tracking down” [6]. I’d add the corollary every one of these five teams hit: a crash mid-run with no external state means losing hours of billable model calls to a rate-limit blip. That’s not a UX complaint, it’s an architecture bug.

The fix, independently arrived at five times, is boring in the best way: externalize state, break the work into narrow single-purpose stages, persist everything to disk or a database keyed by run, and let any stage resume without redoing completed work. Cloudflare keys their SQLite-equivalent state store by (run_id, repo, stage) [5]; Visa’s Vulnerability Agentic Harness (VVAH) does the same with run_manifest.json and a SQLite store outside the scanned repo [1]; Anthropic’s Defending Code Reference Harness (DCRH) resumes by re-checking each result.json’s terminal status [4]. If you’re building one of these and skip persistence to “ship the demo faster,” you will regret it inside a week.

The architecture that actually recurs

Strip the stage names and every one of these five collapses to the same three-part shape:

Finder(s) → adversarial validator → tightly-leashed fixer.

If you’re deciding how to structure your own: the finder stage should be tuned to over-report. Every team says this explicitly or implicitly — recall is the finder’s job, precision is the validator’s. Don’t try to make one agent do both; you’ll get a system that’s mediocre at everything.

The design decisions worth stealing

Fresh-container / cross-model validation. DCRH’s grading agent runs in a clean container that only ever sees PoC bytes, never the finder’s chain of reasoning [4]. Cloudflare takes this further and runs discovery and validation on different models entirely, arguing this is the only way to get a genuinely adversarial second opinion instead of a rubber stamp [6]. If you’ve ever had a code reviewer who also wrote the PR, you understand why this matters. Do not let your validator inherit context from your finder.

No shell for post-discovery agents, full stop. VVAH’s remediation and validation stages get Read/Glob/Grep(/Edit/Write) and never Bash — their own reasoning is that a prompt-injected agent with a host shell is RCE on your own scanner [1]. VulnHunter’s verifier has no Bash and no network [2]. This is the single most copy-pasteable security control across all five reviews. If your harness’s job is to process untrusted, attacker-controlled input (which a vulnerability scanner’s target absolutely is), don’t hand the model doing the processing a general-purpose shell.

PoC-or-it-didn’t-happen. Every team independently landed on requiring executable proof before a finding counts — a compiled crash reproduced 3/3 for DCRH [4], a runnable exploit script for STRIX [3], a RED-then-GREEN test pair for VulnHunter’s fixer [2]. Cloudflare’s line on why is the sharpest one I read: “If a Hunter is allowed to grade its own homework, it will confidently validate everything it outputs” [6]. This isn’t paranoia — it’s the same reason you don’t let a PR author self-approve.

Model-agnostic by config, not by accident. VVAH’s models.<role> keys and STRIX’s STRIX_LLM env var both let you swap providers without touching pipeline code [1][3]. Worth building this in from day one even if you’re only using one provider right now — Cloudflare’s stated reason is blunt: providers change temperature, caching, and inference budgets underneath you even within one model version, and you don’t want your whole system’s behavior to silently drift when that happens [6].

Where I’d push back in review

Framework choice is genuinely unsettled, and I don’t think there’s a clean right answer. VulnHunter and DCRH both went zero-framework by design — direct CLI/SDK calls, full control over retry and resume [2][4]. STRIX went the opposite direction and hard-committed to the OpenAI Agents SDK for its multi-agent primitives [3]. VVAH hedges by making the framework itself config-selectable (via: deepagents is opt-in, not mandatory) [1]. If I were starting today I’d lean toward VulnHunter/DCRH’s position for anything where I need exact control over what gets retried and how — but I’d concede the SDK route saves real implementation time on multi-agent coordination if that’s not your team’s core competency.

“Cost per finding” is still a soft metric everywhere, including Cloudflare’s numbers. Their funnel math is genuinely impressive — 20,799 raw candidates down to 7,245 actionable findings [6] — but note what’s not published anywhere across all five: a false-negative rate. Cloudflare says this outright: “there’s no labeled set of every real bug in a codebase, so any claimed recall number is entirely speculative” [6]. VulnHunter’s benchmark harness is the closest thing to an answer — it scores detection accuracy against a known-vulnerable ground-truth corpus [2] — but that’s measuring against known bugs, which tells you something different from “how many bugs did we miss.”

None of these handle the fix-lives-somewhere-else problem, and it’s a real gap, not a nitpick. Every harness reviewed assumes the operator owns the merge button. VVAH scans Visa’s own repos. VulnHunter’s fixer opens a PR against a branch the operator controls [2]. DCRH’s targets are its own demo apps [4]. That’s fine until the harness correctly flags a bug in a transitive open-source dependency three hops upstream — at which point there’s no PR to open, because you don’t own that repo, and possibly no maintainer actively watching it either. Roughly a quarter of npm’s transitively-affected library versions stay vulnerable after a fix exists upstream just because the fix hasn’t propagated down the graph yet [19]. If you’re building one of these for anything beyond first-party code, budget real design time for “found it, can’t merge it” as a first-class outcome, not an edge case.

There’s no interchange format, and everyone is reinventing evidence packages. SARIF (Static Analysis Results Interchange Format) is the OASIS standard that lets traditional static-analysis tools hand results to CI pipelines and ticketing systems in one common shape — GitHub code scanning ingests it natively [25]. Nothing like that exists for agentic findings: DCRH ships PoC bytes plus a 3-of-3 crash reproduction [4], VulnHunter ships a PoC plus a RED→GREEN test pair [2], STRIX ships an exploit script plus captured evidence [3], Cloudflare ships a validation verdict into its own shared triage pool [6] — four incompatible evidence formats for what’s structurally the same claim. A SARIF-for-agentic-findings would need attached PoC artifacts, model-provenance metadata, falsification history, and stable dedup keys, at minimum. Build one of these harnesses today and you’re not just solving discovery and validation — you’re inventing your own private wire format too, and that’s the same underlying gap that leaves the fix-lives-somewhere-else problem above unsolved: there’s no shared substrate for findings to travel across a boundary the harness itself doesn’t control.

You’re building a system that runs untrusted code and stores exploits — threat-model it like one. This is the part that bites teams who copy the architecture without the authors’ security instincts. Walk the modules: the finder eats attacker-controlled input (the target repo), so target content must never hit a system-prompt slot or pick your model endpoint — VVAH refusing an in-repo config.yaml [1] and DCRH’s <untrusted_data> wrapping [4] are the pattern, and DCRH still calls the wrapping a mitigation, not a guarantee. The validator/sandbox runs hostile code by design: gVisor + allowlist-proxy egress (DCRH [4]) is the bar; plain Docker is a softer boundary and a shared sandbox (STRIX [3]) lets one poisoned target touch sibling runs — go per-run-disposable, default-deny egress. The fixer is why nobody grants it Bash: a prompt-injected agent with a shell is RCE on your scanner [1], so branch-only writes + human merge gate, always. And the module nobody writes up: the findings store is a database of working PoCs for unpatched bugs in your own systems — VVAH keeps SQLite state outside the scanned repo [1], but none of the docs specify encrypting it or scoping access, and if it leaks you’ve handed someone a pre-validated target list. Encrypt it, lock it down, set retention. Same for the credential/routing config: least-privilege short-lived repo tokens, pinned model endpoints, secrets a compromised stage can’t read. Copy the architecture and you inherit all of this attack surface for free.

The takeaway for anyone building one of these

Five teams, independently: separate the finder from the grader, demand executable proof, never give the fixer a shell, make the model swappable. That’s not five coincidences — it’s what the failure modes force you toward once you actually run this at scale instead of in a demo. Build for that from the start and you’ll save yourself the six-week rebuild cycle at least two of these teams describe going through to get there.


References

  1. Visa, Visa Vulnerability Agentic Harness (VVAH), github.com/visa/visa-vulnerability-agentic-harness — open-sourced June 10, 2026 under the Apache License 2.0 (© 2026 Visa, Inc.). Reviewed design documentation.
  2. Capital One, VulnHunter, github.com/capitalone/VulnHunter. Reviewed design documentation.
  3. STRIX, open-source autonomous penetration-testing agent — architecture and design documentation reviewed for this piece. Reviewed design documentation.
  4. Anthropic, Defending Code Reference Harness (DCRH), github.com/anthropics/defending-code-reference-harness. Reviewed design documentation.
  5. Cloudflare, “Project Glasswing: what Mythos showed us,” The Cloudflare Blog, May 18, 2026.
  6. Cloudflare, “Build your own vulnerability harness,” The Cloudflare Blog, June 18, 2026.
  7. Anthropic, “Project Glasswing: Securing critical software for the AI era,” April 7, 2026.
  8. Anthropic Frontier Red Team, “Mythos Preview,” cited via [7].
  9. “Original Sin of npm: A Study on Vulnerability Propagation in JavaScript Dependency Networks,” arXiv.
  10. OASIS, Static Analysis Results Interchange Format (SARIF) v2.1.0, docs.oasis-open.org/sarif/sarif/v2.1.0.

Full source list, including workforce and open-source-remediation data, in the companion piece: “Inside Project Glasswing: How Five Independent Teams Built the Same Machine.”