Component: harness/ — batch scanning & benchmarking (dev-only)
Role. Workstation-scale developer tooling: run the scanner across many repos, and measure how accurately it detects known vulnerabilities. Not part of the shipped product — no skill depends on it.
Kind. Python (~13 production modules, ~2.4k LOC + 11 test files). Importable module
local_harness(packagec1-vulnhunter-harness). Shells out toclaude -p /vulnhunt.
Part 1
Two jobs. Batch: you keep a list of repositories and this clones and scans them all, resumably, so you can process a whole portfolio overnight and collect the results in one place. Benchmark: you keep a set of repos where you already know the real vulnerabilities ("ground truth"), and this scans them, has an AI judge compare the scanner's findings to the known bugs, and tallies how many it caught — so you can tell whether a change to the methodology actually made detection better or worse, instead of guessing.
Explain-it-simply. A test track for the scanner: one lane runs it over your whole fleet; the other lane scores it against an answer key.
Data-flow (this component)
flowchart LR
subgraph batch[Batch]
L[(REPO_LIST.txt)] --> BC[clone+scan] --> BR[results/ + collect]
end
subgraph bench[Benchmark]
GT[(ground_truth/*.json)] --> BS[clone+scan] --> J[LLM judge] --> T[tally metrics]
end
Part 2 — Technical detail
Set the scanning/judging model once via the MODEL constant in local_harness/config.py; it
applies across every workflow. State is persisted after each operation, so any run is fully
resumable.
File map
| Path | Purpose |
|---|---|
local_harness/scan.py |
Shared scan engine — runs /vulnhunt on a cloned repo, with retry/backoff, timeouts, worker pool; returns a ScanResult. |
local_harness/clone.py |
Clone a target repo. |
local_harness/config.py |
MODEL, worker counts, retry/backoff, timeouts, skills/phases dirs. |
local_harness/batch/run.py |
CLI: scan (+--resume, --max-workers), status, collect. Reads REPO_LIST.txt. |
local_harness/batch/utils.py |
Batch helpers. |
local_harness/benchmark/run.py |
Full run: clone → scan → judge → tally (--repos, --judge-only, --tally-only, --force-rejudge). |
local_harness/benchmark/judge.py |
LLM-judge: compare findings against ground truth. |
local_harness/benchmark/tally.py |
Compute + render the accuracy metrics report. |
local_harness/benchmark/analyze_misses.py |
Diagnose missed findings (all, or --finding ID). |
local_harness/benchmark/finding_history.py |
Track findings across runs. |
local_harness/benchmark/ground_truth/*.json |
One file per repo; each finding pinned to a public GitHub URL at a commit. Ships only a synthetic EXAMPLE.json (OWASP Juice Shop / WebGoat / NodeGoat) — bring your own corpus. |
Commands
python -m local_harness.batch.run scan [--resume] [--max-workers N]
python -m local_harness.batch.run status | collect
python -m local_harness.benchmark.run [--repos NAME] [--judge-only --force-rejudge] [--tally-only]
python -m local_harness.benchmark.analyze_misses [--finding ID]
Benchmark state lives in local_harness/benchmark_results/state.json.
Part 3 — Inputs, outputs, model selection & deterministic calls
What happens here. The harness is orchestration + measurement, not analysis. It runs the scanner across many repos (batch) or grades it against known answers (benchmark). The only LLM step it owns is the judge in benchmark mode; everything else is deterministic Python.
Inputs. Batch: local_harness/batch/REPO_LIST.txt (GitHub URLs). Benchmark:
ground_truth/<repo>.json (each real finding pinned to a public URL + commit). Both: settings in
local_harness/config.py.
Outputs. Batch: per-repo *_VULNHUNT_RESULTS_* dirs, gathered by collect into to_upload/.
Benchmark: a persisted benchmark_results/state.json and a tallied accuracy report; analyze_misses
diagnostics for gaps.
How the model version is chosen. A single constant — MODEL = "claude-opus-4-8" in
config.py — is used for both the scan and the LLM judge; change it once to switch every
workflow. There is no per-repo or per-finding routing here.
Deterministic (non-LLM) calls. (1) git clone of each target (clone.py). (2) subprocess
claude -p /vulnhunt … — it shells out to the CLI rather than embedding the SDK (scan.py).
(3) A ThreadPoolExecutor (up to MAX_SCAN_WORKERS = 5) with retry/backoff and per-scan
timeouts. (4) JSON parse of ground-truth and results. (5) Tally — deterministic metric
computation and report rendering (tally.py). (6) State persistence after every operation
(fully resumable). The judge (judge.py) is the one LLM call — it compares the scanner's
findings to ground truth using MODEL, with its own retry/backoff (JUDGE_* settings); the scoring
that follows is deterministic.
See also: Scanner · Runtime agent · module index.