Component: harness/ — batch & benchmark

Workstation-scale batch scanning and detection-accuracy benchmarking (dev-only).

Component: harness/ — batch scanning & benchmarking (dev-only)

Role. Workstation-scale developer tooling: run the scanner across many repos, and measure how accurately it detects known vulnerabilities. Not part of the shipped product — no skill depends on it.

Kind. Python (~13 production modules, ~2.4k LOC + 11 test files). Importable module local_harness (package c1-vulnhunter-harness). Shells out to claude -p /vulnhunt.


Part 1

Two jobs. Batch: you keep a list of repositories and this clones and scans them all, resumably, so you can process a whole portfolio overnight and collect the results in one place. Benchmark: you keep a set of repos where you already know the real vulnerabilities ("ground truth"), and this scans them, has an AI judge compare the scanner's findings to the known bugs, and tallies how many it caught — so you can tell whether a change to the methodology actually made detection better or worse, instead of guessing.

Explain-it-simply. A test track for the scanner: one lane runs it over your whole fleet; the other lane scores it against an answer key.

Data-flow (this component)

flowchart LR
  subgraph batch[Batch]
    L[(REPO_LIST.txt)] --> BC[clone+scan] --> BR[results/ + collect]
  end
  subgraph bench[Benchmark]
    GT[(ground_truth/*.json)] --> BS[clone+scan] --> J[LLM judge] --> T[tally metrics]
  end

Part 2 — Technical detail

Set the scanning/judging model once via the MODEL constant in local_harness/config.py; it applies across every workflow. State is persisted after each operation, so any run is fully resumable.

File map

Path Purpose
local_harness/scan.py Shared scan engine — runs /vulnhunt on a cloned repo, with retry/backoff, timeouts, worker pool; returns a ScanResult.
local_harness/clone.py Clone a target repo.
local_harness/config.py MODEL, worker counts, retry/backoff, timeouts, skills/phases dirs.
local_harness/batch/run.py CLI: scan (+--resume, --max-workers), status, collect. Reads REPO_LIST.txt.
local_harness/batch/utils.py Batch helpers.
local_harness/benchmark/run.py Full run: clone → scan → judge → tally (--repos, --judge-only, --tally-only, --force-rejudge).
local_harness/benchmark/judge.py LLM-judge: compare findings against ground truth.
local_harness/benchmark/tally.py Compute + render the accuracy metrics report.
local_harness/benchmark/analyze_misses.py Diagnose missed findings (all, or --finding ID).
local_harness/benchmark/finding_history.py Track findings across runs.
local_harness/benchmark/ground_truth/*.json One file per repo; each finding pinned to a public GitHub URL at a commit. Ships only a synthetic EXAMPLE.json (OWASP Juice Shop / WebGoat / NodeGoat) — bring your own corpus.

Commands

python -m local_harness.batch.run scan [--resume] [--max-workers N]
python -m local_harness.batch.run status | collect
python -m local_harness.benchmark.run [--repos NAME] [--judge-only --force-rejudge] [--tally-only]
python -m local_harness.benchmark.analyze_misses [--finding ID]

Benchmark state lives in local_harness/benchmark_results/state.json.


Part 3 — Inputs, outputs, model selection & deterministic calls

What happens here. The harness is orchestration + measurement, not analysis. It runs the scanner across many repos (batch) or grades it against known answers (benchmark). The only LLM step it owns is the judge in benchmark mode; everything else is deterministic Python.

Inputs. Batch: local_harness/batch/REPO_LIST.txt (GitHub URLs). Benchmark: ground_truth/<repo>.json (each real finding pinned to a public URL + commit). Both: settings in local_harness/config.py.

Outputs. Batch: per-repo *_VULNHUNT_RESULTS_* dirs, gathered by collect into to_upload/. Benchmark: a persisted benchmark_results/state.json and a tallied accuracy report; analyze_misses diagnostics for gaps.

How the model version is chosen. A single constant — MODEL = "claude-opus-4-8" in config.py — is used for both the scan and the LLM judge; change it once to switch every workflow. There is no per-repo or per-finding routing here.

Deterministic (non-LLM) calls. (1) git clone of each target (clone.py). (2) subprocess claude -p /vulnhunt … — it shells out to the CLI rather than embedding the SDK (scan.py). (3) A ThreadPoolExecutor (up to MAX_SCAN_WORKERS = 5) with retry/backoff and per-scan timeouts. (4) JSON parse of ground-truth and results. (5) Tally — deterministic metric computation and report rendering (tally.py). (6) State persistence after every operation (fully resumable). The judge (judge.py) is the one LLM call — it compares the scanner's findings to ground truth using MODEL, with its own retry/backoff (JUDGE_* settings); the scoring that follows is deterministic.


See also: Scanner · Runtime agent · module index.