Runbook: Run a Detection Scan (static or execution-verified)
Owner: Security analyst (static) / Security engineer (pipeline) | Frequency: As needed — new target onboarding, periodic re-scan, or after upstream target changes | Last Updated: 2026-08-14 | Last Run: —
Verified against ../../../samples/defending-code-reference-harness/
(.claude/skills/vuln-scan/, harness/cli.py's recon/run subparsers).
Purpose
Produce findings for a target — either a fast static read (/vuln-scan) or an
execution-verified crash via the autonomous pipeline (vuln-pipeline run). This is
Step 2 of process-operate-dcrh.md. Detection only — does
not edit code.
Prerequisites
- [ ]
THREAT_MODEL.mdexists in the target directory (Step 1 of the SOP) — both paths use it as scoping context if present - [ ] For the pipeline path only: sandbox set up per setup-and-sandbox.md, credentials exported
- [ ] Know your target:
targets/canary(smoke test),targets/drlibs/alsa/htslib(real libraries), or your owntargets/<name>/with aDockerfile
Procedure
Step 1: Static scan (fast, no sandbox needed)
claude
> /vuln-scan targets/canary
Expected result: VULN-FINDINGS.{json,md} written in the working directory,
scoped by THREAT_MODEL.md if present. Nothing is built or executed — read-only.
If it fails: N/A — this skill only reads/writes files; failures are usually a missing
or malformed THREAT_MODEL.md (skill still runs, just with less scoping context).
Step 2: Execution-verified scan — small first wave
cd samples/defending-code-reference-harness
bin/vp-sandboxed run drlibs --model <model-id> --runs 3 --parallel --stream --max-turns 100
Run a small wave first (--runs 3) to measure token burn and prompt calibration before
scaling up — this is the repo's own recommended sizing practice, not a suggestion to skip.
Expected result: results/drlibs/<timestamp>/ appears immediately; run_000/,
run_001/, run_002/ subdirectories fill in as each find+grade completes. With
--stream, the first reports/bug_NN/report.json typically appears within minutes.
If it fails: "runtime not registered" or "proxy not running" → sandbox isn't set up,
go to setup-and-sandbox.md. Rate-limit (429) errors are absorbed
by the CLI's own backoff, then the pipeline's own retry loop (up to 20 attempts) — no
action needed unless the batch ends in agent_failed.
Step 3 (optional): Recon-first, if not using hand-written focus_areas
bin/vp-sandboxed recon drlibs --model <model-id>
Add --auto-focus to run instead of running this separately — --auto-focus runs recon
automatically as part of the batch and uses its partition. Use the standalone recon
command only if you want to review/edit the proposed focus_areas before committing to a
full run.
Expected result: YAML printed to stdout — focus_areas, an attack-surface partition.
If it fails: N/A — read-only, no target execution beyond static source reading.
Step 4: Watch the run
# how many runs have finished:
ls results/drlibs/<timestamp>/run_*/result.json | wc -l
# what's been submitted so far:
cat results/drlibs/<timestamp>/found_bugs.jsonl
# what an agent is actually doing (tool calls only):
grep '"type":"tool_use"' results/drlibs/<timestamp>/run_000/*.jsonl
Expected result: Growing counts as the batch progresses. Transcripts persist even on failed/killed runs. If it fails: N/A — these are read-only inspection commands.
Step 5: Scale up (once the small wave looks right)
bin/vp-sandboxed run drlibs --model <model-id> --runs 20 --parallel --stream --auto-focus
Expected result: Same layout as Step 2, more run_NNN/ directories, more reports.
If it fails: See Troubleshooting.
Verification
- [ ]
results/<target>/<timestamp>/reports/manifest.jsonlexists and lists at least one accepted bug (or zero, if the target genuinely has none reachable — a valid outcome) - [ ] Each
reports/bug_NN/report.jsonhas a grader score, not just the report agent's own claim - [ ]
found_bugs.jsonlentry count is consistent withrun_*/result.jsonpass count
Troubleshooting
| Symptom | Likely Cause | Fix |
|---|---|---|
Batch has runs marked agent_failed |
Transient API error exhausted the 20-attempt retry budget | bin/vp-sandboxed run <target> --runs N --resume <results-dir> — same --runs N the batch was launched with |
A run is error_max_turns |
Turn budget genuinely exhausted, not transient | Not resumable by design; any crash submitted before the cap was salvaged and graded already. Raise --max-turns and start a fresh run for the remainder |
/vuln-scan finds nothing on a real target |
Expected on well-hardened code, or scope may be too narrow | Cross-check THREAT_MODEL.md's asset/entry-point coverage; consider the execution-verified path for stronger signal |
| Pipeline finds far more on canary than expected | Working as intended — canary is deliberately vulnerable | Use targets/canary for smoke-testing the harness itself, not for judging finding quality |
Rollback
Detection is read-only against the target (agents run in isolated containers; nothing
writes back to the target source). To discard a run's output: rm -rf
results/<target>/<timestamp>/. No target-side rollback needed.
Escalation
| Situation | Contact | Method |
|---|---|---|
| Sandbox/infra failures | Security engineer (sandbox operator) | See setup-and-sandbox.md Escalation |
| Findings suggest a live, exploitable issue in production code | AppSec lead, immediately | Out-of-band (do not wait for the full triage cycle) |
| Consistent rate-limit exhaustion at planned batch size | Security engineering lead | Check account limits in the Claude Console against the repo's ~100-agents-per-1M-input-TPM sizing rule |
History
| Date | Run By | Notes |
|---|---|---|
| — | — | — |