how many runs have finished:

docs / process / runbooks / run-a-detection-scan

← all docs

Runbook: Run a Detection Scan (static or execution-verified)

Owner: Security analyst (static) / Security engineer (pipeline) | Frequency: As needed — new target onboarding, periodic re-scan, or after upstream target changes | Last Updated: 2026-08-14 | Last Run: —

Verified against ../../../samples/defending-code-reference-harness/ (.claude/skills/vuln-scan/, harness/cli.py's recon/run subparsers).

Purpose

Produce findings for a target — either a fast static read (/vuln-scan) or an execution-verified crash via the autonomous pipeline (vuln-pipeline run). This is Step 2 of process-operate-dcrh.md. Detection only — does not edit code.

Prerequisites

Procedure

Step 1: Static scan (fast, no sandbox needed)

claude
> /vuln-scan targets/canary

Expected result: VULN-FINDINGS.{json,md} written in the working directory, scoped by THREAT_MODEL.md if present. Nothing is built or executed — read-only. If it fails: N/A — this skill only reads/writes files; failures are usually a missing or malformed THREAT_MODEL.md (skill still runs, just with less scoping context).

Step 2: Execution-verified scan — small first wave

cd samples/defending-code-reference-harness
bin/vp-sandboxed run drlibs --model <model-id> --runs 3 --parallel --stream --max-turns 100

Run a small wave first (--runs 3) to measure token burn and prompt calibration before scaling up — this is the repo's own recommended sizing practice, not a suggestion to skip. Expected result: results/drlibs/<timestamp>/ appears immediately; run_000/, run_001/, run_002/ subdirectories fill in as each find+grade completes. With --stream, the first reports/bug_NN/report.json typically appears within minutes. If it fails: "runtime not registered" or "proxy not running" → sandbox isn't set up, go to setup-and-sandbox.md. Rate-limit (429) errors are absorbed by the CLI's own backoff, then the pipeline's own retry loop (up to 20 attempts) — no action needed unless the batch ends in agent_failed.

Step 3 (optional): Recon-first, if not using hand-written focus_areas

bin/vp-sandboxed recon drlibs --model <model-id>

Add --auto-focus to run instead of running this separately — --auto-focus runs recon automatically as part of the batch and uses its partition. Use the standalone recon command only if you want to review/edit the proposed focus_areas before committing to a full run. Expected result: YAML printed to stdout — focus_areas, an attack-surface partition. If it fails: N/A — read-only, no target execution beyond static source reading.

Step 4: Watch the run

# how many runs have finished:
ls results/drlibs/<timestamp>/run_*/result.json | wc -l

# what's been submitted so far:
cat results/drlibs/<timestamp>/found_bugs.jsonl

# what an agent is actually doing (tool calls only):
grep '"type":"tool_use"' results/drlibs/<timestamp>/run_000/*.jsonl

Expected result: Growing counts as the batch progresses. Transcripts persist even on failed/killed runs. If it fails: N/A — these are read-only inspection commands.

Step 5: Scale up (once the small wave looks right)

bin/vp-sandboxed run drlibs --model <model-id> --runs 20 --parallel --stream --auto-focus

Expected result: Same layout as Step 2, more run_NNN/ directories, more reports. If it fails: See Troubleshooting.

Verification

Troubleshooting

Symptom Likely Cause Fix
Batch has runs marked agent_failed Transient API error exhausted the 20-attempt retry budget bin/vp-sandboxed run <target> --runs N --resume <results-dir> — same --runs N the batch was launched with
A run is error_max_turns Turn budget genuinely exhausted, not transient Not resumable by design; any crash submitted before the cap was salvaged and graded already. Raise --max-turns and start a fresh run for the remainder
/vuln-scan finds nothing on a real target Expected on well-hardened code, or scope may be too narrow Cross-check THREAT_MODEL.md's asset/entry-point coverage; consider the execution-verified path for stronger signal
Pipeline finds far more on canary than expected Working as intended — canary is deliberately vulnerable Use targets/canary for smoke-testing the harness itself, not for judging finding quality

Rollback

Detection is read-only against the target (agents run in isolated containers; nothing writes back to the target source). To discard a run's output: rm -rf results/<target>/<timestamp>/. No target-side rollback needed.

Escalation

Situation Contact Method
Sandbox/infra failures Security engineer (sandbox operator) See setup-and-sandbox.md Escalation
Findings suggest a live, exploitable issue in production code AppSec lead, immediately Out-of-band (do not wait for the full triage cycle)
Consistent rate-limit exhaustion at planned batch size Security engineering lead Check account limits in the Claude Console against the repo's ~100-agents-per-1M-input-TPM sizing rule

History

Date Run By Notes
— — —