read the diff directly

docs / process / runbooks / patch-and-verify

← all docs

Runbook: Patch & Verify ⚠️ (this runbook edits code)

Owner: Security engineer | Frequency: After Step 3 (Triage) of the SOP produces ranked findings | Last Updated: 2026-08-14 | Last Run: —

Verified against ../../../samples/defending-code-reference-harness/ (.claude/skills/patch/, harness/cli.py's patch subparser, harness/patch_grade.py).

⚠️ This is the runbook that writes to disk. /patch and vuln-pipeline patch generate diffs; in fix mode they can apply them. Review every generated diff before it reaches a real codebase — target-derived text is wrapped in <untrusted_data> at generation time, which is a mitigation against prompt injection, not a guarantee the diff is correct or safe.

Purpose

Draft a candidate fix for a verified finding (static TRIAGE.json or a pipeline results/ directory) and put it through an adversarial grading loop — rebuild, re-run the original PoC, run the test suite, and (unless skipped) a fresh find-agent re-attack — before it's presented as trustworthy. This is Step 4 of process-operate-dcrh.md.

Prerequisites

Procedure

Step 1: Patch from static triage

claude
> /patch ./TRIAGE.json --repo targets/canary

Expected result: PATCHES/ directory with a candidate fix per verified finding. If it fails: N/A — read/write-file only, safe to run unsandboxed; review the plan before approving any file writes if running interactively.

Step 2: Patch from a pipeline run — all bugs

cd samples/defending-code-reference-harness
bin/vp-sandboxed patch results/drlibs/<timestamp>/ --model <model-id>

Expected result: For each unique bug in reports/, a fix is drafted and run through the grade loop (default cap: 5 fix↔grade iterations, --max-iterations to change). Passing patches land as a verified diff under the run's results directory. If it fails: See Troubleshooting.

Step 3 (faster iteration): Patch one bug, skip the re-attack tier

bin/vp-sandboxed patch results/drlibs/<timestamp>/ --model <model-id> --bug 3 --no-reattack

Use this for quick iteration while developing a fix strategy — not for the fix you intend to ship. --no-reattack skips the tier that tries to find a way around the patch, which is the strongest signal that a fix actually closes the hole. Expected result: Faster turnaround, T0–T2 verification only (build + original PoC + test suite — no fresh find-agent re-attack). If it fails: See Troubleshooting.

Step 4: Add the advisory style judge (optional)

bin/vp-sandboxed patch results/drlibs/<timestamp>/ --model <model-id> --style

Expected result: An additional, advisory judgment on whether the fix matches the target's existing code style — informational, does not affect the pass/fail verdict. If it fails: N/A.

Step 5: Human review — before applying anything upstream

# read the diff directly
cat results/drlibs/<timestamp>/reports/bug_NN/patch.diff   # or wherever the diff lands — check step 2's output path

Expected result: You have personally read the diff, understand what it changes, and confirmed it does not touch anything outside the intended fix (no unrelated files, no suspicious additions). If it fails: Do not apply the diff. Escalate per below.

Verification

Troubleshooting

Symptom Likely Cause Fix
Grade loop exhausts --max-iterations without passing The bug may need a more structural fix than the playbook-style approach can draft in N tries Escalate to the target's engineering owner for a manual fix; keep the failed attempts as documentation of what didn't work
Patch builds but the test suite fails Fix introduced a regression Do not apply; the grade loop should have caught and iterated on this — if it passed anyway, treat as a harness bug and report upstream
Re-attack tier finds a new crash after the "fix" The original fix was incomplete Expected outcome of the tier working correctly — treat the new crash as a fresh finding, back to Step 2 (scan) on the narrowed area
Diff touches files outside the expected scope Possible prompt-injection influence from target-derived text, or an overly broad fix Do not apply. Re-run with --no-reattack removed and review prompts/logs; escalate if the cause isn't obvious

Rollback

If a patch was applied to the target's own repo (not just written to PATCHES/ or the pipeline's results directory) and needs to be undone: standard git revert on the commit that applied it, or git checkout -- <file> before committing. DCRH itself never commits on your behalf — applying and committing a patch is always a separate, human-initiated step.

Escalation

Situation Contact Method
Diff looks suspicious (scope creep, unexplained changes) Security engineering lead Immediate — do not apply, do not merge
Fix can't pass the grade loop after reasonable iteration Target's engineering owner Standard handoff — treat as a manually-fixed finding
Suspected prompt injection via target-derived content Security engineering lead Immediate — preserve the transcript for review

History

Date Run By Notes
— — —