Deterministic Tools vs. LLM Reasoning
VulnHunter mixes two kinds of work: LLM reasoning (a model reads code and decides something) and deterministic tool calls (ordinary code that always produces the same output for the same input — parsers, schema validators, git, subprocess calls). This document lays out exactly which is which, per component, and answers a specific question: does VulnHunter use a LangChain/LangGraph-style agent framework anywhere? It does not.
No LangChain / LangGraph / DeepAgents
A full-repository search (grep -rniIE "langchain|langgraph|deepagent" across every .py, .md,
.toml, .json, .txt file) returns zero matches. None of the five components declare
langchain, langgraph, deepagents, crewai, autogen, or any comparable agent-orchestration
package as a dependency. This is a real architectural choice, not an oversight — VulnHunter's own
docs describe the reasoning explicitly (see vulnhunter-fix/vulnhunter_fix/graph/build.py):
graph.build runs Graphify's AST extraction in-process, with a hard check
(check_backend_isolation()) that refuses to proceed if any cloud-LLM environment variable is set —
the code-graph step is required to stay non-agentic.
VulnHunter's only two agentic mechanisms are:
- Native Claude Code subagent dispatch — the
Agenttool, used by all three skills to spawn phase/worker subagents. This is a Claude Code primitive, not a third-party framework. - The Claude Agent SDK (
claude-agent-sdk) — the single orchestration dependency ofvulnhunter-agent/pyproject.toml, used only to drive/vulnhuntheadlessly.
No component builds a state graph, a DAG of LLM calls, or a multi-agent framework of its own; the "agentic" behavior is entirely Claude Code's native subagent model.
Per-component breakdown
vulnhunt/ (scanner) — almost entirely LLM reasoning
There is no static-analysis engine in this component at all — no tree-sitter pass, no AST parse
of its own. Every phase is a subagent that reasons over what Grep/Glob/Read surface. The only
deterministic pieces are the navigation tools themselves and, in bash-enabled mode, two mechanical
steps: dependency install (npm install / pip install -r / go mod download /
mvn dependency:resolve / sbt update — chosen deterministically by which manifest file is
present) and exploit-test execution (the target project's own test runner). /cost after each
phase is a deterministic spend query, not reasoning. In read-only mode, there are no external
process calls at all.
vulnhunter-fix/ (fixer) — a genuine hybrid
This is the component where deterministic code brackets LLM reasoning most heavily:
| Deterministic (always same output for same input) | LLM reasoning |
|---|---|
scripts/detect_mode.sh — in-place vs fork, from env/CWD |
Plan the fix approach per cluster |
scripts/language-detect.py — manifest files, then suffix majority-vote |
Write the exploit demo |
scripts/compute-completeness-tier.py — walks fixed signals conservative-first (WORKAROUND → MITIGATION → FULL), only escalates to a bounded LLM prompt when no signal matches |
Write the RED (failing) security test |
scripts/parse-tier-judgment.py — re-validates that bounded LLM output against a JSON Schema |
Implement the GREEN fix |
vulnhunter_fix/graph/build.py — Graphify AST-only extraction, content-hash cached, refuses cloud-LLM env vars |
— |
git worktree create/reset/commit (setup_worktree.sh, worktree-reset.py) |
— |
gh CLI — PR creation |
— |
pytest / the project's test runner — RED/GREEN execution |
— |
jsonschema (Draft 2020-12) validation of finding/fix-plan/result/triage artifacts |
— |
The six delivery gates (run-gates.py → check-*.py/validate-*.py/anti-merge-check.py) — severity-mask, body-completeness, scope, idempotency, anti-merge, verification-table; each is a stateless pass/fail script |
— |
Per-finding model routing — config.json → models.routing picks opus/sonnet/haiku by severity + CWE prefix + multi-file flag, a deterministic lookup |
(the routed model then reasons about the fix) |
The design intent is explicit: LLM output is only ever trusted after it passes a deterministic check. The completeness-tier classifier is the clearest example — it is documented to never silently return FULL; only a matched deterministic signal, or an escalation to a schema-validated bounded LLM call, produces a result.
vulnhunt-fix-verify/ (verifier) — almost entirely LLM reasoning, by design
The tool envelope is Read/Write/Edit/Glob/Grep/Agent — no Bash, no network, so it structurally
cannot run deterministic external tools like git, a test runner, or a linter. The only
deterministic step is JSON-Schema validation of the emitted verify_disposition.json at Phase
4, plus file-existence checks used as phase synchronization signals. Everything else — reading the
fixed code and the original finding, deciding whether the fix holds — is model reasoning over static
reads. This is intentional: verification is supposed to re-derive the answer independently, not
re-run the fixer's own deterministic machinery.
vulnhunter-agent/ (headless runtime) — a deterministic shell around one LLM call
The runtime is software-engineering plumbing wrapped around a single LLM invocation (the scan itself, which it does not reason inside — it just launches and streams it):
- Deterministic: TOML config load + env overlay, auth (OAuth2 client-credentials token fetch or
AWS SigV4 request signing — both are cryptographic/HTTP operations, not model calls),
git clone, GitHub API calls (publish push, issue create/list, dedup pool paging),jsonschemavalidation ofscan_manifest.jsonandverify_disposition.json, 429 retry/backoff, autocompact-threshold and stall-timeout watchdogs, an optional OS-level sandbox, JSONL audit logging, and a model-name → tag normalizer used only for telemetry. - LLM reasoning: the scan itself (via the Agent SDK), and — the one place this component makes
its own small LLM calls rather than just launching a scan — the issues stage, which extracts
findings from the scan output and semantically deduplicates them, trying
haiku_modelfirst and falling back tosonnet_model.
harness/ (batch & benchmark, dev-only) — deterministic orchestration + one LLM judge
git clone, subprocess.run(["claude", "-p", "/vulnhunt", ...]), a ThreadPoolExecutor with
retry/backoff and per-scan timeouts, JSON parsing of ground truth and results, tally/report
rendering, and atomic state persistence after every operation are all deterministic. The judge
(judge.py) is the one LLM call in this component — it compares the scanner's findings against a
known-vulnerable corpus using the single MODEL constant; the scoring that follows the judge's
output is deterministic arithmetic.
The pattern across all five components
The two prompt-only skills (vulnhunt, vulnhunt-fix-verify) are almost pure LLM reasoning —
by design, since their job is judgment, not mechanical processing. The three Python components
(vulnhunter-fix, vulnhunter-agent, harness) each follow the same shape: deterministic code
handles configuration, credentials, version control, network calls, schema validation, and pass/fail
gating; LLM calls are reserved for the parts that require judgment (writing a fix, extracting
findings from prose, judging accuracy against ground truth) and are always followed by a
deterministic check before their output is trusted — a schema validation, a gate script, or a
human Approve/Pause. No component delegates orchestration itself to an LLM framework; the control
flow is ordinary Python (or Claude Code's native subagent dispatch), and the LLM is called as a
function, not as the thing driving the loop.
See also: HLD · LLD · module index (each module doc's Part 3 has the same inputs/outputs/model-selection/deterministic-calls breakdown, per component).