Agent Orchestration Design
Harness as a graph · handoff contracts · state, credentials and tooling · three worked flows
Companion to the production-grade requirements spec. This document is design, not requirements: it explains how the pieces fit and why, and it carries the normative envelope schemas that the requirements reference.
Companion to An Ideal Vulnerability-Discovery Harness · index
1. The harness is a graph, and it is worth saying so explicitly
The three-stage pipeline is usually drawn as a linear sequence. That drawing is convenient and slightly misleading. What is actually being built is a directed graph with typed edges, durable state and policy evaluated at traversal time. Naming the primitives makes several design decisions fall out rather than needing to be argued.
| Primitive | What it is in the harness |
|---|---|
| Vertex | An agent or a deterministic tool. Recon, finder, validator, exploit-prover, fixer, post-fix validator are AI vertices. The policy evaluator, the scanner adapters, the schema checker and the diff applier are non-AI vertices. Both kinds sit in the same graph and are addressed the same way. |
| Edge | A handoff. Not a pointer but a contract: it declares the trust level of what is moving, the certainty attached to it, its provenance, and the payload. Section 3 gives the schema. |
| State | The findings store, code graph, evidence store, patch state and conformance history. State lives outside the agents and outlives any single run, which is what makes resume-rather-than-restart possible after a failure. |
| Policy | Autonomy tiers, approval gates, scope tokens, evidence-tier thresholds, loop budgets. Evaluated on the edge before traversal, by the orchestrator, never by the agent that wants to traverse it. |
1.1 Three places the graph model needs care
It has cycles, and the cycles are the dangerous part. Finder to validator to fixer is acyclic. Still-present reopening a finding, and configuration drift re-entering remediation, are loops. An agent graph with an unbounded loop is how a run consumes a quarter of the model budget arguing with itself. Loop bounds are therefore properties of the graph, not of any agent: iteration ceiling, wall-clock ceiling and cost ceiling are enforced by the loop governor at the orchestrator level, and breaching one produces an inconclusive terminal state rather than a retry.
Policy sits on the edge, and that is a security property rather than a stylistic preference. If the fixer decides whether the fixer may merge, there is no control, only a prompt. When the orchestrator evaluates the gate before allowing traversal, no amount of reasoning inside the agent reaches past it. This is the same distinction that separates a token that carries its own limits from an API that trusts its caller.
State is typed, directional and deliberately lossy. A stock orchestration runtime accumulates context forward: each node sees everything the previous nodes produced. That default is exactly wrong here. The validator must receive the finder's claim and must not receive the finder's reasoning, because a hypothesis that arrives wearing its own justification arrives as a premise. The edge drops information on purpose. This single departure from default state-passing is the most important thing in this document.
1.2 What this means for framework choice
Any graph runtime with conditional edges and durable checkpoints will carry the shape. The requirements that constrain the choice are not about the graph API:
- edges must be able to transform and reduce state, not only forward it
- checkpoints must be durable and resumable across process restarts
- the runtime must expose traversal decisions as events for the audit log, not just the final result
- a node must be able to run without inheriting the caller's credentials
A runtime that makes shared mutable state the path of least resistance will produce a harness that launders conclusions, no matter what the prompts say.
2. State, credentials and tooling
Section 1 named state and policy as graph primitives. This section is what they are made of.
2.1 State storage
State separates into four tiers by durability, and the tiering is a control rather than an optimisation.
| Tier | Contents | Lifetime | Notes |
|---|---|---|---|
| Durable, authoritative | Findings, verdicts, evidence pointers, lineage, approval records, conformance history | Retained to the audit clock, years | Append-only where the record is an audit artifact. A verdict is never overwritten; a new verdict supersedes it and both remain. |
| Durable, working | Code graphs, asset inventory, declared and observed configuration state | Retained to the debugging clock, weeks | Rebuildable from source. Cheap to discard, expensive to recompute. |
| Ephemeral | Agent scratch memory, intermediate reasoning, tool call transcripts within a run | Destroyed with the sandbox | Never promoted to durable storage without passing through an envelope. |
| Never persisted | Raw secrets encountered in target code, full prompt and completion text where it contains target payloads | Not stored | Redacted at capture, not at read. Redacting on the way out means the secret was already written down. |
Two rules govern movement between tiers. Nothing moves from ephemeral to durable except as a validated envelope, which prevents an agent from persisting its own reasoning as fact. And exploit artifacts stay in the region that produced them; only conclusions and counts aggregate to the global control plane.
Short-term versus long-term memory. Short-term memory is the run's working context and dies with it. Long-term memory is the findings store plus the learning signal, and it is read through queries rather than injected wholesale. An agent that carries forward everything it has ever concluded will eventually recognise its own past guesses as established fact. Memory is retrieved deliberately, scoped to the current target, and tagged with its provenance so the agent can tell what it is reading.
2.2 Short-lived credentials
No agent holds a standing credential. The pattern is minting rather than holding:
- The orchestrator presents the run's identity and the intended action to a credential broker.
- The broker evaluates policy: this agent, this target, this scan, now.
- The broker mints a token scoped to one target, one action class, one run, expiring in minutes.
- The agent uses it and never sees a longer-lived secret.
- The tool independently verifies the token and does not trust the caller's assertion about what it is allowed to do.
Delegation narrows and never widens. If the orchestrator holds read and write across repositories A, B and C, the finder receives read on A and B, and a tool call within the finder receives read on A alone. The effective permission of any chain is the intersection of every link, not the union. This is what stops a hostile instruction file from borrowing a privileged agent's hands: the file can persuade an agent, but persuasion does not enlarge a token.
Workload identity is preferred to any stored secret. The best-protected secret is the one that does not exist, so where the platform supports attested workload identity the broker mints against that rather than against a stored key. Revocation is per agent identity and must be tested rather than merely designed, because an untested revocation path is an assumption.
2.3 Tooling: deterministic tools, MCP, and skills
The harness reaches the outside world through three distinct mechanisms, and conflating them is a common source of trouble.
Deterministic tools are called when the answer does not require judgement: run the parser, compute the call graph, evaluate the policy rule, apply the diff, diff two configuration states. These are ordinary function calls with typed inputs and outputs. They are preferred wherever they can decide, because a deterministic answer costs nothing to validate and cannot hallucinate. In the configuration flow specifically, most findings never reach a model at all.
MCP servers are how the harness reaches third-party and internal systems: source control, ticketing, scanners, cloud APIs. Three properties matter:
- Tool descriptions are instruction text. The model reads them. A description is therefore part of the trust boundary and must be pinned to a reviewed version, with changes treated as a code change rather than a configuration refresh.
- Each server gets its own credential and mutual authentication. A single shared identity across servers reintroduces the ambient authority the design is trying to remove.
- Output is data, never instruction. Tool results are tagged at their trust tier on the way in, and content that looks like a directive is neutralised rather than obeyed.
Custom skills are versioned prompt-and-procedure bundles held in the agent repository: how to hunt a particular vulnerability class, how to shape a remediation for a given framework, how to phrase a disclosure. They are artifacts with owners, versions and review history, not strings pasted into a system prompt. Two consequences follow: a skill change is attributable in the lineage of every finding produced after it, and a model swap can be evaluated against a fixed skill set so that the variable under test is actually the model.
The distinction that matters operationally: a tool is something the harness calls, an MCP server is something the harness connects to, and a skill is something the harness knows. They fail differently and are governed differently.
3. Agent-to-agent handoff
3.1 What travels, and what must not
Every hop carries the same envelope. The fields exist to stop specific failures rather than to be complete.
| Field | Purpose | Failure it prevents |
|---|---|---|
trust_tier | Marks target-derived content as untrusted for the whole life of the record | Content gaining authority by crossing a boundary |
certainty | Carries the claim's epistemic status and can only degrade or hold across a hop | Conclusion laundering: a hypothesis arriving as a premise |
provenance | Which agent, which model version, which skill version, which evidence | Untraceable findings; inability to attribute a regression to a model swap |
payload | The claim and its evidence pointers, explicitly not the sender's reasoning | The receiver inheriting the sender's argument and agreeing with it |
authority | Absent by design. Permission travels separately, minted per hop | The confused deputy |
3.2 The envelope
{
"envelope_id": "uuid",
"run_id": "uuid",
"hop": { "from": "finder@2.4.1", "to": "validator", "sequence": 3 },
"trust_tier": "untrusted | sanitised | verified | operator",
"certainty": {
"level": "hypothesis | corroborated | proven | refuted",
"confidence": 0.0,
"may_increase": false
},
"provenance": {
"agent": "finder",
"agent_version": "2.4.1",
"model": "vendor/model@snapshot",
"skills": ["injection-hunt@1.7"],
"derived_from": ["envelope_id", "..."]
},
"payload": {
"claim": "string, one assertion",
"locations": [{ "uri": "...", "line": 0, "symbol": "..." }],
"evidence": [{ "tier": "T1|T2|T3|T4", "ref": "evidence-store://...", "kind": "..." }],
"reasoning": null
},
"budget": { "iterations_used": 2, "iterations_max": 6, "cost_used_usd": 0.41 }
}
Three constraints are enforced by the orchestrator rather than requested of the agent:
payload.reasoningis stripped on the finder-to-validator hop. Not omitted by convention, removed by the edge.certainty.may_increaseis false on every hop except the validator's output. Only an adversarial attempt to disprove may raise a claim's standing, and only a live re-test may raise it toproven.derived_fromis mandatory and forms the lineage chain. A finding with a broken chain isinconclusiveby definition, because its history cannot be reconstructed.
3.3 Certainty transitions
| Hop | Entering | Leaving | Rule |
|---|---|---|---|
| Recon → Finder | — | hypothesis | Recon produces structure, never findings |
| Finder → Validator | hypothesis | hypothesis | Unchanged. Reasoning stripped |
| Validator → Exploit-prover | hypothesis | corroborated | refuted | Refuted terminates and is written to the learning signal |
| Exploit-prover → Fixer | corroborated | proven | corroborated | Only a successful proof reaches proven. Failure to reproduce does not refute |
| Fixer → HITL gate | proven | unchanged | The fixer cannot alter the standing of the finding it is fixing |
| HITL → Post-fix validator | proven | unchanged | Approval authorises a change, it does not strengthen a claim |
| Post-fix validator → terminal | proven | closed-verified | still-present | regressed | inconclusive | Four exits. Absence of signal is inconclusive, never closed |
The last row is the one most often collapsed in practice. A re-scan that returns nothing has not proved the fix worked; it has proved the scan saw nothing. Recording that as closed converts a measurement gap into a false assurance, and does so at exactly the point where nobody is looking any more.
3.4 Human-in-the-loop gates
A gate is a vertex, not a callback. It has an identity, an inbox, an SLA and an audit record, and the graph cannot traverse past it without a decision artifact.
| Gate | Flow | Trigger | Payload to the human | Non-removable because |
|---|---|---|---|---|
| Pre-exploitation | B (systems) | Any action against a live system that could be observed as an attack | Target, blast radius, abort conditions, SecOps pre-notification status | An exploit-capable system acting unannounced is indistinguishable from an intrusion |
| Pre-merge | A (code) | Any change to a repository | Diff, test result, evidence bundle, what the validator tried and failed to break | A model that writes the fix cannot be the sole judge that the fix is correct |
| Pre-change | B, C | Any mutation of a live system or configuration | Change plan, risk assessment, rollback, conformance impact | Blast radius on live infrastructure is not recoverable by revert alone |
| Escalation | All | Confusion, contradiction, or budget exhaustion | The disagreement itself, both positions, what evidence would settle it | An agent that cannot decide must surface that, not pick |
Escalation deserves particular attention because it is the gate most often omitted. The harness must be able to say I do not know, and must have somewhere to send that. The trigger conditions are concrete: two agents reach contradictory verdicts on the same claim; an agent's confidence falls below threshold on a high-severity finding; the loop governor exhausts a budget; a tool returns a result the schema cannot parse; or an evidence tier cannot be established at the level the severity requires. Each produces a structured escalation carrying both positions and, critically, a statement of what evidence would resolve it — which is what makes the human's job tractable rather than a re-adjudication from scratch.
4. Source code: discovery to verified remediation
The code flow is the one with the most forgiving failure mode and the most subtle correctness problem. Forgiving, because a wrong fix is caught by tests and review before it reaches production. Subtle, because the finder is deliberately tuned for recall and will therefore produce a majority of claims that are wrong, which places the entire burden of precision on the validator.
The hard limits are the design. The finder never asserts certainty, so its output is structurally a hypothesis. The validator never sees the finder's reasoning, so it cannot be persuaded by an argument it did not construct. The exploit-prover runs only in a sandbox, so a proof attempt cannot become an incident. The fixer has no shell and cannot merge, so the worst outcome of a bad fix is a rejected pull request. The post-fix validator cannot record an unverifiable result as closed, so the pipeline cannot quietly conclude in its own favour.
The loop back from still-present reopens as a new hypothesis, not as the old premise. This matters more than it appears: carrying the original claim forward would mean the second attempt inherits the first attempt's conclusion, and the validator would be asked to disprove something it has already accepted once.
5. Running systems: discovery to verified remediation
The infrastructure flow differs from the code flow in one respect that changes everything downstream: it touches live systems. Every control that follows exists because of that.
Authorisation is explicit and bounded. Scope comes from the CMDB rather than from discovery, so the harness cannot expand its own remit by finding something interesting adjacent to its target. The scope token is minted per job, names an exact allowlist, and expires. Nothing adjacent is in scope by virtue of being reachable.
Blast radius is bounded by construction. Rate limits, no destructive verbs, and abort conditions that halt a run mid-action. These are enforced outside the agent, because an agent that can disable its own throttle does not have one.
SecOps is pre-notified, and this is not a courtesy. An exploit-capable system operating inside the estate without the security operations team knowing will be detected as an intrusion, and correctly so. The pre-notification path is drawn as a first-class flow because omitting it produces a predictable and expensive incident.
Two gates rather than one. Approval before exploitation and approval before change application are different decisions with different risk profiles and often different approvers. Collapsing them into a single sign-off means whoever approves the exploit has implicitly approved the remediation, which is not what they were asked.
The four re-validation exits carry more weight here than in the code flow, because inconclusive on live infrastructure is common and legitimate. A host that was unreachable during the re-scan has not been verified.
6. Configuration: drift to verified conformance
The configuration flow makes a point the other two do not: most of this does not need a model at all.
Drift between declared and observed state is a deterministic comparison. Whether a control satisfies a policy is a rule evaluation. Both are cheaper, faster, fully reproducible and impossible to hallucinate. The policy engine therefore short-circuits the AI entirely for anything a rule can decide, and the diagram draws that as a visible bypass rather than burying it.
The AI is for the residue, and the residue is genuinely hard. A publicly readable bucket is a finding or a non-event depending on what is in it, who can reach it, and what else must be true for it to matter. That is context and blast radius, which is exactly what rules are bad at and reasoning is good at. Scoping the model to the residue makes the cost profile sane and makes the AI's contribution auditable: if the rule decided, the model was not consulted, and the lineage says so.
Remediation is proposed as code, never applied to a console. The output of the remediation agent is an infrastructure-as-code change on a branch. A console edit fixes the symptom and guarantees the drift returns on the next apply, so a harness that edits live configuration is manufacturing its own future findings.
Recurrent drift is a process failure, not a new finding. If the live state diverges again after remediation, one of three things is true: the change did not hold, the drift rule did not tighten, or an out-of-band edit reverted it. None of those is a newly discovered vulnerability, and triaging it as one inflates the finding count while hiding the actual problem.
7. What this design does not solve
Stating the limits is part of the design.
A single-vendor estate constrains validator independence. The strongest hallucination control is a validator that does not share the finder's failure modes, which in practice means a different model family. An organisation standardised on one provider gets a weaker version of this control, and should know that rather than assume the architecture compensates.
The evidence ladder has a soft bottom rung. T1 executable proof and T2 deterministic static assertion are strong. T4, human-confirmed, is a judgement, and judgement under volume degrades. The cap on T4 findings is a real constraint that will be uncomfortable when a scan produces more findings than the cap allows, and the correct response is to improve evidence generation rather than raise the cap.
Post-remediation validation inherits the coverage of whatever performs it. A closed-verified result means the re-test did not find the issue, which is a statement about the re-test as much as about the fix.
Cost per true positive is only meaningful once the true-positive rate is stable. Early in a deployment the denominator moves for reasons unrelated to model performance, and reading model quality from that ratio too early will produce confident conclusions from noise.