Increasingly I am being told to “use a security harness” when analyzing code for potential vulnerabilities, but what is a “harness” and how does it differ from just running Claude Code and asking it nicely to find me some sweet vulns?
This post is literally my notes on reading up on this topic via a series of Cloudflare blog posts. That is, this post is a summary and does not intend to provide anything you cannot find anywhere else (with a lot more words used).
This Cloudflare blog post has a good introduction to the main ideas — that coding agents are tuned for a single thread of work at a time while supervised by a human. That is, they are trained not to take tangents, or hold multiple theories at once. In a general coding session an agent will interact with a relatively small part of a code base, not systematically scan it for all potential vulnerabilities. A harness will instead gather theories as it encounters them, record them in a database, and then investigate them all in parallel. Its a funny coincidence that such a process eats a lot of tokens right?
The follow-up Cloudflare blog post then builds on this idea with further justification for why coding agents aren’t the answer — they clearly received some push back on that idea. Specifically, they argue that sub-agents are useful, but don’t scale to where you need them to for a holistic review of a code base. They argue that sub-agents share a context window but that is simply not true with Claude Code, so I am unsure where that idea is coming from. They are right however that persistence, de-duplication, resumability, and dependency tracing at scale are all lacking for coding agents. Basically this sort of compute intensive security analysis is an orchestration problem, not a development problem.
Cloudflare makes a series of recommendations based on their playtime with Anthropic’s Mythos 5 model. Specifically they recommend the following:
- keep the scope of a given investigation narrow — more “I think there’s an issue with image format confusion attacks” not “read all the qemu code and find me sweet vulns kthx”.
- have the harness apply adversarial review to its findings — that is, ask it to disprove what it thinks it found. If the finding survives, then its more likely to be real. Its even better if that second review is done by a different agent or model with a different prompt.
- splitting the vulnerability across different agents can work well.
- parallel tasks are better than one exhaustive agent — then have a later phase de-duplicate the findings.
The follow-up post further recommends:
- Do not build to a specific model. The models are evolving too quickly, and models are taken offline as the resources that host them are repurposed to host their successors. Treat models as interchangeable components.
- Similarly, using a different model for validation than was used for discovery continues to be an important technique, especially as a single model will “tend to look at code paths through the exact same lens”.
- Ensure you’re looking at the entire ecosystem you code runs in, not just the code base of the application.
- Start by building “skills” (the summaries for how to perform an action that many LLMs can ingest).
Cloudflare found they needed to address three main problems when building their harness:
- Context management — solved with external state management in a database.
- Persistence and crash recovery.
- Cross repository reasoning.
They also implemented two separate stages, noting the earlier recommendation that each stage use a different LLM model:
- Vulnerability discovery, which they call a Vulnerability Discovery Harness (VDH).
- Vulnerability validation, which they call a Vulnerability Validation System (VVS).
One interesting idea is that instead of providing a lot of static analysis tools to the models, they instead provided a wish list function where the agents could record their desire to use a tool, and then re-run work when that tool becomes available. This was at least partially motivated by having put a lot of work into deploying semgrep, for the models to then never actually execute it.
One interesting quote from along the way:
“More than one team we have spoken with is now operating under a two-hour SLA from CVE release to patch in production… Faster is not going to be enough, and we think a lot of teams are about to spend a lot of time, effort, and money learning that the hard way.” — Cloudflare
Tune in next time to watch me try to find a security harness I can actually run using a leading frontier cyber model… or Anthropic Opus.