The first time an MCP (Model Context Protocol) server felt real to me, it wasn't because of a clean demo. It was because of the noise.
TL;DR: The harness matters more than the protocol, and the evidence matters more than both. MCP earns its keep when it shortens the path from a good security question to trustworthy evidence, and almost everything interesting about making that work happens in the harness wrapped around the model. In this series, I will cover how to build an MCP for an AI SOC.
Building a capable tool is only half the battle. Making sure it works in the trenches and adapts on the fly is what really matters. Modern networks demand security platforms that are dynamic and able to navigate new and sometimes ambiguous threats. In this chapter, I look at how dynamic retrieval and autonomous skills can work to increase your SOC analyst’s’ abilities.
This chapter includes:
Once the AI framework was in a place where it could pull accurate information and give reliable results to users wherever they were, it was time to move to the next level. This is where playbooks come in. The first ones were fairly rigid, structured flows for things like alert validation: do this, then this, then this. They were useful, but they created a new context problem. Each step carried its token history forward. I eventually realized some playbooks weren't completing at all. They were failing somewhere in the middle, recovering, and still returning an answer. That was clearly not good enough for security work.
Once that was fixed, some playbook runs became huge. A single alert-validation playbook could use over a million tokens. That forced better compression, better retrieval, more careful context handling, and parallel execution where it helped.
Then code mode changed the idea again. If the model could inspect data and write bounded code, the playbook didn't have to be static. The AI agents could pick their own playbooks. They could adjust one. If an agent saw SMB traffic, it could follow SMB-specific pivots. If it saw file activity, it could follow that thread. When the data changed the investigation path, the workflow adapted.
At that point, we stopped having "AI runs a query" and started having agentic investigation. I don't mean an unconstrained agent loose in the SOC. I mean a harness where specialized agents or workflows did bounded jobs. Black Hat Asia was where this shape went live for real, with the full multi-agent harness running in production:
In my testing, a single orchestrator managing communication has been easier to reason about, inspect, and control than letting every agent talk to every other agent. Tradeoffs exist. Centralized orchestration has won on operability.
In the current version of this framework, three retrieval patterns do most of the heavy lifting. I wish someone had handed me this list in the first month of development:
If you only give your system embedding-based RAG, it'll miss structural investigations and it'll miss the "what changed across this window" questions.
Code mode + RL: this is the S-tier combination.
Code mode gives the model the freedom to write the query. RL means the next version of that query is better because the last one ran. Without code mode, RL has less to edit. Without RL, code mode is just a fancier execution path. Together, they turn a static tool surface into something that gets sharper with each use.
The first RL priority is to watch for and correct errors like calling the wrong function. When a query or tool call fails, the trace gets examined, the failure pattern gets matched, and the system proposes a code change to fix it.
The second is query optimization, and this is where the leverage shows up. When a query works but is slow, inefficient, or clearly going down the wrong investigation path, the system watches how successful runs look and rewrites its own Python to take the shorter path next time. That's how we jumped two versions (from gen-5 to gen-7 of the Splunk Python code mode) from the February CTF traces described in Chapter 2.
In a single-operator setup, I'll sometimes let it auto-apply, because the blast radius is me and I diff them to audit the changes. In production, I save the proposed diff rather than auto-applying it. I want to read what the system thinks it learned before it ships to users. The catch is that all of this runs on traces, so the compliance story gets real very fast, which I'll come back to in the final chapter of this series.
One pattern I keep seeing goes something like this: Someone stands up a self-improving agent. It works in a demo. It works on real, live data. Two weeks in, everything starts silently failing. That is almost always due to drift. The agent kept learning, nobody was watching, and what it learned wasn't what you wanted.
The fix isn't to stop letting agents improve themselves. It is to be selective about which ones get that power, and to monitor the ones that do. If an agent has settled into known-good behavior, the move is usually to freeze it, save that version as a baseline, and only propose new versions for human review rather than auto-applying them.
I use Hodoscope to watch this. It's an open-source trajectory-analysis tool from AR-FORUM that takes agent traces, has an LLM summarize each action, embeds the summaries, and projects them onto a 2D map. Density-difference overlays show where one model or configuration is behaving differently from others, which is exactly the pattern you want to catch early. I layered a simple alert on top of it: Anything more than 20% off my own rolling baseline gets flagged for review. That gives me a checkpoint before a self-improving agent quietly wanders somewhere I didn't want it to go.
With all of these pieces in place, it was time to address oversight and governance. In the final chapter, Chapter 4: Governing the machine: safety, learning, and results, I tackle the responsibility of bringing AI agents into production environments. I’ll also share some final takeaways from this journey.
