Multi Agent Systems done right: no rogue agents

Stop blaming “rogue”; start bounding capability

The industry loves a good “rogue agent” headline. It’s tidy, it’s dramatic, and it avoids the awkward truth: agents act within the latitude we grant them. If they email the wrong person, it’s because we exposed an unconstrained email tool. If they scrape the web indefinitely, it’s because we forgot to cap loops, time or costs. In other words, there are no rogue agents—only unbounded designs.

Large vendors are now productising guardrails and runtime controls. That’s welcome. But you don’t need to wait for a platform release to ship safer agents. You can implement the building blocks today with ordinary software engineering: least-privilege tools, plans-as-code, budget governors, deterministic tool-review middleware, and simulation tests. These work for single agents and scale even better for Multi Agent Systems, where coordination can otherwise amplify small mistakes.

Treat agents as untrusted, probabilistic programs. Give them only the tools they need, with narrow scopes. Separate “decide” (the model) from “do” (your code). Force them to propose a plan you can statically inspect. Charge them for time, tokens and money. Route every tool call through a review layer. Then attack the system in a simulator until it breaks in the lab instead of production.

Capability-scoped tools, not carte blanche

Tools are your hard power. If a tool can do anything, the agent can do anything. Define tools with explicit, enforceable scopes. Think: fetch_url but only for whitelisted domains; write_file but only to a per-request sandbox; send_email but only to addresses generated from a fixed allow-list; shell access? not in production, ever.

Concretely, route all tool invocations through a registry that enforces policy before execution. Policies should be code, not prompts, and deterministic where possible. Here’s a minimal Python sketch:

pythonfrom dataclasses import dataclass
from typing import Callable, Dict, Any, Set
from urllib.parse import urlparse
import os, requests

@dataclass
class Tool:
    name: str
    func: Callable[[Dict[str, Any]], Any]
    policy: Callable[[Dict[str, Any], Dict[str, Any]], None]

class ToolRegistry:
    def __init__(self):
        self.tools: Dict[str, Tool] = {}
    def register(self, tool: Tool):
        self.tools[tool.name] = tool
    def call(self, name: str, args: Dict[str, Any], ctx: Dict[str, Any]):
        tool = self.tools[name]
        tool.policy(args, ctx)  # raise if disallowed
        return tool.func(args)

# Example tools
fetch = Tool(
    name="fetch_url",
    func=lambda a: requests.get(a["url"], timeout=10).text,
    policy=lambda a, c: (
        (_ := urlparse(a["url"]).hostname) in c["allowed_domains"]
        or (_ for _ in ()).throw(PermissionError("domain not allowed"))
    ),
)
write = Tool(
    name="write_file",
    func=lambda a: open(a["path"], "w").write(a["content"]),
    policy=lambda a, c: a["path"].startswith(c["work_dir"]) or (_ for _ in ()).throw(PermissionError("path not allowed"))
)

registry = ToolRegistry(); registry.register(fetch); registry.register(write)
ctx = {"allowed_domains": {"docs.example.com"}, "work_dir": f"/tmp/agents/req_123/"}
# registry.call("fetch_url", {"url": "https://docs.example.com/guide"}, ctx)

A few rules of thumb: one tool per capability; scope via code, not via model instructions; include input validation; make tools idempotent where possible; and tag every tool call with request_id and actor for audit trails.

Plan-as-code beats vibes

Relying on a model’s free-form chain-of-thought to navigate complex tasks invites drift. Instead, force a structured plan: a sequence (or DAG) of steps with explicit inputs, required tools, preconditions, approvals and expected artefacts. Plans are data, so you can render them, diff them, and validate them—just like infrastructure-as-code.

For the broader question of what changes when software starts to act, see designing for agents.

A simple YAML schema is enough to start. The agent proposes a plan, you (or a policy) validate it, and then an executor runs it step by step under budget and policy controls. Example:

yamlversion: "1"
objective: "Summarise onboarding docs and email the team"
budget: {tokens: 12000, usd: 0.50, seconds: 90}
steps:
  - id: fetch
    tool: fetch_url
    args: {url: "https://docs.example.com/onboarding"}
    expect: {artifact: "raw_docs"}
  - id: summarise
    tool: llm_summarise
    args_from: {text: "raw_docs"}
    expect: {artifact: "summary_md"}
  - id: email
    tool: send_email
    approvals: ["human"]
    args_from: {body: "summary_md", recipients: "onboarding_list"}
policies:
  allowed_domains: ["docs.example.com"]
  write_dir: "/tmp/agents/req_123/"

You can reject plans that reference unknown tools, violate domain/dir policies, exceed budgets, add unapproved external calls, or skip required approvals. If a plan changes at runtime, force a re-validation rather than silently executing the delta.

Budget guards and rate governors

Runaway loops aren’t malice; they’re missing limits. Enforce budgets in three dimensions: tokens (LLM usage), currency (API/cloud spend), and wall-clock time. Charge every model call and tool invocation against the budget and stop cleanly when exhausted. Couple this with rate limits to avoid API back-pressure turning into failure storms.

Keep the guard deterministic and close to where spend happens. Here’s a compact pattern:

pythonimport time

class BudgetExceeded(Exception): ...

class BudgetGuard:
    def __init__(self, tokens:int, usd:float, seconds:int):
        self.tokens=tokens; self.usd=usd; self.deadline=time.time()+seconds
    def charge(self, tokens=0, usd=0.0):
        self.tokens -= tokens; self.usd -= usd
        if self.tokens < 0 or self.usd < -1e-9 or time.time() > self.deadline:
            raise BudgetExceeded("budget exhausted")

# usage
budget = BudgetGuard(tokens=12000, usd=0.50, seconds=90)
# before each LLM/tool call:
# budget.charge(tokens=512, usd=0.01)

Design trade-offs: tighter budgets improve safety and predictability but reduce exploration; per-step budgets introduce overhead and can starve later steps if under-estimated; global budgets are simpler but make attribution harder. In production, persist spend per request and surface it in your observability stack.

Enterprise AI strategy support helps leadership teams decide where AI belongs before the tooling decisions are made.

Tool-review middleware and human gates

Between the agent’s intent and your tool execution, insert a deterministic review layer. It should: validate schema and argument types; enforce per-tool policies (domain allow-lists, directory prefixes, size limits); apply budget charges; redact secrets; record structured logs; and optionally require approvals (automatic or human) for sensitive actions.

Make approvals explicit in the plan schema. A simple rule: anything that can reach the outside world (email, payments, tickets) gets an approval step at first. You can later auto-approve common low-risk patterns by encoding them as rules. Avoid LLM-in-the-loop for the reviewer unless you need semantic judgements; favour deterministic code for security decisions.

Implement review as middleware so it’s impossible to call a tool without passing through it. If your runtime is event-driven, the middleware is a topic or queue that all tool requests must traverse. If it’s function-call driven, it’s a wrapper with no escape hatches. Either way, keep a complete audit trail: who asked, what they asked, what policy matched, what was executed, what it cost.

Designing Multi Agent Systems with bounded roles

Multi Agent Systems amplify both value and risk. Coordination, shared memory and parallel tools can speed throughput, but they also multiply the surface area for mistakes. The cure is the same: bound each role, then orchestrate explicitly.

A pragmatic MAS pattern I ship: – Orchestrator: owns the plan and budget, assigns steps to roles, and enforces global policies. – Roles: single-purpose agents with minimal tools (e.g., Researcher can fetch_url and summarise; Writer can draft; Reviewer can compare output to acceptance criteria). No role has all tools. – Shared artefacts: a typed store (e.g., S3 + metadata or a vector DB) for passing documents by reference, not by guesswork in prompts. – Event bus: all tool calls and results flow through the same middleware for policy checks and spend accounting. – Sandboxes: per-request working directories or containers; never share mutable state across requests. – Timeouts and circuit-breakers: each role and the orchestrator should fail fast and emit actionable errors, not silent retries forever.

An independent AI architecture check is the practical next step when these risks apply to your own systems.

Trade-offs: more roles add latency and coordination overhead; too few roles force broader tool scopes. Start with two or three clear roles and iterate. Resist “self-healing” agents that rewrite their own plans without validation; prefer proposing a delta, then re-validate. Finally, make cross-agent budgets explicit: split the global budget across roles so a chatty researcher can’t starve the writer.

Simulation tests and what to ship next week

If you don’t attack your agent, your users will. Build a simulation harness that runs plans end-to-end against fake environments (mock network, mock filesystem, mock SaaS APIs). Write tests for the failure modes you fear: over-broad URLs, path traversal, email to unapproved domains, infinite loops, budget exhaustion, hallucinated tools, and inconsistent artefacts.

Keep the harness lightweight: in Python, a few fakes and pytest are enough to start. Here’s an example that proves a domain policy actually bites:

pythondef test_policy_blocks_disallowed_domain():
    calls = []
    def fake_fetch(args):
        calls.append(args["url"]); return "ok"
    def policy(args, ctx):
        from urllib.parse import urlparse
        host = urlparse(args["url"]).hostname
        if host not in ctx["allowed_domains"]:
            raise PermissionError("domain not allowed")
    registry = type("R", (), {"tools": {}})()
    registry.tools["fetch_url"] = type("T", (), {"policy": policy, "func": fake_fetch})()
    ctx = {"allowed_domains": {"docs.example.com"}}
    # allowed
    registry.tools["fetch_url"].policy({"url": "https://docs.example.com/x"}, ctx)
    # blocked
    try:
        registry.tools["fetch_url"].policy({"url": "https://evil.example.com/x"}, ctx)
        assert False, "should have raised"
    except PermissionError:
        assert True

What to ship next week: – Wrap every tool in a registry with code-enforced scopes. Remove any direct calls from agent code to the outside world. – Introduce a plan-as-code schema. Make your agent propose a plan, validate it, and only then execute. – Add a BudgetGuard around LLM and tool calls. Emit spend metrics to logs. – Insert tool-review middleware with at least: schema checks, allow-lists, path guards, size limits, audit logs. – Write five simulation tests that must pass before deploy. – If you run multiple roles, allocate budgets per role and verify cross-role artefact passing. If you want another set of eyes on the design, involve a Solution Architecture Consultant to thread policies through your stack, or an Agentic AI Consultant to tune your role and tool boundaries. And if you’re exploring model selection and prompt shapes for these patterns, a Generative AI Consultant can help you keep model behaviour predictable under the constraints you’ve set.

Related: Shopify AI agents checkout: a CTO’s hardening plan and Enterprise AI Agents After Amazon vs Muse: Build Resiliently.