Multi Agent Systems that Ship Software

More code isn’t more software

We have coding agents proposing refactors, generating boilerplate, and opening pull requests faster than humans can review them. Yet product managers still wait for features to land. The gap is not in code generation; it’s in the machinery that turns patches into reliable, deployable changes.

In teams I’ve advised, the common blockers are painfully consistent: acceptance criteria exist only in tickets, not as executable tests; the environment the agent runs in drifts from CI; dependency resolution is non-deterministic; and repo policies are tuned for humans, not automated workers. The result is high churn and low throughput.

I now start from a simple premise: the only definition of “done” an agent will respect is a failing test that turns green. To make that work, we wire three pieces together: a tests-first task graph that plans the work, a hermetic agent sandbox that reproduces results, and repo merge gates that only reward production-quality outcomes. You can add all three to an existing codebase in a week.

A tests-first task graph

Start by expressing the feature as a directed acyclic graph where every edge is justified by a test. The graph is the agent’s plan and the reviewer’s map. Importantly, the roots are acceptance tests that fail today. Unit and integration tests anchor behaviour at narrower levels so generated code has a stable target.

A minimal graph includes: acceptance tests that exercise the user-visible behaviour; unit tests that pin algorithms and error handling; contract tests for any external APIs; and non-functional checks (performance budgets, memory bounds) where regressions hurt you most. The agent can create tests too, but the top-level acceptance tests should be human-authored or human-reviewed to encode intent clearly.

Represent the graph in a file in-repo so it’s versioned with the work. Keep commands boring: shell, pytest, make. Don’t hide domain logic in your orchestration.

Example: feature task graph file

yamlfeature: user-can-export-invoice-pdf
version: 1
workspace: .
tasks:
  - id: acceptance_cli_pdf
    kind: acceptance
    command: pytest -q tests/acceptance/test_export_pdf.py::test_cli_export
    needs: []
  - id: unit_pdf_renderer
    kind: unit
    command: pytest -q tests/unit/test_pdf_renderer.py
    needs: [acceptance_cli_pdf]
  - id: contract_storage
    kind: contract
    command: pytest -q tests/contract/test_storage.py
    needs: [acceptance_cli_pdf]
  - id: integration_generate
    kind: integration
    command: pytest -q tests/integration/test_generate_invoice_pdf.py
    needs: [unit_pdf_renderer, contract_storage]
  - id: lint_and_format
    kind: quality
    command: ruff check . && ruff format --check .
    needs: [integration_generate]
  - id: docs_update
    kind: docs
    command: make -C docs html
    needs: [integration_generate]

You don’t need a sophisticated scheduler to start. A simple wrapper can walk the graph, run each command, capture logs, and mark nodes as green or red. The point is that agents and humans see the same artefacts and the same pass/fail signal. When the acceptance test goes green in a clean environment, it’s ready to propose a pull request.

A hermetic agent sandbox

Agents are sensitive to state. If their environment leaks global config, internet access, or mixed toolchains, you’ll get flakiness and hard-to-reproduce failures. Put them in a sandbox that is: ephemeral per run; pinned to a specific OS image and language toolchain; seeded only with the repository and declared caches; allowed to resolve dependencies via a lockfile; and instrumented for logs, traces and resource limits.

Basic shape: mount the repository read-only; copy it into a writable workspace; create a virtual environment; install from a lock file with hashes; run the graph; emit artefacts and JUnit reports; discard the workspace. Keep secrets out. If an agent truly needs an API token for a contract test, inject a short-lived token via environment variables and scope it to a fixture URL.

You can do this with Docker or any container runtime you already use. The following script is enough to stabilise Python projects. Adapt the idea for Node, Java, or polyglot repos. The important constraints are read-only source, deterministic deps, and a clean, disposable workspace.

Example: spin up a sandboxed agent run

bash#!/usr/bin/env bash
set -euo pipefail
IMAGE=${IMAGE:-python:3.12-slim}
WORKDIR=$(mktemp -d)
trap 'rm -rf "$WORKDIR"' EXIT

docker run --rm \
  -v "$PWD":/repo:ro \
  -v "$WORKDIR":/work \
  -w /work \
  -e PIP_NO_INPUT=1 \
  -e PIP_DISABLE_PIP_VERSION_CHECK=1 \
  "$IMAGE" bash -lc '
    cp -r /repo/. .
    python -m venv .venv
    source .venv/bin/activate
    pip install --upgrade pip
    pip install --require-hashes -r requirements.lock
    pytest -q --junitxml=artifacts/test-results.xml
  '

echo "Artifacts in $WORKDIR/artifacts"

Notes: use a lock file with hashes (pip-compile or Poetry’s lock) to eliminate version drift. If builds must be fully offline, pre-bake a base image with wheels in a local index and point PIP_INDEX_URL at it. To constrain egress during test execution, split dependency install and test run into separate containers and disable networking in the latter.

The design principles behind this are set out in Designing for Agents: When Software Starts to Act.

Multi Agent Systems that cooperate, not collide

With a clean plan and a stable environment, Multi Agent Systems become useful. I generally compose four roles around the task graph:

– Planner: reads the feature file and expands missing nodes (e.g., proposes unit tests for uncovered branches).

– Test author: writes or refines failing acceptance and unit tests, grounded in the repository’s idioms.

– Implementer: edits code to make the tests pass, constrained to the workspace and toolchain.

– Reviewer: comments on diffs, suggests simplifications, and flags unmet non-functional checks.

The orchestrator binds them. It exposes a tiny set of tools: run_task(id), read_file(path), write_file(path), git_diff(), and propose_pr(). It forbids network calls outside the dependency install phase, shelling out to arbitrary binaries, or writing outside the workspace. Each agent gets the same telemetry: task graph state, last command logs, and JUnit results. Concurrency is controlled at the task level: the implementer can’t race ahead of failing acceptance tests.

Two pragmatic details make a big difference: first, persist the workspace and test results for every attempt so humans can inspect failures later; second, bound the implementer’s edit budget per attempt (e.g., limit changed files or total diff size) so reviews stay tractable.

Repo merge gates that reward shipping

If your repository gates are tuned only for human developers, agents will look noisy. Tighten gates around objective quality signals from the task graph: all tests green in CI; coverage delta non-negative in touched packages; lints and type checks clean; dependency diffs limited to whitelisted ranges or lockfile-only; and optional performance checks within budget. Make these conditions visible in the pull request.

Turning this into a prioritised plan is the work of AI strategy advisory: choosing the few use cases worth doing and sequencing them.

CI configuration shouldn’t be exotic. Keep it standard so the same rules apply to humans and agents. Here’s a compact GitHub Actions job you can paste into .github/workflows/gate.yml and extend as needed.

Example: GitHub Actions merge gate

yamlname: gate-pr
on:
  pull_request:
    types: [opened, synchronize, reopened, ready_for_review]
    branches: [main]
concurrency:
  group: pr-${{ github.ref }}
  cancel-in-progress: true
jobs:
  quality:
    if: ${{ !github.event.pull_request.draft }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.12' }
      - name: Install deps (locked)
        run: |
          python -m venv .venv
          . .venv/bin/activate
          pip install --upgrade pip
          pip install --require-hashes -r requirements.lock
      - name: Tests + coverage gate
        run: |
          . .venv/bin/activate
          coverage run -m pytest -q
          coverage report --fail-under=85
      - name: Lint + security
        run: |
          . .venv/bin/activate
          ruff check .
          bandit -q -r src || true

Tie branch protection to this job and a code-owner review. Adjust the coverage threshold and tools to your stack. For performance-sensitive code, add a benchmark step that runs a small, deterministic dataset with a time budget and fails if exceeded. If you must allow occasional waivers, make them explicit via a labelled PR and a human approver; don’t special-case agent authorship.

Observability, drift, and rollback

Agents amplify any ambiguity in your system. If you can’t see what they did and why it failed, you will spend your time spelunking through logs. Instrument the sandbox and the orchestrator with structured logs and spans per task node. At minimum, capture: the exact container image digest; lockfile hash; task id; command; wall time; exit code; and a pointer to artefacts.

Watch for two failure modes in particular. First, environment drift: flakiness that appears only in CI usually means your local sandbox isn’t the same as the runner, or the lockfile allows optional extras to float. Fix by pinning OS-level deps (e.g., image digest, not tag) and using –require-hashes or equivalent. Second, test brittleness: text snapshot tests and time-dependent assertions often become flaky under agent refactors. Stabilise them with normalisation (strip timestamps, sort keys) and fixed seeds.

Make rollback cheap. If an agent lands a PR behind a feature flag, production risk drops. Teach the implementer to wrap new paths behind flags and add acceptance tests for both on and off states. Your merge gates don’t need to know about the flag; they only need to see that both behaviours are exercised and green.

What to do this week

If you want the agents you already have to ship features, here’s a practical seven-day plan:

– Day 1: Pick one feature and write a failing acceptance test that captures the business outcome. Put it under tests/acceptance and make sure it’s runnable in CI.

– Day 2: Add 2–3 unit tests that pin key behaviours the acceptance test will exercise. Add a simple contract test for any external API calls using a stubbed host.

– Day 3: Create a feature.yml task graph like the example above. Check it into the repo next to the tests.

– Day 4: Add the sandbox script and run the graph locally. Fix any dependency pinning or path assumptions until it’s repeatable.

– Day 5: Wire up the GitHub Actions gate with tests, coverage, and lints. Protect main with that job and a code-owner review.

– Day 6: Let your agent (or AI assistant in your IDE) act as the implementer inside the sandbox. Limit the diff size and iterate until green.

– Day 7: Review, merge, and write down what was painful. Turn those notes into more tests or task nodes.

If you’d like a sounding board, I’m happy to act as an Agentic AI Consultant to set up the task graph and sandbox with your team. If you need broader integration into CI/CD and platform guardrails, bring me in as a Solution Architecture Consultant. And if you’re evaluating models, tooling, and prompt interfaces, I provide pragmatic LLM Consulting Services focused on maintainability over novelty.

The point is simple: Multi Agent Systems aren’t a magic wand, but with a tests-first plan, a hermetic runtime, and clear repo gates, they become another reliable tool in your software delivery pipeline.