Reworking CI for AI Teams with Multi Agent Systems

The bottleneck shift: AI coding, Multi Agent Systems and CI

Code is arriving faster than CI can clear it. Pair a human with an LLM and they’ll easily open more, smaller pull requests; drop in a couple of autonomous refactor agents and the change velocity goes up another notch. Multi Agent Systems also widen the surface area of concurrent edits: configuration, tests, and multiple services touched together. If your pipelines were built for one or two big PRs a day, they’ll crumble under dozens of small PRs an hour.

The failure mode is familiar: queue times spike, feedback drifts from minutes to hours, and developers either batch changes (riskier) or push directly to main (riskier still). Compute spend rises with little impact on cycle time because the real enemy is serialisation and cold starts, not raw CPU.

I reworked our CI/CD with five pillars: aggressive sharding, pragmatic test impact analysis, speculative builds with a strict merge queue, distributed caches, and auto-merge guards that keep flakiness out of main. The goal is not theoretical perfection; it’s steady, predictable throughput and fast feedback under load. What follows is the blueprint we ended up with, including the trade-offs that matter.

What ‘fast enough’ looks like: SLOs and cost guardrails

Before changing anything, set service-level objectives for CI. Without clear targets, you’ll optimise the wrong things.

– PR feedback SLOs: p50 end-to-end under 10 minutes, p95 under 30 minutes. Include queue time, setup, build, tests, and artefact upload. If your baseline is far off, aim for stability first, then squeeze speed.

– Merge gate SLOs: entering the merge queue to merged-on-main under 20 minutes at p50 in steady state. Spikes are acceptable during large refactors, but the queue must drain predictably.

– Flake budget: less than 1% of PRs should rerun due to flaky tests. Anything flaky is quarantined automatically and tracked to closure.

– Cost cap: set a per-PR compute budget (e.g., ‘do not exceed X runner-minutes on average’). It’s a control loop: when budget is hit, trim work by tightening test impact analysis or scaling back speculative fan-out.

Instrument these with: structured step timings, queue depth, cache hit rate, cold-start rate, and per-job cost (runner minutes times cost per minute). Emit them to your existing observability stack so product and platform share the same ground truth. You can’t tune what you can’t see.

Shard everything that can be sharded

Parallelism beats big machines. The fastest lever is to shard test and build work across many small runners and keep each shard consistent and short. Two points matter most:

– Balance shards by expected duration, not test count. Use historical timings to bucket tests so each shard ends near the same runtime.

– Keep shards hermetic. Each shard should fetch only the data and dependencies it needs and write results to a shared artefact store, not to a mutable shared disk.

Static sharding is a fine starting point; you can move to time-aware sharding once you’ve collected timings. Below is a minimal GitHub Actions example splitting unit tests across eight shards, using a tiny helper script to feed pytest only the tests allocated to that shard.

yamlname: ci
on: [pull_request, merge_group]
jobs:
  tests:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [0,1,2,3,4,5,6,7]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install -r requirements.txt
      - name: Run shard
        run: ./ci/shard-tests.sh ${{ matrix.shard }} 8 | xargs -r pytest -q --maxfail=1

The shard script collects tests and deterministically assigns them by modulo. Later, you can replace this with a timing-aware distributor that rebalances based on recorded durations.

bash#!/usr/bin/env bash
set -euo pipefail
IDX=${1:-0}; TOTAL=${2:-1}
# Collect all test node IDs and split by modulo.
pytest --collect-only -q | awk '/::/{print $1}' | nl -v0 | awk -v i=$IDX -v t=$TOTAL '($1%t)==i{print $2}'

Trade-offs to mind: – Over-sharding inflates cold start time and cache misses. There’s a sweet spot where each shard runs 3–8 minutes after warm-up. – If your stack is polyglot (e.g., Python + Node + JVM), shard by language first, then by package, then by tests. – Keep fail-fast off at the matrix level; failing one shard shouldn’t cancel the rest, or you’ll lose useful signal and waste re-runs.

Do the same for builds: compile native code with sccache on each shard, bundle front-end packages in parallel, and split container image builds by service with layer caching enabled.

Test impact analysis that actually works

Test impact analysis (TIA) is your compute budget’s best friend if done carefully. The principle is simple: only run the tests that could be affected by the change. The implementation is where many teams go astray.

You need a dependency graph from source files to tests. There are three pragmatic ways to get it: – Coverage-derived mapping: periodically run full coverage and record which tests touch which files. Map source files to test files from the coverage database. – Static graph: for monorepos, use build graph tools (Bazel, Buck) or workspace tools (Nx, Turborepo) to compute impacted projects and their tests. – Heuristics: fallback to directory-based rules (e.g., src/foo maps to tests/foo) and extend with tags/markers for cross-cutting tests.

Here’s a small Python helper that looks at the diff versus main, consults a precomputed mapping, and prints impacted tests or a smoke suite as a safe default. It outputs test paths that you can pipe into pytest or your test runner of choice.

pythonimport json, subprocess, os

# Diff against main; adjust for your default branch
changed = subprocess.check_output([
    "git", "diff", "--name-only", "origin/main...HEAD"
]).decode().splitlines()

with open("ci/tests_map.json") as f:
    graph = json.load(f)  # {"src_to_tests": {"path.py": ["tests/test_x.py::TestY::test_z", ...]}, "smoke": [...]}

impacted = set()
for fpath in changed:
    for t in graph.get("src_to_tests", {}).get(fpath, []):
        impacted.add(t)

if not impacted:
    impacted = set(graph.get("smoke", []))

print("\n".join(sorted(impacted)))

Operationally: – Treat TIA as an optimisation, not a gate. Run impacted tests on PRs, then a full test run overnight to catch any misses and refresh the mapping. – Cache the mapping artefact so it’s cheap to load; rebuild it on the main branch after full coverage runs. – Make it observable: record the ratio of impacted-to-full tests and the escape rate (tests that only fail in the nightly full run). If escapes rise, widen the impact rules.

If you’re on a JS-heavy monorepo, start with Nx ‘affected’ for a quick win: it’s pragmatic and well-trodden. Layer coverage-derived mapping later if you need granular test selection across packages.

Speculative builds, merge queues and auto-merge guards

Speculative builds give developers rapid signal by starting CI early and often, while a merge queue ensures only green, up-to-date commits land on main. The recipe that has held up well:

– Run the same checks on pull_request and on merge_group (or your platform’s equivalent). The merge queue revalidates the exact commit that will land, serialising merges through a guarded path.

– Use concurrency groups to cancel superseded runs. Every new push cancels the previous PR run so you don’t waste runners on obsolete commits.

– Require merge-queue checks in branch protection. Do not allow direct merges from PR status alone.

Here’s a minimalist Actions workflow: it runs on PR updates and in the merge queue, cancels in-flight duplicates, and reuses your TIA entry point for speed.

yamlname: verify
on:
  pull_request:
  merge_group:
jobs:
  build-test:
    runs-on: ubuntu-latest
    concurrency:
      group: ${{ github.workflow }}-${{ github.ref }}
      cancel-in-progress: true
    steps:
      - uses: actions/checkout@v4
      - run: ./ci/run-impacted.sh  # calls TIA then sharded test/build scripts

Auto-merge guards worth adopting from day one: – Flake quarantine: if a test fails and passes on a single retry, tag it @quarantine and file an issue automatically. Quarantined tests still run on nightly full builds but do not gate PRs. – Retry budget: allow one retry per shard only; repeated retries hide real problems and waste compute. – Dependency freeze window: freeze lockfile updates during peak hours so merge-queue churn doesn’t cascade across PRs. – Size caps: require explicit approval for PRs above a change-size threshold; large changes often swamp caches and create queue spikes.

A final word on behaviour: speculative builds do not mean speculative merges. Keep the gate strict, the queue narrow, and the rules simple. Developers will adapt quickly if the system is fast and predictable.

Distributed caching and hermetic artefacts

Caching is where most of the real-world speed comes from once you’ve sharded. There are three layers to treat explicitly:

1) Dependency caches: language package caches for pip, npm, Maven/Gradle. Key them on the lockfile content and the toolchain version, not on time. Evict with a TTL but prefer content addressing; it avoids stampedes after minor changes.

2) Build caches: compiler and bundler caches such as sccache/ccache for C/C++, and incremental caches for TypeScript/Webpack/esbuild. For sccache, point all shards to a shared object store (S3, GCS, or MinIO). Key includes compiler version, flags, and input hashes. Watch out for cache poisoning; lock compiler versions per branch or per image.

3) Remote build graph caches: if you can adopt Bazel or Buck for the heaviest parts, do it for the cache alone. A good remote cache turns cold starts into warm ones and enables cross-shard reuse. Keep cache writes allowed from main builds only (to reduce inconsistent writes), and PRs read-from but do not write-to the shared cache.

Hermeticity matters. Build and test containers should embed exact toolchain versions and not reach the public internet during steps; all network fetches should go via a proxy cache you control. This both speeds things up and makes outcomes repeatable.

Artefacts: upload test results, coverage reports, and build outputs with content hashes. Downstream jobs should fetch by hash, not by job ID. This allows reusing artefacts between PR and merge-queue runs when the commit is the same, further reducing duplicate work.

Operational guardrails: – Track cache hit rate per job. If it drops suddenly, look for accidental key changes (toolchain updated, path differences, environment variables leaking into keys). – Cap cache size and enforce eviction policies. Slow caches are worse than no caches. – Pre-warm: periodically populate hot cache entries for critical packages and compile units; it’s often cheaper than waiting for the first PR of the day to do the work.

What to do next

If your CI is already creaking under AI-accelerated change volume, don’t attempt a big-bang rewrite. Sequence changes so each step buys you time and data:

Day 1–3: – Add concurrency groups to cancel superseded PR runs. – Introduce a basic shard for your slowest test suite (start with 4–8 shards). – Enable dependency caching keyed by lockfiles. – Turn on a merge queue and require its checks for main.

Day 4–7: – Add a simple TIA: directory-mapped or coverage-derived if you have it. Run impacted tests on PRs, full suite nightly. – Push build/test timeouts and retry budgets into code so they are versioned with the repo. – Start capturing timings per test and per shard; store them as artefacts for analysis.

Week 2–3: – Balance shards by historical duration. Aim for tight runtime spread across shards. – Introduce sccache/ccache or a Bazel remote cache for heavy builds. – Quarantine flaky tests automatically and track their closure as a metric.

Month 1: – Move PRs to fully impacted runs with nightly full verification; keep escape rates visible. – Optimise runner images: slim base, preinstall toolchains, and isolate secrets. – Set and publish your CI SLOs. Invest in alerts for breach of p95 targets and merge-queue backlog growth.

Throughout, measure cost per PR and per merged commit. If it trends up without a corresponding gain in speed or stability, stop and reassess your cache keys or sharding level before throwing more hardware at the problem.

If you’d like a hands-on review or a tailored blueprint, this is the kind of work where an external perspective helps. I’m happy to engage as an Agentic AI Consultant to align your developer workflows with your AI tooling, or as a Solution Architecture Consultant to evolve your build graph and environments. If your bottleneck is upstream of CI—prompt engineering, evaluation harnesses, or model-in-the-loop tests—LLM Consulting Services can help you instrument and stabilise those paths so CI has a clean target to verify.

The teams that win with AI aren’t just faster at writing code; they are faster at proving it’s safe to ship. Make CI a product with clear SLOs, and design for the reality of agents and assistants hammering your repo. Technology made practical.