Multi Agent Systems for reliable QA agents

Why agentic QA now

There’s momentum behind agents that use your application the way a user does. Momentic’s demo aligns with where many teams are heading: let an LLM plan and drive a browser, explore user paths, and surface bugs your scripted tests never considered. But bridging from demo to a dependable QA lane takes engineering discipline. The trick is to adopt Multi Agent Systems thinking without turning your test suite into a research project. Focus on a small, composable loop you can operate and debug.

In this article I’ll lay out a pragmatic, production‑minded blueprint: connect an LLM to a headless browser with guardrails; generate tests and then minimise them to stable failing repros; capture enough state to rerun locally and in CI; and de‑flake with oracles, time control, and golden traces. You can ship a first slice this week alongside your existing Playwright or Selenium tests and iterate from there. No magic, no hype—just careful wiring, explicit trade‑offs, and boring reliability.

Wire the LLM to the browser

The core loop is simple: the LLM proposes a step; your browser driver executes it; you collect observations; the LLM critiques and proposes the next step. Keep the step schema concrete—actions like click, type, select, assert_text—with targets that prefer stable selectors (data‑testids, roles and names) over brittle CSS.

Two important guardrails: first, expose the accessibility tree and any semantic metadata your app already has (ARIA roles, labels, data‑testids). Second, gate actions behind deterministic preconditions (element exists, visible, enabled) with timeouts under your control, not the model’s. Don’t let the model invent selectors; let it choose from a catalogue built from your current DOM snapshot.

pythonfrom playwright.sync_api import sync_playwright

# Minimal step runner with selector guardrails

def find(page, target):
    if 'role' in target:
        return page.get_by_role(target['role'], name=target.get('name'))
    if 'testid' in target:
        return page.locator(f"[data-testid='{target['testid']}']")
    if 'css' in target:
        return page.locator(target['css'])
    raise ValueError('No valid target')

def run_step(page, step):
    action, tgt = step['action'], step.get('target', {})
    el = find(page, tgt) if action in ('click','type') else None
    if action == 'click':
        el.wait_for(state='visible'); el.click()
    elif action == 'type':
        el.wait_for(state='visible'); el.fill(step['text'])
    elif action == 'assert_text':
        page.get_by_text(step['text']).wait_for(timeout=3000)
    else:
        raise ValueError(f"Unsupported action {action}")

# plan_with_llm(goal, dom_catalogue)->[steps] is your LLM boundary (not shown).

Architecturally, keep the LLM boundary thin: you provide a DOM catalogue (roles, labels, testids, URLs) and an explicit goal; the model returns a bounded list of steps as JSON. Run those steps deterministically and stream back observations (errors, screenshots, text snippets) for the next planning round. This separation lets you replace the planner, tune prompts and models, or add retrieval (component docs, feature flags) without touching the runner.

Generate and minimise failing tests

Exploration is useful, but teams need deterministic failing repros. After the agent discovers an issue, immediately switch from exploration to reproduction mode: record the action sequence and relevant inputs, then try to minimise it. Classic delta debugging (ddmin) reduces a long event list to the smallest sequence that still fails. Smaller repros are faster, more portable, and far less flaky.

Treat the app as a function from an event sequence plus state to an outcome. A test harness replays events against a fresh browser context, seeded with the captured state (see next section). ddmin drives the harness until no further reductions fail. Below is a minimal implementation you can paste into your harness; it expects test(trial) to return ‘FAIL’ or ‘PASS’.

pythondef ddmin(seq, test):
    n = 2
    while len(seq) >= 2:
        chunk = max(1, len(seq) // n)
        reduced = False
        i = 0
        while i < len(seq):
            trial = seq[:i] + seq[i+chunk:]
            if test(trial) == 'FAIL':
                seq = trial
                n = max(2, n - 1)
                reduced = True
                break
            i += chunk
        if not reduced:
            if n >= len(seq):
                break
            n = min(len(seq), n * 2)
    return seq

# Example: test is your replay(lambda events: 'FAIL' if bug repros else 'PASS')

In practice I also normalise the sequence before minimisation: canonicalise selectors to testids or roles, collapse equivalent events (e.g., consecutive types into a single fill), and strip non‑semantic waits in favour of explicit preconditions. If a repro only fails with just‑so timings, assume flake by default and keep shrinking or capture more state.

Capture state for reproducibility

A bug report without state is a rumour. When the agent detects a failure, persist a compact, portable bundle: HAR (network), DOM snapshot, storage (cookies, local/session), feature flags, environment variables that affect behaviour, and the app/version hash. The goal is reproducibility on a clean machine and in CI, independent of the explorer’s specific environment.

Prefer Playwright’s built‑in HAR and storage exports. DOM snapshots via page.content() are usually sufficient; pair them with the URL, viewport, and user agent. For server‑side determinism, log upstream API responses that influence the UI, or stub them during repro. Here is a small Playwright snippet that captures the essentials into a ‘repro’ directory:

For the broader question of what changes when software starts to act, see designing for agents.

pythonfrom playwright.sync_api import sync_playwright
import json, time, os

with sync_playwright() as p:
    browser = p.chromium.launch()
    context = browser.new_context(record_har_path="repro/trace.har")
    page = context.new_page()
    page.goto("http://localhost:3000")
    # ... perform steps discovered by the agent ...
    snapshot = {
        "url": page.url,
        "dom": page.content(),
        "localStorage": page.evaluate("Object.fromEntries(Object.entries(localStorage))"),
        "sessionStorage": page.evaluate("Object.fromEntries(Object.entries(sessionStorage))"),
        "cookies": context.cookies(),
        "viewport": page.viewport_size,
        "timestamp": int(time.time())
    }
    os.makedirs("repro", exist_ok=True)
    json.dump(snapshot, open("repro/state.json","w"), indent=2)
    context.storage_state(path="repro/storage.json")
    context.close(); browser.close()

On CI, ingest this bundle to recreate the context before replay: restore storage, set user agent/viewport, serve stubbed network from the HAR (or a curated golden), and then run the minimal event sequence. Keep bundles small (target under a few MB) by excluding large responses unless they affect the failing path.

De-flake with oracles and golden traces

Exploration inevitably triggers flakiness: time, randomness, network, async renders, third‑party widgets. Your first defence is determinism: freeze time (Date.now, timers), seed RNGs, and stub nondeterministic endpoints behind fixtures. If that’s impossible for some flows, at least stabilise assertions.

I use a layered oracle: hard invariants first (element present, HTTP status, arithmetic totals), then semantic checks that allow variability (e.g., discount text may differ but the computed total must match). Screenshot diffs are a last resort—use them for layout regressions only and aggressively mask dynamic regions.

Golden traces push reliability further. Record a curated network trace for a given repro and assert that the live trace matches a whitelist of request shapes: method, path, minimal headers, and response schema. When the UI depends on third‑parties, validate the contract rather than byte‑identical bodies. If you keep one thing from this section: only let the LLM propose where to look; keep the truth conditions purely programmatic and deterministic.

Designing Multi Agent Systems for QA

A single monolithic agent is tempting but brittle. A small Multi Agent Systems design gives you better observability and recovery. I split responsibilities along these lines:

– Orchestrator: owns the goal, budgets tokens/time, and advances the loop.

– Navigator: proposes next steps from the DOM catalogue and recent observations.

If you are running systems like this in production, an AI architecture review tests retrieval, evaluation, permissions and cost before they become incidents.

– Executor: runs steps in the browser with guardrails and emits structured events.

– Observer: summarises observations (DOM diffs, console errors, network anomalies).

– Shrinker: minimises event sequences when a failure is detected.

– Reporter: packs bundles, deduplicates similar failures, and posts to your tracker.

They communicate via a simple event bus (an append‑only log is fine). Each agent is restartable and stateless beyond the log offset it consumes. Keep prompts tiny and role‑specific. For example, Navigator only sees: goal, current URL, a pruned DOM catalogue (top‑K elements by salience), and the last N observations. Observer converts raw artefacts into compact facts: ‘XHR POST /cart returned 500’, ‘button[Checkout] disabled: true’.

Failure modes are then explicit: if the Navigator loops, the Orchestrator times out and asks the Shrinker to repro what it has; if the Executor hits a flaky selector, it falls back from role/name to testid to CSS with a penalty signal the Navigator can learn from. You can scale this to multiple browsers or devices by sharding the event bus. Crucially, any agent can be replaced, upgraded, or turned off without collapsing the whole system.

Ship it this week: a step-by-step plan

Day 1: Add stable selectors. If you don’t already expose data‑testids or strong roles/labels across critical flows, do that now. Without this, the agent will learn bad habits and you’ll be chasing selectors.

Day 2: Build the thin LLM boundary. Implement a ‘plan_with_llm’ function that accepts a DOM catalogue and a goal and returns a small list of steps. Start with a single model and a tiny prompt; resist the urge to add memory and tools before you can rerun one failing repro end‑to‑end.

Day 3: Integrate the step runner and basic repro capture. Use Playwright or Selenium to execute steps with guardrails. When a failure occurs, capture HAR, DOM, storage, and viewport, and dump the action list and inputs.

Day 4: Add ddmin and oracles. Minimise failing sequences. Replace brittle text assertions with invariants on network responses and computed values. Freeze time and stub the noisiest endpoints for the repro path.

Day 5: Wire CI and reporting. Reproduce failures in CI from bundles. Post minimal repros to your issue tracker with links to artefacts. Triage by deduping similar network stacks or DOM diffs rather than titles.

Optional: split responsibilities into small agents if you need throughput or isolation, but only after you’ve stabilised a single‑agent path. From there, layer on exploration strategies (random goals, sitemap crawling, property‑based input generation) behind strict budgets.

If you’d like experienced help stitching this into your stack, speak to an Agentic AI Consultant who has shipped browser automation to CI before, or a Solution Architecture Consultant to align the artefacts with your release and on‑call processes. And if your team is early in model selection and prompt design, the right LLM Consulting Services can accelerate the planner/oracle boundary without overcomplicating it.

This blueprint doesn’t replace your handcrafted tests; it complements them. Aim for fast, boring repros first; scale exploration later. The moment your agent can hand you a one‑click failing test that runs green when fixed, you’ll know you’ve crossed from demo to dependable engineering.