Enterprise AI Agents for Code Review

After Postgres 19: volume beats teams

When a mature project like Postgres pauses a release because AI reviewers produced more findings than humans could triage, it’s a reminder that raw detection capacity is not the bottleneck. People and process are. Models are good at enumerating possibilities; teams are good at making trade-offs under pressure. If we let models set the tempo, we get backlog growth and morale problems; if we shape the tempo, we get calmer delivery and fewer defects escaping.

This piece isn’t about whether AI reviewers are “good”. They’re already useful at surfacing risky edge cases and consistency problems you might otherwise miss. The question is how to deploy them so they help you ship, rather than help you stall. My answer: treat the reviewer like a chatty junior engineer—useful with the right guardrails. Start with precision, deduplicate relentlessly, score risk realistically, and make CI gates that encourage improvement without punishing progress.

Taming Enterprise AI Agents in code review

I group the moving parts into four layers:

– Collection: static tools, unit tests, and one or more AI reviewers produce candidate findings.

– Aggregation: a small service merges, fingerprints, and deduplicates findings across tools and commits.

– Prioritisation: a risk model converts raw findings into severities based on impact and context.

– Enforcement: CI gates and chat/issue workflows apply budgets and route work to owners.

Two cross-cutting constraints matter from day one:

– Data hygiene: keep only what you need. Store fingerprints, snippets (minimised), and metadata; avoid shipping whole repositories to third parties. Redact secrets and tokens at source.

– Human feedback: every suppression, escalation, or fix should feed back to the reviewer prompt or allow/deny lists. The system must get quieter over time, not louder.

If you can implement those constraints, the rest becomes a set of pragmatic rules rather than a compliance nightmare. The aggregation service can be a lightweight process with a Postgres (the irony isn’t lost on me) or SQLite store and a message queue (SQS, Pub/Sub, or Redis Streams). Keep it simple and observable.

Precision-first triage: start narrow, earn trust

The usual mistake is to maximise recall early. Don’t. Start by maximising precision, then widen. I do it in phases:

Phase 0 (dry-run): annotate PRs only, no issues or failing builds. Collect acceptance data: how often reviewers mark a comment as helpful, inactionable, or incorrect. Keep comments ephemeral and thread-level to avoid clutter.

Phase 1 (precision gate): set a confidence threshold and an allow-list of rule categories. Security sinks, resource lifetime bugs, SQL parameterisation, and unsafe concurrency constructs are fair starting points. Style, micro-optimisations, and speculative refactors can wait.

Phase 2 (limited enforcement): allow the gate to fail a build only for new, high-confidence, high-severity findings in the diff. Everything else becomes an optional review task.

To wire this into CI, you don’t need a vendor dependency. A small script that reads the diff and the AI reviewer’s output, then enforces a threshold, is enough.

Deduplication that holds the line

Multiple engines will often describe the same underlying problem in slightly different words. Even one engine may report the same issue across commits as the line numbers shift. Without dedup, volume balloons and credibility drops.

I use a fingerprint scheme: rule_id + normalised file path + a stable slice of the code (e.g., the AST or a whitespace-stripped window) + a normalised message stem. For near-duplicates (same sink/source within a few lines), I cluster with a locality-sensitive hash and keep one canonical finding per cluster.

You also need temporal dedup: if the same fingerprint appears in consecutive commits and the code hasn’t changed meaningfully, suppress re-reporting unless severity or confidence rises. That keeps PRs readable and dashboards calm.

For how to weigh hosted, dedicated and private models, see when a private LLM is worth it.

yamlname: ai-review
on:
  pull_request:
    types: [opened, synchronize, reopened]
jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Run AI reviewer (dry-run)
        run: |
          python3 scripts/ai_review.py --diff > ai-findings.json
      - name: Gate on precision and severity
        env:
          MIN_CONFIDENCE: "0.85"
          MAX_NEW_HIGH: "0"  # block if any new High
        run: |
          python3 scripts/triage_gate.py \
            --findings ai-findings.json \
            --diff-base origin/${{ github.base_ref }} \
            --min-confidence $MIN_CONFIDENCE \
            --max-new-high $MAX_NEW_HIGH

Severity scoring that mirrors risk

Raw model confidence is not severity. I’d rather ship with ten medium-confidence style nits unaddressed than one high-likelihood data-loss bug. My scoring model is simple and explainable:

– Base severity: from the rule category (e.g., injection sinks, concurrency hazards, unsafe deserialisation).

– Change proximity: whether the finding touches modified lines or code owners’ boundaries in this PR.

– Blast radius: file/service criticality, based on runtime ownership and past incident tags.

– Test cover: if the affected code lacks tests that execute the branch, raise severity one notch.

– Churn factor: if the file churns frequently, downgrade a notch unless it’s security-critical (we’ll touch it again soon).

The output is a discrete level (Info, Low, Medium, High, Critical) and a budget weight. CI then applies budgets to new High/Critical items in the diff, not to historical debt elsewhere.

typescriptimport { execSync } from 'node:child_process';
import fs from 'node:fs';

type Finding = {
  fingerprint: string;
  file: string;
  line: number;
  rule: string;        // e.g., SQL_INJECTION_SINK
  confidence: number;  // 0..1
  in_diff: boolean;
  covered: boolean;    // test executes this path
  critical_asset: boolean; // e.g., payment, pii
  churn: number;       // edits in last 90 days
};

function base(rule: string): number {
  const m: Record<string, number> = {
    SQL_INJECTION_SINK: 4,
    UNSAFE_DESERIALISATION: 4,
    DATA_RACE: 3,
    TAINT_PROPAGATION: 3,
    STYLE_NIT: 0,
  };
  return m[rule] ?? 1;
}

function score(f: Finding): number {
  let s = base(f.rule);
  if (f.in_diff) s += 1;
  if (!f.covered) s += 1;
  if (f.critical_asset) s += 1;
  if (f.churn > 20 && s < 4) s -= 1; // we’ll touch it again soon
  if (f.confidence < 0.7) s -= 1;
  return Math.max(0, Math.min(5, s));
}

const findings: Finding[] = JSON.parse(fs.readFileSync('ai-findings.json', 'utf8'));
const scored = findings.map(f => ({ ...f, severity: score(f) }));
fs.writeFileSync('ai-findings.scored.json', JSON.stringify(scored, null, 2));

You can tune those weights with team feedback. The important part is determinism and explainability—engineers should be able to predict whether a PR will pass. If they can’t, they’ll disable the check at the first opportunity.

CI gates that raise quality, not the backlog

My CI policy rules, in order of strictness:

– Never fail a build for historical debt unless the PR touches the code that hosts it.

– Block on any new Critical finding in the diff; block if the count of new High findings exceeds a small budget (often zero).

– Warn (don’t block) on Medium findings unless the team has unused budget for the sprint (make this visible).

An independent AI architecture check is the practical next step when these risks apply to your own systems.

– Auto-suppress duplicates across commits; re-open only if severity or confidence rises or the code meaningfully changes.

If you need a stable schema for findings, keep it minimal and future-proof. A simple JSON structure is enough for most pipelines and easy to map into SARIF later if you want code-scanning UIs.

json{
  "fingerprint": "b8a1c0e2-...",
  "title": "Possible SQL injection in order lookup",
  "rule": "SQL_INJECTION_SINK",
  "file": "src/orders/dao.ts",
  "line": 142,
  "severity": "High",
  "confidence": 0.92,
  "in_diff": true,
  "cluster": "sql-sink::src/orders/dao.ts@130-160",
  "status": "open",   
  "owner": "@team-orders",
  "expires_at": "2026-12-31T00:00:00Z"
}

A few operational notes I’ve learned the hard way:

– Use CODEOWNERS to route High/Critical findings to the right maintainers, and label PRs accordingly. Don’t farm everything to a platform team.

– Don’t email people about new findings; send thread-level PR comments or a concise Slack summary in the service channel.

– Suppressions should expire. If something stays suppressed for months, either fix it or document the exception in code.

– Record acceptance signals (helpful/unhelpful) so the reviewer prompt and allow-lists evolve.

What to do next

If you’re about to enable an AI reviewer, or you’ve already been burned by alert floods, here’s a practical path:

Week 0–1:

– Choose a pilot repo with active maintainers and good tests. Enable the reviewer in dry-run mode (annotations only).

– Implement the minimal aggregation script and finding schema. Store to a simple database and retain only fingerprints and small snippets.

– Define your first allow-list of rule categories (security sinks, concurrency hazards, unsafe resource lifetimes).

Week 2–3:

– Add deduplication by fingerprint and near-duplicate clustering. Suppress re-emits across commits unless severity/confidence rises.

– Wire the severity scorer and make its weights explicit. Share examples with the team and gather feedback.

– Start a soft gate: block only on new Critical in the diff. Everything else is warn-only.

Week 4–6:

– Expand to two or three more repos. Keep the same rules. Track precision (accepted vs rejected annotations), mean time-to-triage, and block rate.

– Add budgets for High severity per PR (often zero). Publish a weekly summary: new High/Critical fixed, reopened, suppressed.

– Introduce expiring suppressions and CODEOWNERS routing for High/Critical.

Week 6+:

– Tune for recall in selected categories if precision remains strong. Avoid adding style rules until engineers ask for them.

– If you need to scale, consider a tiny service for aggregation and scoring with a queue in front, backed by Postgres and a simple API. Keep it observable (latency, backlog depth, duplicate rate, acceptance rate).

A final word on posture: there’s no prize for the biggest list of findings. Teams win when they ship safely and sleep at night. Model capacity is plentiful; focus yours on the few issues that matter, at the right moment in CI, with predictable rules.

If you’d like outside eyes on your setup, an Agentic AI Consultant can help shape the reviewer’s prompts, rules, and feedback loops with your developer culture in mind. For larger programmes, you may want LLM Consulting Services to benchmark precision and define an adoption plan that avoids the Postgres-style flood. And if you’re re-architecting pipelines, a seasoned Solution Architecture Consultant can turn the aggregation, scoring, and gating design into something maintainable alongside your existing CI.

Technology made practical is rarely glamorous. In this space, that translates to boring pipelines, defensible scores, short feedback loops, and a gate that every engineer understands. Do that, and your AI reviewer will feel like a multiplier, not a megaphone.