Enterprise AI Agents: Code Isn’t Software

More code isn’t more software

Coding agents can now open pull requests, run tests, and respond to reviewer comments. It feels like leverage, until you look at what makes it to production. Many teams report the same pattern: repositories fill up, review queues bloat, and the number of deployments or shipped features hardly moves. The distinction is simple but brutal — code is a cost until it reaches customers safely and starts working for the business.

Agents are excellent at routine synthesis: stitching APIs together, drafting boilerplate, and patching obvious issues. They are less consistent at understanding the intent behind a change, the non-functional constraints, and the trade-offs that make software valuable. Without a firm way to measure delivery outcomes, teams end up managing the visible thing — code volume — rather than the valuable thing — working software in production.

We need to reset the goal. Success is fewer handoffs, faster cycle time, lower change failure rate, and stable operations after release. That is what the DORA metrics capture, and what escaped-defect rates illuminate. If agents move those needles in the right direction, they belong. If they don’t, they are noise, no matter how impressive the PR feed looks.

Measure what matters: DORA and escaped defects

When I introduce AI into engineering workflows, I start by baselining four measures: deployment frequency, lead time for changes, change failure rate, and mean time to restore. These DORA metrics define delivery performance. I add escaped-defect rate — issues discovered post-release — because it links speed to safety. If you only track throughput, you will reward risk-taking; if you only track quality, you’ll strangle flow. The point is balance.

Before any rollout, tag and attribute agent-originated work. Label their pull requests, commits, and pipelines. Instrument dashboards that separate human-only, agent-assisted, and agent-originated changes. Try to run small, time-boxed experiments in parallel with a control group, or at least compare to a stable historical baseline. This avoids attributing seasonal fluctuations to the agent.

Normalise your metrics. Compare like-for-like services, similar change sizes, or equalised ticket classes. Don’t fall into vanity measures — tokens consumed, lines of code, number of PRs. Those say more about how busy the system is than how effective it is. If agents are helping, you will see shorter lead times, stable or falling change failure rate, and flat or improving mean time to restore. If they’re not, the metrics will tell you without argument.

Where Enterprise AI Agents help (and where they don’t)

In my experience, agents pull their weight when the work is well-bounded, specs are tight, and correctness can be verified automatically. Good candidates include: generating or updating tests; codifying repetitive API clients; writing migration scripts and configuration changes; refactoring with strong test coverage; building scaffolding for new services; and producing developer documentation tied to code.

Before acting on any of this, test whether the organisation is ready: The AI Readiness Assessment: 20 Questions Before You Build sets out the questions.

They struggle with ill-defined product changes, cross-cutting architecture decisions, concurrency-heavy logic, and anything with ambiguous acceptance criteria. They are also brittle around security-sensitive code paths, compliance logic, and performance optimisations where subtle trade-offs matter. In these domains, agent contributions often increase review overhead and raise the chance of regressions.

Think of agents as diligent junior colleagues who never tire and work at machine speed, but who need tight briefs and strong supervision. The more a task can be described precisely and verified by automated checks, the more value you’ll get. The fuzzier the task and the broader the blast radius, the more likely the agent will waste time or create rework.

Confine agents to testable lanes with automated checks

Create explicit lanes where agents are allowed to operate. A lane is a narrow scope with clear inputs, outputs, and tests. Examples: updating dependencies within defined version ranges; generating unit tests for uncovered functions; adding telemetry to predefined patterns; making changes only within a particular directory or service; performing codemods governed by rules; or editing infrastructure-as-code modules that have policy checks. Document each lane, the permitted file paths, the allowed change size, and the acceptance tests.

Automate the brakes. Enforce branch protection with mandatory status checks: unit tests, contract tests, integration suites, static analysis, secret scanning, software composition analysis, linting, and coverage thresholds that gate merges. Require small pull requests with linked issues and change descriptions in human-readable language. Use CODEOWNERS and a two-person rule for production-facing services. Put agents on merge queues that perform canary deploys and automated rollbacks when health signals degrade.

Strengthen provenance. Sign commits, keep a cryptographic trail for changes produced or assisted by agents, and feed that into your change management. Run agents in sandboxes with least privilege and short-lived credentials. Pre-generate ephemeral environments for PR validation rather than depending solely on mocks, and include realistic data contracts and schema checks. If an agent cannot pass the automated suite, it cannot progress. If it passes, a human reviewer still decides whether the change is necessary and aligned with the roadmap.

Above all, make reviews non-negotiable. Agents can propose; humans dispose. Reviewers should be accountable for what lands, not the agent. That accountability creates the right incentives: fewer sprawling changes, better test discipline, and a culture that prizes clarity over cleverness.

Where the organisation needs an independent baseline first, an AI readiness assessment scores strategy, data, architecture, people, governance and measurement.

Governance and funding: pay only for faster, safer change

Treat agent adoption as a portfolio of experiments, not a blanket mandate. For each lane, define entry and exit criteria tied to DORA and escaped-defect metrics. For example: “We will allow the agent to generate tests in Service A for one month. If lead time for changes on that service decreases while change failure rate and escaped defects do not rise, we expand the lane. If not, we stop.” No hero narratives, just rules agreed in advance.

Tie funding to these rules. Budget not per seat or per token, but per lane and per service. Allocate a small pot to instrument and run the experiment, and a larger pot that unlocks only if the metrics hold over a meaningful period. If cycle time shortens but incidents spike, the spend halts automatically. This avoids the sunk-cost fallacy and the political pressure that often keeps underperforming tools alive.

Bake policy into code. Express lane rules, risk scoring, and required checks in your CI/CD configuration and repository policies. Record which changes were agent-assisted in your change advisory workflow, even if it’s a light-touch process. Over time, you’ll build a body of evidence that tells you where the agents are compounding your team’s effort — and where they’re just shifting work from coding to reviewing.

What it really costs (and the risks you must price in)

The line item for an agent rarely reflects the true cost. There are model inference charges if you use hosted LLMs, or compute and maintenance if you host models yourself. You’ll pay for orchestration, context indexing, evaluation pipelines, and security hardening. Then there’s integration work: wiring the agent into source control, CI/CD, ticketing systems, and secrets management. None of this is glamorous, but it determines whether the tool behaves inside your environment.

There are opportunity costs as well. Reviewing agent PRs takes time. If the agent tends to produce larger diffs or speculative changes, reviewers spend longer to build confidence. If your tests are flaky, the agent will push more churn into the pipeline. If your specifications are fuzzy, the agent will iterate noisily and still miss the mark. These frictions make seemingly cheap automation expensive at the point of use.

Risk must be explicitly priced. Consider accidental exposure of secrets, incorporation of code with problematic licences, subtle security regressions, and performance degradation under load. Introduce provenance and dependency controls to reduce supply-chain ambiguity. Assume you will need to respond to an incident caused or amplified by an agent decision — practise rollback and restoration, and log the cost of that readiness into your business case.

The simplest unit economics I use are cost per merged, production-deployed change and the associated risk profile. Compare that number before and after agent introduction, and segment by lane. If the total cost per safe, merged change goes down and service reliability holds, you have a case. If it rises or reliability dips, you don’t — regardless of how many drafts the agent produced along the way.

What to do next

Start with a baseline. Measure the DORA metrics for your key services and record escaped-defect rates for the last few months. Label your changes so you can attribute impact cleanly. Choose two or three narrow lanes where tests are strong and the blast radius is small — for example, test generation for a single service, codemods within a guarded directory, or dependency updates with strict version policies.

Implement the minimum viable guardrails: branch protection with mandatory checks, CODEOWNERS for the affected paths, small-PR limits, and an approval policy that requires humans to sign off. Run the experiment for a fixed period and publish the results internally. If lead time improves and change failure does not rise, expand carefully to an adjacent lane. If it doesn’t, stop, learn, and try a better-bounded lane — do not chase sunk costs.

If you want a pragmatic outside view, I offer AI Automation Consulting and Agentic AI Consulting focused on this kind of outcome-driven adoption. Through my CTO Advisory Services, I work with leadership teams to set up the metrics, lanes, and governance so you fund only what actually ships. Technology made practical: measure, constrain, deliver — then scale.