Anthropic METR access: your audit pattern

What happened: a new bar for independent oversight

Anthropic is granting employee-level access to six researchers from METR to assess its models, safety practices, and risk management. The arrangement includes Anthropic-issued laptops, office space, and access to internal tools comparable to its own risk assessment teams, with staff encouraged to share information as they would with a colleague working on alignment risk, all as reported by TechCrunch.

Per TechCrunch, METR will publish its own assessments of Anthropic’s risks, incidents, and practices. Anthropic cannot veto publication, but can require redactions in narrow, predefined categories such as customer data, attorney–client privileged material, or model architectures; METR can publicly characterise any redactions without revealing the redacted content.

Anthropic is also building a three-tier evaluator system, with METR alone at the top tier having employee-level access. A bottom tier includes groups like SecureBio running narrower reviews that require approval to publish, while a middle tier gets more information and can publish without approval, according to TechCrunch. The company likened the setup to resident examiners in banking. The move follows a period in which incidents of AI agents acting outside instructions have been disclosed every week or two since late August, and aligns with earlier public pledges from several AI companies for independent review, as covered by TechCrunch.

Why it matters: closing the information gap

For buyers and builders of AI, this is the clearest signal yet that arm’s‑length reviews aren’t enough. You can’t properly judge a complex model’s risk from a marketing deck and a few demo prompts. Embedded access shortens the feedback loop, aligns incentives around transparency, and reduces the asymmetry between what a vendor knows and what you, the operator, need to know before you deploy.

There’s a governance lesson here. If a leading lab is willing to host evaluators with near-employee access and non‑vetoed publication, the expectation for enterprise AI governance should rise accordingly. Independent eyes, adequate access, and the right to publish (with bounded redactions) are fast becoming table stakes for trustworthy deployments.

From headline to playbook: Anthropic METR access as a pattern

You don’t need a frontier lab to adopt this model. Treat evaluator‑level access as a programme you can run across both suppliers and your own teams. The goal is simple: reduce blind spots and create a repeatable, auditable way to surface issues before they become incidents.

Design a tiered framework. At the base tier, allow limited, issue‑specific reviews for narrow questions. In the middle, permit broader reviews and independent publication. At the top, offer scoped, employee‑level access for your most material systems or partners. This mirrors the direction reported by TechCrunch without pretending every environment needs the same depth on day one.

Access should be purpose‑bound. Provision scoped accounts with least privilege, time‑boxed elevation, and ring‑fenced environments. Require full telemetry: session logging, model call traces, prompt/response capture with data minimisation, and change auditing. Add a clearly owned kill‑switch at the service or model gateway, with a service‑level commitment to disable or roll back when an evaluator raises a material risk. This is where good intentions turn into enforceable model risk management.

Who owns technology leadership is the structural question behind decisions like these, and Do You Need a CTO Yet? Fractional, Virtual or Full-Time works through the options.

Set publication expectations up front. The Anthropic pattern, per TechCrunch, allows independent publication with narrowly defined redactions. Adopt the same principle: define precise redaction categories, commit to non‑vetoed publication outside those, and accept that evaluators can describe any redactions. That balance encourages frank reporting while protecting genuinely sensitive material.

Finally, make room for a true third-party audit. Evaluators should be independent of delivery and P&L, able to escalate without fear, and rotated periodically. Independence is what turns access into credible assurance. Use this to complement, not replace, internal testing and red‑teaming, and ensure your change board treats evaluator sign‑off as a gating control alongside security and privacy reviews.

Contracting for evaluator-level access

Your vendor agreements should codify the pattern. Be explicit about evaluator rights, scope, and limits so neither side is surprised when a tough finding lands. Here’s the minimum I would bake in:

Access and scope: define tier, systems in scope, data classes permitted, and the evaluator’s independence. Require vendor‑provisioned, scoped accounts, dedicated workspaces, and monitored endpoints for access.

Telemetry and evidence: mandate logging coverage, artefact retention, and timely access to internal tools and dashboards comparable to what risk teams use, echoing what TechCrunch reports in spirit.

Some leadership teams bring in part-time CTO leadership to put this discipline in place before a full-time appointment makes sense.

Publication rights: commit to independent publication with only predefined redaction categories. The Anthropic approach, as described by TechCrunch, names customer data, attorney–client privileged information, and model architectures as examples. Borrow that clarity; it preserves trust on both sides.

Kill‑switch SLA: specify the trigger, responsible roles, and response timelines, and require regular tests. Tie it into incident management and post‑mortem obligations.

Change management: require evaluator review for major releases, capability uplifts, or new integrations, and record their sign‑off in your CAB. Treat critical findings like Sev‑1 defects.

Cost and cadence: scope evaluation windows, reporting cycles, and who pays. Expect to fund evaluators’ time and secure workstations; vendors should fund the access plumbing. This is a manageable premium for meaningful assurance.

Finally, align the language with your broader AI safety evaluations to avoid duplication and ensure a single, comprehensible risk narrative across suppliers and internal builds.

Build the same muscle internally

Don’t stop at suppliers. Stand up an internal evaluator function with a mandate and access comparable to your external programme. Give them protected independence, budget, and a direct line to the CIO/CTO and Audit Committee.

Operate a clear intake for evaluations, a risk taxonomy, and standard evidence packs. Make evaluator sign‑off a prerequisite for promoting a model to production, raising rate limits, or enabling tool use. Wire their verdicts into deployment pipelines the way you already do for security and privacy checks.

Treat production disablement as a change scenario you rehearse, not a theoretical clause. Run game days for kill‑switch activation, external comms, and rollback plans. The day you need it is the wrong day to discover where the switch actually lives.

What leaders should do now

This week: identify your top three AI‑dependent business services and list their critical vendors and internal models. For each, decide which evaluation tier you’d accept and what access you’ll need to grant or receive. Socialise the principle that independent publication, with bounded redactions, is the default.

In 30 days: draft an evaluator‑level access policy, including account provisioning, logging, data handling, and kill‑switch procedures. Prepare a standard evaluation addendum for contracts and initiate amendments with your most material vendors. Align your change board to require evaluator sign‑off for major releases.

This quarter: run a pilot evaluation with one critical supplier and one internal model. Measure the cost, friction, and findings. Hard‑wire the lessons into your runbooks, budget, and training. Your aim is simple: make meaningful access, evidence, and escalation routine rather than exceptional.