Private LLM for Enterprise: KV Cache Safety

KV/attention caches leak when boundaries blur

A transformer’s attention cache (the K/V states you reuse across decoding steps) exists to cut latency and cost. In many frameworks it’s presented as an optimisation you can enable across tokens, across turns, and sometimes across requests. That’s where risk creeps in. If the cache isn’t scoped to a clear security boundary, a later request can retrieve state created by a different user, session or even tenant.

Cloud providers and model hosts have published remediations for cross-request cache exposure — notably around GLM-5.3 on Workers AI — but the root cause is architectural, not vendor-specific. It’s straightforward to save partial decoding state to a key–value store and equally straightforward to mis-key it. When keys collide, or when validation is absent, you serve another caller’s state. The result ranges from dose-of-latency weirdness to outright data disclosure.

In practice you’ll see three failure modes: weak keying (e.g., key built from only model name and prompt hash), lack of validation on retrieval (you accept any value that decodes), and lifetimes that outlive provenance (caches survive redeploys or topology moves). The remedy is equally practical: bake provenance into the key and the value, prove it on read, and expire it aggressively.

Below I’ll outline the patterns I use in production to reduce the blast radius, including exact fields to key by, how to HMAC and validate, and what to observe. This is all applicable whether your cache is in-process, on Redis, or fronted by a cloud KV service.

Scoping a Private LLM for Enterprise cache

Start with the boundary. For enterprise workloads your minimum security scope is tenant. For many deployments, user or session is also appropriate, but tenant is non-negotiable. From there, scope by compute and build context so you never conflate incompatible states.

I recommend the following fields be mandatory for cache scoping and validation:

– tenant_id: the contractual isolation boundary.

– model_id: a logical name (e.g., glm-5.3) plus a build hash of weights and tokenizer.

– deployment_id: your service build or container image digest; if you roll, the cache is cold.

– region/zone: avoid mixing state across POPs or fault domains.

– session_id: the conversational thread or sequence origin (if you persist state across turns).

– request_epoch: coarse timestamp bucket (e.g., 5–15 min) to gate TTL and rotation.

– prompt_fingerprint: a salted HMAC of input features you actually depend on, never raw text.

For how to weigh hosted, dedicated and private models, see when a private LLM is worth it.

If you batch decode across multiple sequences, include batch_id as well and only reuse within the same batch and boundary. Resist the temptation to scope on just a prompt hash: it’s reproducible by an attacker, brittle under tokenisation quirks, and unable to express tenancy.

Finally, decide when a cache must be logically invalidated. Any of these should invalidate on read: different tenant/user, model/tokenizer hash mismatch, deployment rolled, region changed, or maximum age exceeded. If you can’t positively match all required attributes, drop the entry.

Derive stable, non-guessable keys

An LLM cache key must be deterministic within a boundary and unguessable from outside it. That means concatenating the scoping fields in a canonical order and protecting them with an HMAC using a server-held secret. Never put raw prompts, emails or PII in the key; if you need to depend on input, use a salted fingerprint. Keep the key short enough for your backend (Redis keys <= 512 MB is fine, but shorter improves memory use).

Here is a small Python helper that builds a namespaced key using HMAC-SHA256 and URL-safe base64. You can embed a short prefix (e.g., kv:glm) for observability and routing.

pythonimport base64, hashlib, hmac, json

NAMESPACE = "kv:glm"

def b64url(data: bytes) -> str:
    return base64.urlsafe_b64encode(data).rstrip(b"=").decode()

def make_kv_key(secret: bytes, *, tenant_id: str, model_id: str,
                weights_sha: str, deployment_id: str, region: str,
                session_id: str, step: int, prompt_fpr: str, epoch_bucket: str) -> str:
    fields = {
        "t": tenant_id, "m": model_id, "w": weights_sha, "d": deployment_id,
        "r": region, "s": session_id, "p": prompt_fpr, "e": epoch_bucket, "k": step,
    }
    blob = json.dumps(fields, separators=(",", ":"), sort_keys=True).encode()
    tag = hmac.new(secret, blob, hashlib.sha256).digest()
    return f"{NAMESPACE}:{b64url(blob)}.{b64url(tag[:16])}"

# Example
# key = make_kv_key(SECRET, tenant_id="acme", model_id="glm-5.3",
#                   weights_sha="a1b2...", deployment_id="img@sha256:...",
#                   region="lon1", session_id="conv-123", step=128,
#                   prompt_fpr="fpr-wkQ2...", epoch_bucket="2026-10-27T12:00Z")

Notes: sort JSON keys for canonicalisation; truncate the tag used in the printable key to keep it short but retain the full tag server-side for validation if you store it with the value. If your store has strict key length limits, compress the blob first or move some fields into the value and validate after fetch.

Validate on read: refuse mismatched caches

Keying is half the story. On read you must prove the cache belongs to the caller and is compatible with the current runtime. I store a compact metadata header alongside the tensor payload, then recompute the HMAC over that header and compare. If any field mismatches, treat it as a miss. Avoid silent downgrades; log a counter and move on.

If you expose a binder back to the client (e.g., to resume a stream), sign it with a server secret and re-verify on the next call. Here’s a minimalist TypeScript example for a binder header using Node’s crypto. It avoids JWT ceremony while giving you integrity and expiry.

typescriptimport { createHmac, timingSafeEqual } from "crypto";

type Binder = { t:string; m:string; w:string; d:string; r:string; s:string; exp:number };

function sign(b: Binder, secret: Buffer): string {
  const payload = Buffer.from(JSON.stringify(b));
  const mac = createHmac("sha256", secret).update(payload).digest();
  return Buffer.from(Buffer.concat([payload, mac.subarray(0,16)])).toString("base64url");
}

function verify(token: string, secret: Buffer): Binder | null {
  const buf = Buffer.from(token, "base64url");
  const payload = buf.subarray(0, buf.length - 16);
  const tag = buf.subarray(buf.length - 16);
  const mac = createHmac("sha256", secret).update(payload).digest().subarray(0,16);
  if (!timingSafeEqual(tag, mac)) return null;
  const b = JSON.parse(payload.toString()) as Binder;
  return (Date.now()/1000 < b.exp) ? b : null;
}

Store a Binder-equivalent header with your cache value (e.g., at the start of a binary blob). On fetch: decode, HMAC-verify, check expiry, confirm tenant/model/weight/region/session match the current request, then and only then use the tensor state. If you upgrade model weights or tokenizer, bump the weights hash; the binder will fail closed and you pay a cold start rather than risk leakage.

TTL, rotation, and versioning: fail safe by default

Caches must be short-lived. For attention caches, think minutes not hours. A sensible starting point is a TTL of 5–15 minutes for interactive workloads and even shorter for sensitive contexts. For long-running streams, prefer in-memory caches scoped to the request and only persist if you genuinely need cross-request reuse.

Rotate the epoch bucket (e.g., every 10 minutes) and bake it into the key so entries naturally roll off. On deployment, change deployment_id so old caches miss. On model updates, change weights_sha and tokenizer hash. If you migrate regions or POPs, include region so nothing travels implicitly across zones.

Before scaling, it is worth having someone review your LLM or agent architecture against the controls it will need.

If your store supports it, use write-with-TTL primitives and avoid manual expiry daemons. Keep the payload minimal: a small header with provenance plus the tensor buffers. Avoid putting full prompts or user input in cache values; if you must, encrypt with a tenant-specific key and independent nonce. Even then, validation must succeed before decryption to prevent oracle misuse.

Here’s a Redis-oriented snippet showing namespacing, SETEX for TTL, and a side-channel hash for observability. Adjust values to your platform.

bash# write
redis-cli SETEX kv:glm:acme:lon1:seq-123:step-128 600 "<binary-or-base64-payload>"
# minimal metadata for dashboards (no PII)
redis-cli HSET kvmeta:glm:acme:lon1:seq-123 hits 0 model "glm-5.3" build "img@sha256:..."
# on hit
redis-cli HINCRBY kvmeta:glm:acme:lon1:seq-123 hits 1
# observability: list top tenants by hits
redis-cli ZINCRBY kvstats:tenants 1 acme
# sanity: ensure TTL is present and short
redis-cli TTL kv:glm:acme:lon1:seq-123

Be ruthless about invalidation. When in doubt, bin it. The cost of a recalculated step is almost always lower than the risk of cross-tenant disclosure in regulated environments.

Batching, streaming and agents: the concurrency traps

Batch decoding and speculative execution are fertile ground for cross-request exposure. When you batch, ensure every sequence in the batch shares the same cache boundary. That usually means batching by tenant and model build at minimum. Never allow the scheduler to coalesce sequences from different tenants into the same attention cache lines. If your library hides batching internals, put a boundary-aware wrapper in front of it.

Streaming retries introduce another trap. When a client reconnects mid-stream and you resume from cache, you must require a valid binder (as shown earlier) that proves tenant, session and expiry. Otherwise, a malicious client could probe for partial completions from other sessions. If you support backpressure and replay, pin the resume token to a monotonically increasing step and refuse out-of-order resumes.

Agentic patterns complicate matters because they mix multiple tools and contexts. Keep each agent’s context in a separate namespace and never let a tool call fetch a general-purpose cache. If the agent framework maintains shared memory across tools, treat that memory as data, not as an execution cache, and don’t serialise it into your KV attention cache unless you encrypt and bind it with the same strict metadata.

For server-side batching queues, record the batch_id and the boundary digest in logs and metrics. A useful invariant: for any cache hit, the boundary digest of the request must equal the boundary digest stored in the value. If these ever diverge, raise an alert and treat it as a security incident, not an SRE ticket.

Audit and test: prove your cache can’t bleed

You can deploy hardening today and still be unsure it works. Bake tests into CI and run smoke checks in staging and production.

Unit tests: feed make_kv_key with varying fields and assert that any difference in tenant, model hash, deployment, region, session or epoch produces a different key. Property tests are good here. Also check that identical inputs yield identical keys across versions for stability.

Integration tests: spin up two tenants (acme and zephyr). Issue identical prompts that would, absent scoping, produce identical attention states. Verify that acme never hits zephyr’s cache and vice versa. Repeat across regions and after a rolling deploy to ensure deployment_id invalidates correctly.

Security tests: fuzz the binder verification with truncated tags, altered payloads, expired tokens and swapped tenant_ids; ensure every case is rejected. Run a synthetic adversary that cycles through plausible prompt fingerprints and steps to prove that HMAC’d keys are not enumerable. Attempt to fetch without a binder in resume scenarios and ensure it fails closed.

Observability: add counters for kv_hit, kv_miss, kv_reject_mismatch, kv_reject_expired, kv_reject_region, kv_reject_modelhash, kv_write. Enable cardinality control by hashing tenant ids before exporting. Sample payload sizes and TTL remaining to catch runaway entries. Alert when any reject counter is non-zero or when TTL on hot keys exceeds your policy.

Runbooks: document invalidation triggers (deploy, model swap, region move) and the exact commands to purge keys by namespace or tenant. Practice the purge in a lower environment so it’s not your first time during an incident.

What to do next

If you own a model-serving stack today, pick one service and do this in the next week:

1) Add a boundary struct (tenant, model build, deployment id, region, session, epoch) and pass it through your request path. 2) Replace ad-hoc keys with an HMAC-derived key. 3) Embed a compact header with the same fields in every cache value. 4) On read, verify HMAC, check expiry and exact equality of all fields; on any mismatch, treat as miss. 5) Set TTLs to minutes and wire invalidation to deploys and model changes. 6) Ship counters for hit/miss/reject reasons and set alerts.

If you run on a managed platform, review the provider’s cache isolation guarantees and then enforce your own at the application layer. Keep the cache optional: a feature flag should disable it instantly if you suspect bleed-through.

For leaders building a Private LLM for Enterprise, these patterns form your minimum bar for trust. They also give you clear artefacts for auditors: key derivation code, validation logic, TTL policy, and test evidence. If your teams are stretched, bring in a Solution Architecture Consultant to help retrofit boundaries without stalling roadmaps. If you’re designing a greenfield platform or an agent framework, consider a short engagement under LLM Consulting Services to design keying, validation and observability upfront. And if you’re exploring tool-using agents across tenants, an experienced Agentic AI Consultant can help you isolate context stores and execution caches safely.

None of this is glamorous, but it is shippable this week. Draw the boundary, codify it in keys and values, validate relentlessly, and keep lifetimes short. The fastest model won’t save you from a leaky cache; the right architecture will.