Enterprise RAG in the Era of 1M‑Token Context

What 1M‑token windows change—and what they don’t

A million tokens lets you feed an LLM whole subsystems: multi‑file codebases, compliance packs, entire meeting transcripts or knowledge bases. Models like Step‑5 preview on OpenRouter show what’s coming: far fewer contortions to fit context and much simpler flows for long‑form tasks.

But first principles remain. Tokens are metered and latency grows with input size. The longer the prompt, the more you pay and the more you wait. Using a huge window to shove raw source in is easy; using it responsibly is engineering. Enterprise RAG still matters because targeted recall, freshness, provenance, and permissioning don’t solve themselves just because the buffer is large.

Think of a 1M window as breathing room for better structure: coherent manifests, sectioned inputs, and small but precise retrieval. The goal is to reduce orchestration complexity while preserving discipline about what you actually send per request.

Keep or trim RAG? A pragmatic decision for Enterprise RAG

You don’t drop retrieval just because you can fit more text. You trim it when the marginal value of selection is lower than the orchestration overhead. Here’s a quick decision frame:

– Keep RAG when: content changes often, you need citations or audit trails, you have per‑user permissions, or your corpus dwarfs even a 1M context. Enterprise RAG typically earns its keep in regulated domains, customer support across tens of thousands of articles, or multi‑tenant knowledge with fine‑grained ACLs.

– Trim RAG when: you’re repeatedly querying a stable, bounded artefact (e.g., a single code repository, a product spec bundle, or a legal pack) under one access policy. In these cases, an up‑front condensed context (with an index/manifest inside the prompt) may be simpler and fast enough.

– Go hybrid when: you have a bounded artefact plus a ‘long tail’ of references. Push the main pack in full, but live‑fetch a narrow set of external sections via retrieval. That preserves freshness without re‑sending the entire world.

Two architectural patterns I’ve found reliable:

1) Manifest + On‑Demand Snippets: Send a compact manifest (titles, section IDs, 1–2 line synopses) and only the top N full sections by default. Tool the model to request additional sections by ID when needed (function/tool calls or a second turn).

2) Rolling Cache for Known Tasks: For workflows that repeat (weekly reporting, policy Q&A), pre‑compose a canonical long context and cache it server‑side. Top up deltas via retrieval per request, not the entire base pack.

For most Enterprise RAG scenarios, retrieval remains the gatekeeper for relevance, access and provenance. The big window simply lowers the pain of including richer context around the retrieved bits: nearby sections, diagrams in text form, and a well‑structured index.

How to structure very long prompts that don’t drift

Long prompts collapse if they’re a heap of glue. At million‑token scale you need deterministic layout and explicit addressing. The model should know where instructions live, where documents begin and end, and how to cite facts back to stable IDs.

I use a strict header + manifest + body pattern:

– A YAML header with task, constraints, output schema and citation rules.

– A manifest listing documents and sections with stable IDs and short synopses.

– A body with full sections in a fixed order matching the manifest.

– Sentinels (BEGIN/END markers) so truncation is obvious, and the model can reference segments deterministically.

For how to weigh hosted, dedicated and private models, see when a private LLM is worth it.

– A short ‘reading plan’ instruction, encouraging the model to consult the manifest first and only then deep‑read relevant sections.

yaml### INSTRUCTIONS (YAML) ###
role: "assistant"
goal: "Answer the user query using the provided corpus."
constraints:
  - "Cite sources as [doc_id#section_id] after each claim."
  - "If uncertain, say so and request specific section IDs."
  - "Only use content within BEGIN_CORPUS/END_CORPUS."
output_schema:
  format: "json"
  fields: ["answer", "citations", "uncertainties"]

### MANIFEST ###
- doc_id: D1
  title: "Payments Architecture"
  sections:
    - { id: S1, title: "Overview", synopsis: "Scope, actors" }
    - { id: S2, title: "Sequence Flows", synopsis: "Auth, capture" }
- doc_id: D2
  title: "PCI Policy"
  sections:
    - { id: S1, title: "Storage", synopsis: "PAN rules" }

### BEGIN_CORPUS ###
[D1#S1]
...text...
[D1#S2]
...text...
[D2#S1]
...text...
### END_CORPUS ###

User Query: "How do we handle auth retries without storing PAN?"

This gives the model the scaffolding to reason first, read second, and cite deterministically. It also makes truncation and partial updates manageable in code.

Chunking and compression that scale with million‑token contexts

Bigger windows don’t excuse sloppy segmentation. The aim is to align chunks with semantic boundaries so the model doesn’t have to reassemble intent across arbitrary cuts. For code, use file and symbol boundaries; for policy, section and sub‑heading boundaries; for transcripts, speaker turns and agenda items.

Pragmatic rules that work:

– Prefer sections (0.5–3k tokens) with stable IDs; include 200–400 token overlap only where references bridge sections (e.g., a function split across files).

– Build a manifest of sections with synopses to place near the front of the prompt; optionally include a micro‑TOC for navigation.

– Compress boilerplate and legalese with extractive summaries; keep normative clauses verbatim.

– Deduplicate near‑identical sections before sending; sort by dependency order (e.g., base classes before subclasses).

If you still need retrieval, use it as a filter rather than a full packer: select candidate sections, then expand to include adjacent context when necessary, constrained by a soft token budget. Here’s a small helper to construct a manifest with stable IDs and byte‑size budgeting:

pythonimport hashlib
from typing import List, Dict

class Section:
    def __init__(self, title: str, text: str, doc_id: str, sec_idx: int):
        self.title = title
        self.text = text
        self.doc_id = doc_id
        self.sec_id = f"S{sec_idx}"
        self.key = f"{doc_id}#{self.sec_id}"
        self.digest = hashlib.sha1(text.encode()).hexdigest()[:8]


def build_manifest(docs: Dict[str, List[Section]], max_tokens: int) -> Dict:
    # naive token estimate: 4 chars ~ 1 token; adjust as needed
    budget_chars = max_tokens * 4
    used = 0
    manifest = []
    body = []
    for doc_id, sections in docs.items():
        entries = []
        for s in sections:
            snippet = s.text[:280]
            est = len(s.text)
            if used + est > budget_chars:
                break
            used += est
            entries.append({"id": s.sec_id, "title": s.title, "synopsis": snippet})
            body.append((s.key, s.text))
        manifest.append({"doc_id": doc_id, "sections": entries})
    return {"manifest": manifest, "body": body, "used_chars": used}

This is intentionally simple but forces explicit budgeting and stable IDs, which you can later line up with citations and caching.

A minimal OpenRouter harness to profile latency and cost

Before you re‑write a pipeline around big context, measure. You want end‑to‑end latency, token usage, and an estimated cost per request against realistic payloads. OpenRouter exposes usage fields that make this straightforward.

If you are running systems like this in production, an AI architecture review tests retrieval, evaluation, permissions and cost before they become incidents.

The snippet below targets Step‑5 preview, measures wall time, reads token usage, and computes a simple cost estimate from environment‑provided prices. Swap in your model of choice if you’re comparing options.

typescriptimport process from 'node:process';

async function run(prompt: string) {
  const t0 = Date.now();
  const res = await fetch('https://openrouter.ai/api/v1/chat/completions', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Authorization': `Bearer ${process.env.OPENROUTER_API_KEY}`,
    },
    body: JSON.stringify({
      model: 'stepfun/step-5-preview',
      messages: [{ role: 'user', content: prompt }],
      temperature: 0.2
    })
  });
  const data = await res.json();
  const t1 = Date.now();
  const u = data.usage || {};
  const pin = Number(process.env.PRICE_IN) || 0; // per-token
  const pout = Number(process.env.PRICE_OUT) || 0;
  const est = (u.prompt_tokens||0)*pin + (u.completion_tokens||0)*pout;
  console.log({
    latency_ms: t1 - t0,
    prompt_tokens: u.prompt_tokens,
    completion_tokens: u.completion_tokens,
    est_cost: est,
    sample: data.choices?.[0]?.message?.content?.slice(0, 200)
  });
}

run('Summarise the manifest and answer the query per instructions.');

Use this harness to sweep input sizes: 50k, 200k, 500k, and so on. Capture P50/P95 latency and tokens. Decide whether to compress, re‑chunk, or introduce a small retrieval step. Keep an eye on failure behaviour for very large inputs—timeouts and network retries become materially more likely as payloads grow.

Production patterns: budgets, monitoring, and fallbacks

Once you have measurements, codify limits. In production I set soft and hard budgets per route (tokens in, tokens out, wall time) and emit metrics when exceeded. Design your prompts and retrieval to degrade gracefully under pressure.

Practical guardrails:

– Soft budget guards: pre‑flight count tokens; if above threshold, switch to compressed sections (extractive summaries) or reduce adjacency expansion.

– Hard ceilings: fail fast with a helpful message or open a two‑turn flow (send manifest first, then let the model request section IDs).

– Retry policy: idempotent request IDs, truncated back‑off, and a max inflight semaphore to avoid stampedes during spikes.

– Caching: cache the manifest and stable sections by digest; only resend deltas. If you control both client and server, consider ETag‑style digests for large payloads.

– Observability: track tokens, latency, model choice, prompt hash, error class. Add a sampled log of the first 500 chars of input/output for diagnosis (mind PII).

Patterns that save real money at large context sizes:

– Plan‑then‑read: instruct the model to produce a short ‘reading plan’ listing section IDs; verify server‑side, then follow with a second call including only those sections.

– Map‑reduce summarisation: for corpora that don’t fit in one go, summarise per section with a tight format, then combine with a final pass in a larger window.

– Delegate for structure: ask the model to emit structured extraction (JSON) instead of prose where possible; you’ll compress output tokens and gain machine‑readability.

– Task‑aware packing: only send the parts relevant to the user’s verb (explain, refactor, compare, verify). The manifest helps you branch the packing logic predictably.

What to do next

If you’re holding a large, stable artefact (a codebase, a policy bundle, a solution design), try the manifest pattern first: ship the instructions, manifest, and the top sections. Measure. If your latency or cost is unacceptable, introduce a retrieval step that selects fewer sections by default and expands on demand.

If your domain is volatile or access‑controlled—customer support, regulated policies, multi‑tenant knowledge—keep Enterprise RAG as your spine. Use the big window to include richer surrounding context and cleaner structure, not to bypass indexing, permissions or provenance.

Concrete steps for the coming week:

– Build the OpenRouter harness and profile P50/P95 against 3–4 realistic payload sizes.

– Enforce soft/hard token budgets and add a manifest to your prompt layout.

– Trial a hybrid path: retrieval for candidates, then expand nearby sections within budget.

– Add two failure modes: compressed fallback and a plan‑then‑read two‑turn path.

If you want an external sounding board, I’m happy to review an architecture as a RAG Consultant lens or from a Solution Architecture Consultant angle. And if you’re spinning up pilots and need engineering muscle plus governance advice, that’s squarely in LLM Consulting Services territory.

The theme is constant: measure, structure, and only send what you need. Million‑token context makes that easier, not optional.