Everyone's first instinct when building an agent is the same: add a vector database, do RAG, ship it. You get retrieval. You tell yourself retrieval is context. It is not.
Context is retrieval plus memory plus permissions plus business semantics — assembled into a coherent, governed picture before the agent reasons over a single token. A vector database gives you document chunks ranked by cosine similarity. Context gives the agent an understanding of who is asking, what they're allowed to see, what happened previously, and what the business rules say about the current task.
We learned this while running AI screening across thousands of concurrent candidates. "Context" wasn't a retrieval problem — it was an architectural contract between data, identity, and reasoning.
Why "Just Add RAG" Isn't a Context Strategy
RAG solves a narrow problem: given a query, find relevant chunks and inject them into the prompt. It's useful. It's also roughly 20% of what a production agent needs to reason correctly. What it doesn't give you:
- Identity awareness. Who is making this request, and what data are they authorized to access?
- Session continuity. What happened in previous conversations that should inform the current response?
- Business semantics. What does "qualified" mean for this specific role at this specific company?
- Governance constraints. Is this data subject to retention policies? Can it cross tenant boundaries?
Calling your RAG pipeline a "context layer" is like calling a database connection a "backend." It's a component, not an architecture.
The Three Dimensions of Context
Once you stop thinking about context as "stuff we inject into the prompt" and start thinking about it as a governed information tier, a useful taxonomy emerges. Context has three distinct dimensions, and production systems need all three.
| Dimension | What It Contains | Example |
|---|---|---|
| Data context | Retrieved docs, database records, API responses | Candidate resume, job description, screening rubric |
| Semantic context | Business rules, domain ontology, evaluation criteria | "Senior" means 8+ years at this company; different rubric for leadership roles |
| Governance context | User identity, permissions, data retention, audit trail | Recruiter can see scores but not raw transcripts; data purged after 90 days |
Data context is what most teams build first — and often the only dimension they build at all. Important, but structurally incomplete.
Semantic context is the business logic that tells the agent what the data means. Without it, the agent retrieves a job description but doesn't know this company weights communication skills at 2x for customer-facing roles. The difference between a generic AI and one that understands the domain.
Governance context is the layer most teams skip until a customer's security team asks uncomfortable questions. It determines what the agent is allowed to know — not what it can retrieve, but what it should retrieve given identity, permissions, and compliance posture.
Memory Lifecycle Management
Deciding what to remember is hard. Deciding what to forget is harder. And deciding when to transition between remembering and forgetting is the hardest part of the entire context layer.
Remember Everything
Remember Nothing
Neither extreme works. The answer is a structured lifecycle — explicit policies governing how context enters, persists, degrades, and gets evicted.
- Ingestion. New context enters tagged with metadata: source, timestamp, tenant, confidence score, TTL.
- Active use. The context is in the current session's working set, consuming tokens and influencing reasoning.
- Compression. No longer in the active window but still relevant — summarized, distilled, or moved to a lower-resolution store.
- Eviction. Exceeded its retention window or been superseded. Removed — and the removal is logged for audit.
Hierarchical Memory Architecture
The lifecycle model maps naturally to a tiered memory architecture — three layers with different retention windows, compression strategies, and eviction policies.
Message buffer is the current conversation — full resolution, every token preserved. Fast, expensive, temporary. When the session ends, the raw buffer is processed, not stored wholesale.
Episodic memory holds compressed summaries of past sessions. Not full transcripts — key outcomes: what was discussed, what the user corrected, what feedback was given. In our production system, a 45-minute screening session compresses to a structured summary consuming roughly 10–15% of the original token count.
Archival memory holds long-term facts extracted from repeated episodes — persistent preferences, calibration benchmarks, evaluation criteria refined over dozens of sessions. Stored as structured records, not embeddings, because they need to be directly retrievable, updatable, and auditable.
Strategic Forgetting
The instinct is to keep everything. More data, more context, better results. This instinct is wrong.
Unbounded memory doesn't make agents smarter — it makes them noisier. Stale context injected into today's reasoning window causes the same class of failure as a corrupted cache: outputs that are coherent, confident, and wrong in ways that trace back to outdated information.
Strategic forgetting requires explicit policies:
- TTL-based eviction. Every memory record has a time-to-live. Episodic memories expire after weeks unless refreshed. Archival facts have longer TTLs but still expire — nothing lives forever.
- Confidence decay. A preference confirmed last week has high confidence. The same preference unconfirmed for three months gets deprioritized in context assembly.
- Contradiction-triggered invalidation. When new information contradicts a stored memory, the conflict is detected and the old record is flagged for review.
The goal is a context layer where the signal-to-noise ratio improves over time, not one where noise eventually drowns the signal.
The Context Layer as Infrastructure
A retrieval pipeline and a context layer are not the same thing. The difference is architectural scope.
Retrieval Pipeline
Context Layer
The context layer is a first-class infrastructure tier — same as your authentication system or data pipeline. Its own scaling concerns, failure modes, observability, and on-call responsibilities.
In production, the context layer handles identity resolution, permission scoping, memory retrieval across all three tiers, semantic enrichment, and final assembly into the structured context block that enters the model's window. That's a system, not a helper function.
Identity and Permissions in Context
Multi-tenant context isolation is the kind of problem that sounds straightforward until you're debugging a production incident where Agent A's context bled into Agent B's reasoning because a shared embedding cache didn't partition by tenant.
In a multi-tenant AI system, every context operation must be scoped:
- Memory retrieval must be tenant-isolated. A query against episodic or archival memory must never return records from another tenant.
- Semantic context must be customer-specific. The business rules for Customer A's evaluation criteria must not influence Customer B's sessions.
- Governance context must enforce the requesting user's permissions, not the agent's permissions. The agent has access to everything — the user does not.
We learned this during multi-tenant AI screening. Early architecture shared a single vector index with metadata filtering at query time. It worked — until a filter bug surfaced one customer's screening rubric in another customer's evaluation. The output was valid, well-structured, and used the wrong company's definition of what a good candidate looks like.
The fix was architectural: tenant-scoped namespaces, tenant-tagged memory records with enforcement at the retrieval layer, and a validation step that checks every assembled context block for cross-tenant contamination before it enters the model.
// key takeaway