Building AI Agent Memory That Actually Works in Production

The majority of AI agents deployed today have a fundamental flaw: they forget everything the moment a session ends. Each conversation starts from a blank slate. The user re-explains their preferences, re-establishes context, and watches the agent repeat mistakes it already "learned" three sessions ago. This is not an AI intelligence problem—it is a memory architecture problem, and it is entirely solvable.

Production-grade AI agent memory is not a single database or a prompt-stuffing strategy. It is a multi-tier system that balances recall accuracy, retrieval speed, storage cost, and—critically in regulated industries—the right to be forgotten.

Why Context Window Stuffing Fails

The naive approach to agent memory is "dump everything into the context window." This breaks in production for three compounding reasons:

1. Cost scales non-linearly. Processing 1 million tokens costs orders of magnitude more than 10,000 tokens. An agent handling thousands of daily sessions cannot afford loading full histories into every prompt.

2. "Lost in the middle" degradation. LLMs retrieve information at the beginning and end of a context window more reliably than information buried in the middle. A 500,000-token raw history results in the model reliably missing facts that appeared in the center.

3. Dynamic data goes stale. If a user's location changes, a raw append-only log now contains two contradictory addresses. The model may hallucinate the wrong one. You need contradiction-aware memory, not just fact accumulation.

The Three-Tier Memory Architecture

Production agent memory is organized into three distinct tiers, each optimized for different access patterns and retention durations:

┌─────────────────────────────────────────────────────────┐
│         TIER 1: Working Memory (In-Session)             │
│   Redis / In-Process Buffer • Millisecond Access        │
│   Active turns • Current task state • Tool call results │
└─────────────────────────┬───────────────────────────────┘
                          │ Flush on session end
                          ▼
┌─────────────────────────────────────────────────────────┐
│       TIER 2: Episodic Memory (Session History)         │
│    PostgreSQL / MongoDB • Low-latency, Filterable       │
│    Session summaries • Key decisions • Corrections      │
└─────────────────────────┬───────────────────────────────┘
                          │ Extracted entities & embeddings
                          ▼
┌─────────────────────────────────────────────────────────┐
│       TIER 3: Semantic Memory (Long-Term Facts)         │
│   Qdrant / pgvector + Graph Layer • Fuzzy Recall        │
│   User preferences • Learned facts • Patterns           │
└─────────────────────────────────────────────────────────┘

Tier 1: Working Memory

Stored in Redis or an in-process buffer, accessed in under 5ms, and ephemeral by design. Holds the rolling conversation turns, current task state, tool call results, and active session context. When a session ends, key facts are extracted and promoted to Tier 2 rather than discarded.

Tier 2: Episodic Memory

Stores what happened across sessions, organized by session identity in a relational or document database. Efficient filtering by user_id, session_id, and timestamp allows targeted recall without loading full histories.

interface EpisodicMemoryRecord {
  id: string;
  userId: string;
  sessionId: string;
  timestamp: Date;
  summary: string;          // LLM-generated summary of the session
  keyDecisions: string[];   // Extracted decisions made
  corrections: string[];    // Cases where user corrected the agent
  entityChanges: {
    entity: string;
    previousValue: string | null;
    newValue: string;
  }[];
  embeddingId: string;      // Reference to vector for semantic search
}

Tier 3: Semantic Memory

Stores facts, preferences, and patterns that persist indefinitely. This tier requires two components working together:

• Vector Database (Qdrant or pgvector): Semantic fuzzy search. Finds "everything related to the user's database preferences" without exact keyword matching.
• Relational or Graph Layer (PostgreSQL): Precise retrieval by entity type, validity window, and relationship. Answers "what is the user's current home address?" unambiguously.

Vector-only storage cannot reliably answer factual lookups because semantic similarity is ambiguous. Structured storage alone cannot handle open-ended retrieval. You need both.

The Memory Injection Pipeline

Production systems run retrieved memory through a pipeline before it reaches the LLM:

async function buildAgentContext(
  userMessage: string,
  userId: string,
  sessionId: string
): Promise<AgentContext> {
  const [workingMemory, episodicContext, semanticFacts] = await Promise.all([
    getWorkingMemory(sessionId),
    queryEpisodicMemory(userId, { limit: 5, orderBy: 'recency' }),
    querySemanticMemory(userId, userMessage, { limit: 10 }),
  ]);

  // Resolve conflicts between tiers (newer facts win)
  const resolvedFacts = resolveConflicts([
    ...semanticFacts,
    ...episodicContext.entityChanges,
  ]);

  // Compact to fit context budget — critical step often missed in prototypes
  const compacted = await compactMemory(resolvedFacts, {
    maxTokens: 4096,
    strategy: 'relevance-weighted',
    referenceQuery: userMessage,
  });

  return {
    workingContext: workingMemory.recentTurns,
    longTermContext: compacted,
    systemPromptAdditions: buildMemorySystemPrompt(resolvedFacts),
  };
}

The compaction step is frequently skipped in prototypes and becomes a serious production issue at scale. Without budget-aware compaction, memory retrieval grows unbounded as users accumulate history and token costs explode.

Temporal Supersession: Handling Fact Updates

One of the hardest problems in agent memory: managing contradictions. A user tells the agent they live in London. Six months later, they mention they moved to Berlin. Both facts are now in the store. The agent must use Berlin—not London, and not both.

The solution is temporal supersession: storing facts with validity windows and automatically invalidating outdated records when contradictions are detected.

CREATE TABLE semantic_facts (
  id            UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id       UUID NOT NULL REFERENCES users(id),
  entity        TEXT NOT NULL,         -- e.g., 'user.location'
  value         TEXT NOT NULL,         -- e.g., 'Berlin, Germany'
  confidence    FLOAT NOT NULL,
  valid_from    TIMESTAMPTZ NOT NULL DEFAULT now(),
  valid_until   TIMESTAMPTZ,           -- NULL = currently valid
  superseded_by UUID REFERENCES semantic_facts(id),
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Always query only currently valid facts
SELECT value, confidence
FROM semantic_facts
WHERE user_id = $1
  AND entity = $2
  AND valid_until IS NULL
ORDER BY valid_from DESC
LIMIT 1;

When a new fact is stored for an entity that already has a current record, the old record's valid_until is set to now(). This creates an auditable history of how the agent's knowledge evolved—essential for debugging and compliance.

Vector Database Selection in 2026

For most teams: pgvector if already on PostgreSQL; Qdrant if you need dedicated vector performance. Always use hybrid queries—combining semantic similarity with exact metadata filters:

results = client.query_points(
    collection_name="agent_memory",
    query=embedding_of_user_query,
    query_filter=Filter(
        must=[
            FieldCondition(key="user_id", match=MatchValue(value=user_id)),
            IsNullCondition(key="valid_until", is_null=True),  # Only valid facts
        ]
    ),
    limit=10,
    with_payload=True,
)

Pure semantic search without metadata filtering will return facts belonging to other users and outdated facts in the same result set. Both are catastrophic in production.

GDPR and the Right to Be Forgotten

In EU, UK, and increasingly US state-level privacy regulations, users have the right to request deletion of personal data. An append-only memory system without per-record keys cannot comply.

The correct architecture uses per-record identifiers at every layer and a propagating deletion handler:

async function handleDeletionRequest(userId: string): Promise<DeletionReceipt> {
  const deletionId = crypto.randomUUID();

  await Promise.all([
    flushUserWorkingMemory(userId),                                // Tier 1
    db.episodicMemory.deleteMany({ where: { userId } }),          // Tier 2
    vectorDB.delete('agent_memory', { filter: { user_id: userId } }), // Tier 3 vectors
    db.semanticFacts.deleteMany({ where: { userId } }),           // Tier 3 facts
  ]);

  // Retain only the audit record — the fact of deletion is not personal data
  await db.deletionAudit.create({
    data: { deletionId, userId, completedAt: new Date() }
  });

  return { deletionId, status: 'complete' };
}

Never use anonymous bulk inserts into vector databases. Every record needs a stable ID that links back to your relational data model so it can be individually targeted for deletion.

Memory Staleness and Poisoning

Memory poisoning occurs when incorrect or outdated facts persist in long-term memory uncorrected. Common causes:

• The agent misunderstood a statement and stored a wrong inference as fact
• A fact that was true (job title, location) is now outdated but was never explicitly corrected
• The user provided incorrect information early in their history

Mitigation strategies:

• Confidence decay: Facts gain a score that decays over time. High-impact facts trigger lower-confidence retrieval after 90 days, signaling the agent to verify rather than assume.
• Correction detection: When a user explicitly corrects the agent ("No, I said Berlin, not Paris"), the correction is automatically flagged for fact update.
• Periodic reconciliation: For critical attributes, run a reconciliation job that compares semantic memory against canonical sources (user profile database) and flags divergences.

Benchmarking Your Memory System

Before shipping to production, validate against these tasks:

Conclusion

Three-tier storage—working, episodic, and semantic—handles the full lifecycle of agent knowledge without the cost and degradation issues of context-window stuffing. Temporal supersession handles fact updates correctly. Per-record IDs and propagating deletion make privacy compliance achievable.

Agents that remember build trust. Agents that forget build frustration. The memory architecture is where that distinction is made.