AI agent memory: how agents remember context

Most AI agents forget everything the moment a task ends. That's not a bug — it's the default. A language model has no persistent state. Each time you call it, you get a fresh start. For a single-turn chatbot that answers a question and moves on, that's fine. For an agent working across multi-step workflows, sessions, and users, it's a serious design constraint.

Memory is something you have to deliberately build in. This post explains the three types of agent memory, when to use each, and the design decisions that separate a useful agent from one that keeps asking you the same questions.

---

Quick answer

Agents don't have memory by default — each call starts fresh. There are three types you can design in: in-context memory (what's in the current prompt), external memory (facts and history retrieved from a store), and procedural memory (refined instructions that encode what the agent has learned). Most production agents need at least two.

---

Why memory matters for agents working across tasks

A stateless agent can answer "summarise this document." It can't answer "based on the last three reports you summarised, what's the trend?" — because it doesn't know about the last three reports.

As agents move from demos to production, memory becomes the central design question. Teams build a working agent, deploy it, and quickly discover that users expect it to know things: their preferences, past decisions, open issues, what was already tried. Without memory design, every session starts from zero.

The cost of this isn't just user frustration. It's operational: agents without memory can't improve over time, can't hand off context between sessions, and can't build on their own outputs.

---

In-context memory: what's in the prompt window

In-context memory is the simplest form — it's everything in the current prompt. System instructions, conversation history, retrieved documents, tool outputs. If it's in the prompt, the model can use it. If it's not, the model doesn't know it exists.

What it's good for: short tasks, single sessions, passing structured data from one step to the next within a workflow run.

Where it breaks down: context windows have limits. Long conversations, large documents, and multi-session histories all push against those limits quickly. You can't fit a year of customer history into a prompt. And even if you technically could, more context means higher latency and cost — and often lower quality, because models don't weight everything in a long context equally.

In-context memory is necessary but not sufficient for most production agents. It covers the current task. Everything else needs something else.

---

External memory: storing and retrieving what matters

External memory is a datastore — a database, a vector store, a file — that the agent can read from and write to. The agent doesn't carry the information in its prompt; it retrieves what's relevant when it's needed.

This is where retrieval-augmented generation (RAG) fits. The agent takes a query, retrieves the most relevant chunks from the store, and includes them in context. The prompt stays manageable; the memory can grow indefinitely.

External memory works well for:

• Customer history — past interactions, preferences, open issues
• Task outputs — summaries, decisions, artefacts from previous runs
• Domain knowledge — internal documents, policy, product information
• Cross-session state — anything that needs to persist beyond a single conversation

The design question isn't just "should I use external memory?" It's "what should I store, and how should I retrieve it?"

Vector stores

A vector store converts content into numerical embeddings and indexes them so you can search by semantic similarity. When the agent needs to retrieve something, it embeds the query and finds the closest matches — not by exact keyword, but by meaning.

This is the right choice when you're working with unstructured content: support tickets, internal documentation, email threads, product descriptions. The agent doesn't need to know the exact phrasing used in a document — it just needs to find content that's conceptually related to what it's working on.

The practical trade-off: vector search is excellent at finding relevant chunks from large corpora, but it doesn't handle precise lookups well. If you need to find a customer's account status or the exact price of a product, you don't want semantic similarity — you want a database query. Vector stores and structured databases serve different retrieval needs, and most production agents use both.

When to use a vector store:

• Searching a knowledge base or documentation corpus
• Finding past tickets or cases similar to the current one
• Retrieving relevant policies or procedures from unstructured text
• Matching user intent against a catalogue of workflows or options

Structured databases for exact retrieval

For facts with clear identifiers — account records, order history, user preferences, product data — a structured database is the right tool. You query it by ID or field, you get back a precise record. No approximation, no chunking, no embedding required.

The pattern here is straightforward: the agent uses a tool (a function call to your data layer) to look up a specific record, then includes the result in context. The tool takes a known identifier as input and returns a structured response.

When to use structured retrieval:

• Looking up a specific user's account or history by ID
• Fetching current state of a task, order, or case
• Retrieving fields that need to be exact (dates, prices, statuses)
• Checking permissions or entitlements before taking an action

Retrieval patterns in practice

The two patterns most agents use are query-time retrieval and pre-loaded context.

Query-time retrieval means the agent decides what to look up based on the current task. It calls a retrieval tool mid-run, gets back relevant content, and uses that in the next step. This is flexible and efficient — you only fetch what's actually needed. The downside is latency: every retrieval adds a round-trip.

Pre-loaded context means you retrieve relevant information before the agent starts — typically at session start — and include it in the system prompt or early context. This works well when you can predict what the agent will need (for example, always loading a customer record at the start of a support session). It reduces mid-run latency but requires you to know what's relevant upfront.

Most real agents combine both: pre-load high-confidence context (customer profile, current task state) and retrieve dynamically as the task evolves.

---

Procedural memory: what the agent has learned to do

Procedural memory is different from the other two — it's not stored facts or conversation history. It's encoded knowledge about how to do the task: refined instructions, updated prompts, accumulated heuristics.

A simple example: an agent that drafts customer emails starts with a generic prompt. After a few hundred runs, you've learned what tone works, what errors are common, what edge cases need special handling. That learning can be encoded back into the system prompt — the agent effectively "remembers" what it took to do the job well, even though it doesn't remember individual emails.

How this works in practice:

The update cycle has three steps. First, review outputs — either manually or by running an evaluation pass over a sample. Look for patterns: types of errors that repeat, cases where the agent hedged unnecessarily, formatting issues, edge cases that keep showing up. Second, distil those patterns into instruction updates. Not "fix this output" but "when the customer mentions X, always do Y." Third, test the updated prompt against the same sample before deploying.

Some teams build a meta-agent to automate the review step — a separate agent that reads output logs, classifies failure types, and drafts instruction updates for a human to approve. This works well at scale, but a human review cycle is more reliable early on, when you don't yet know what failure categories matter most.

What procedural memory is not:

It's not fine-tuning. Fine-tuning bakes facts and behaviours into model weights — it's expensive, opaque, and goes stale when conditions change. Procedural memory lives in the system prompt, which means it's readable, editable, version-controllable, and can be updated without touching the model.

It's also not automatic. The agent doesn't update its own instructions mid-run. The loop is: observe outputs → identify patterns → update instructions → test → deploy. That loop should run on a cadence, not continuously.

Procedural memory is updated deliberately, not automatically. But it's often more valuable than conversation history because it improves every future run, not just continuations of a specific thread.

---

What to remember vs. what to forget

Not everything should persist. Designing memory means deciding what's worth keeping — and what creates noise, privacy exposure, or stale data problems.

Keep: facts that change behaviour (user preferences, past decisions, relevant history), outputs that will be referenced again, domain knowledge that won't be in the base model.

Discard: transient reasoning steps, intermediate outputs that aren't referenced downstream, raw conversation turns that add volume without value.

Scope carefully: memory that persists across users needs access controls. Memory that persists across agent versions needs versioning. Memory that comes from external sources needs freshness checks. None of these are hard problems, but they all need to be in the design.

The failure mode to avoid is the firehose: storing everything, retrieving too broadly, and flooding the prompt with context the model can't usefully weight. More memory isn't better — relevant memory is better.

---

Worked example: a support agent that remembers

A support agent handles inbound customer tickets. Without memory, it treats every ticket as new — no history, no preferences, no record of past resolutions. The agent asks the customer to re-explain their setup on every contact. It escalates issues that were already escalated and resolved last month. It can't notice when a problem is recurring.

With memory design, here's how the three types work together:

In-context memory covers the current interaction: the ticket text, a summary of the customer's last three contacts (retrieved at session start), and any tool outputs from the current run. This is what the agent is actively working with.

External memory has two components. The first is a structured customer record store, queried by account ID at session start. It returns: account tier, known configuration, open issues, and a log of the last ten interactions with dates and resolution status. The second is a vector-searchable resolution library — thousands of past tickets, each with the problem description and the resolution applied. When the agent identifies an issue type, it searches this library for similar past cases and retrieves the top three resolutions. These get included in context before the agent drafts a response.

Procedural memory is the system prompt, updated weekly. The update process: a support lead reviews a sample of 50 tickets from the previous week, tags any where the agent's first-draft response was wrong or off-tone, and identifies the pattern. Common additions have included: a rule about not suggesting configuration resets for enterprise accounts without checking their contract tier first, a list of known product bugs to mention proactively when certain symptoms appear, and a tone instruction for customers who have contacted support more than five times in a month.

The result is an agent that can say: "You reported this same error last month. We resolved it by reverting your integration settings — has that approach stopped working?" That's not model intelligence. It's memory architecture. The model is the reasoning engine; the memory system is what gives it context to reason from.

---

Common mistakes

Over-stuffing context. Dumping everything you have into the prompt slows the agent, raises cost, and often degrades quality. Retrieve only what's relevant to the current task.

Relying on model memory. Fine-tuned models can encode facts, but that knowledge goes stale and can't be updated without retraining. External memory is easier to maintain and easier to audit.

Forgetting to scope what persists. Memory without access controls is a security and privacy problem. Memory without expiry becomes a garbage pile. Design the boundary.

No memory at all. The opposite failure — building an agent that handles repeated tasks but starts from scratch every time, repeatedly asking users for context they've already provided.

---

Where Envelope fits

When you design an agent in Envelope, the role definition includes what the agent knows at runtime — its tools, its scope, its inputs. External memory connections (retrieval, customer records, knowledge bases) are modelled as tools. The design makes explicit what each agent can access and what persists between calls — before you build anything.

→ How to write a good AI agent role definition → Sub-agent design: the five components every agent needs → How to design AI agents: a practical guide

---

Frequently asked questions

Do AI agents have memory by default?

No. A language model starts fresh with every call — there's no persistent state between requests. Memory is something you explicitly design into the agent architecture, either by including history in the prompt, connecting an external store, or encoding learned behaviour into the system instructions.

What's the difference between in-context memory and external memory?

In-context memory is everything currently in the prompt window — conversation history, instructions, retrieved data. It exists only for the current call and disappears when the call ends. External memory is stored outside the model (in a database or vector store) and can persist indefinitely. The agent retrieves what's relevant into context when needed.

What is RAG and how does it relate to agent memory?

RAG (retrieval-augmented generation) is a pattern for implementing external memory. The agent takes a query, searches a knowledge store for relevant content, and includes the results in the prompt before generating a response. It's the standard approach for giving agents access to large bodies of knowledge that won't fit in a context window.

How do I decide what an agent should remember?

Start with the question: what information, if the agent had it, would meaningfully change its outputs? Customer history, past decisions, domain-specific facts, and unresolved issues are usually worth storing. Transient reasoning steps and intermediate outputs usually aren't. Also consider freshness — stale memory can be worse than no memory.

Can an agent improve its own instructions over time?

Not automatically — but you can design a process that does this. Review the agent's outputs periodically, identify patterns in what works and what fails, and update the system prompt accordingly. This is procedural memory: encoding what the agent has learned back into its instructions. Some teams build a meta-agent that does this review step, but a human review cycle is often more reliable early on.

What's the biggest memory mistake teams make?

Storing too much and retrieving too broadly. It's tempting to log everything and include it all in context, but this inflates prompts, increases latency and cost, and often produces worse outputs because the model struggles to weight a noisy context. The better approach is to be deliberate about what you store, retrieve only what's relevant to the current task, and keep memory scoped to what actually changes agent behaviour.