Agent Memory

How to add memory support to your agent

A guide for adding Agent Memory support to an AI agent or harness.

The integration has three responsibilities: compile the memory root into context, tell the model how to reach the rest, and keep the loaded copy fresh. If your agent already has file tools, most of the work is prompt assembly.

Prerequisites: Familiarity with the specification, which defines the directory layout, the core contract, and the optional extras.

The core principle: three tiers

Like Agent Skills, memory loads through progressive disclosure. The harness owns tier 1; the model drives tiers 2 and 3 with its file tools.

Tier What's loaded When Loaded by
1. Core Every root Markdown file, including MEMORY.md when present Session start Harness
2. Discovery A nested MEMORY.md or directory listing When an index or task leads to a folder Model
3. Content Individual memory files When an index or listing points to them Model

Step 1: Locate the memory root

The spec does not mandate where memory directories live, only what is inside them. Choose a location that matches your scoping model:

Scope Example location Used by
Per agent ~/.<client>/agents/<id>/memory/ Letta Code (agents are first-class, memory travels with the agent)
Per project ~/.<client>/projects/<project>/memory/ Claude Code
Per user ~/.<client>/memories/ OpenClaw, Hermes, Codex

Per-agent scoping is the most general: a "user" memory is just a single-agent deployment. Whatever the scope, the directory itself follows the same format, which is what makes memory portable between clients.

If a .memoryspec version marker is present, read it to select layout handling. If your client also supports older layout generations, the marker is how you tell them apart without guessing.

Step 2: Compile the root into context

At session start (or prompt compilation time):

  1. Load every Markdown file at the memory root into the system prompt or an equivalent always-visible context region. Skip dotfiles and non-Markdown files.
  2. Label the region so the model can tell memory apart from other instructions, and include the absolute path of the memory root so file tools can reach it. The harness does not need to inject a separate directory listing.
<memory root="/home/user/.myclient/agents/a1/memory">
  <file name="MEMORY.md">...</file>
  <file name="persona.md">...</file>
  <file name="human.md">...</file>
</memory>

If a root Markdown file carries YAML frontmatter, you may strip it from the compiled prompt or leave it in place. Both are conforming.

Step 3: Tell the model how to use memory

Add a short instruction block alongside the compiled memory. It needs to convey three things:

  • Root Markdown files are already loaded; there is no need to re-read them.
  • Files below the root are not loaded. To recall from them, follow the root MEMORY.md or use file tools to explore, then read specific files.
  • If the harness provides write access, explain where the agent may save durable knowledge and which attached files or folders are read-only.

Keep it concise; models already know how to use file tools. The instructions are pointing, not teaching.

Step 4: Freshness and optional writes

Honor the freshness rule regardless of who changes memory: when a root Markdown file changes, the model must either see the new content or be told its copy is stale.

Conforming strategies, in increasing order of effort:

  • Session-start compilation: the prompt is compiled once per session; edits apply next session. Tell the model this explicitly so it writes for its future self rather than expecting instant effect.
  • Stale markers: after a write to a root Markdown file, inject a notice that the loaded copy is outdated.
  • Recompilation: rebuild the memory region on change (or at commit boundaries, if memory is git-tracked). Note the cache tradeoff: memory that changes mid-session invalidates prompt-cache prefixes, so batching refreshes at natural boundaries is usually better than refreshing per edit.

Write access is optional. A harness may expose all memory as read-only, provide one writable area, or apply different permissions to different attached files and folders. If the model can write, it may use ordinary file tools or a dedicated memory tool that mediates validation and auditing.

Step 5: Optional extras

  • Git: initialize the memory directory as a repository and commit on change, with the author identifying who wrote it (agent or harness). This buys traceability, portability, and recovery. See the specification.
  • Budget enforcement: warn (or refuse the write) when the root exceeds the recommended budget, with a message telling the agent to demote detail into a directory. A pre-commit hook is the natural place when memory is git-tracked; see the example hook.
  • Index validation: check MEMORY.md entries against the real tree and surface dangling or missing entries to the agent as feedback rather than failing silently.
  • Managed indexes: a harness may generate all of MEMORY.md, or maintain a file-tree section while leaving descriptions and retrieval guidance to the agent.

Cloud-hosted and sandboxed agents

If the agent's harness runs remotely, the memory directory needs to reach the machine where prompt compilation happens. Git-tracked memory makes this a clone-and-pull problem: the memory repository is the unit of synchronization, and commit boundaries become the natural refresh points. Object storage or a database work too, as long as the projected directory obeys the core contract when the agent's file tools look at it.

Conformance checklist

  • Root Markdown files (excluding dotfiles) are loaded into always-visible context.
  • Directory contents are not eagerly loaded.
  • The agent can read any file in the memory tree with file tools.
  • Root-file changes are reflected or flagged per the freshness rule.
  • Memories that use no extras (no frontmatter, no git, no .memoryspec) still load.