Why I gave Claude Code a memory (and what I got wrong about retrieval)

Every time I opened Claude Code on a project, I re-explained the project.

Not the code — it can read the code. The decisions. Why the auth module is structured that way. Which migration we abandoned and why. That the legacy/ folder is not legacy, it's load-bearing, and the name is a lie we haven't fixed yet. I'd type the same three paragraphs at the start of every session, or I'd skip them and watch the agent confidently re-derive a wrong conclusion I'd already ruled out last Tuesday.

CLAUDE.md helps, but it's a static file I have to maintain by hand, and it doesn't know what happened yesterday. What I wanted was a memory that fills itself.

So I built OpenMem: a small Go binary that wraps Claude Code, injects relevant project memory at start, exposes memory as MCP tools during the session, and captures what happened when the session ends.

This post is about the part that actually turned out to be hard, which was not the MCP plumbing. It was deciding what to show the model.

The shape of the problem

A memory layer for a coding agent has three jobs:

1. Capture — turn a messy session transcript into something worth keeping.
2. Store — keep it somewhere that survives, cheaply, with no external services.
3. Retrieve — at the start of the next session, pick the handful of things that matter for this session and put them in front of the model.

Job 3 is the product. Jobs 1 and 2 exist to make job 3 possible. I got this backwards at first and spent two weeks on ingestion before I had any idea whether the retrieved output was useful.

Three layers, not one

My first version stored one thing: session summaries. It failed in an obvious way — after twenty sessions, "summarise the last session" and "what is this project" are completely different questions, and a flat list of summaries answers neither well.

The version that works has three layers:

┌─────────────────────────────────────────────┐
│  Knowledge cards                            │  small, durable facts
│  "Prisma models are referenced by string    │  ← injected at start
│   in 14 places; grep before renaming"       │
├─────────────────────────────────────────────┤
│  Session summaries                          │  what happened, when
│  "2026-08-02: migrated auth to Lucia;       │  ← retrieved on demand
│   dropped the session table"                │
├─────────────────────────────────────────────┤
│  Transcripts                                │  the raw thing
│  full tool calls and outputs                │  ← only when asked
└─────────────────────────────────────────────┘

Knowledge cards are the thing the model most needs and the thing it's worst at producing. A good card is one durable fact with enough context to act on. "Uses Postgres" is useless. "Postgres via Supabase; local dev uses the pooler URL on port 6543, direct connection times out from Docker" is a card.

Session summaries answer "what happened recently". They decay in usefulness fast, which matters for ranking (more below).

Transcripts are kept but never injected. They exist so the agent can call openmem_transcript(session_id) when a summary isn't enough — "you said we tried X and it failed, show me the actual error".

The ingestion pipeline runs on session exit: an LLM pass extracts candidate cards, writes the summary, and — this took an embarrassingly long time to add — redacts anything that looks like a secret before it's written to disk. If your memory layer stores the API key that scrolled past in a tool output, you have built a very convenient exfiltration target.

Retrieval: where I was wrong

At session start OpenMem has maybe 5,000 tokens of budget to spend on memory. Which cards go in?

Attempt 1: full-text search. SQLite FTS5, query = the working directory name and the last git commit message. It's fast and it's in the same file as everything else. It also completely misses the case where the card says "authentication" and the commit says "login". FTS is a lexical filter, not an understanding.

Attempt 2: vectors only. Embed the cards, embed the query, cosine similarity. Better recall on paraphrase, worse on the thing FTS is great at: exact identifiers. If the commit mentions BullMQ, the card that mentions BullMQ should win, and vector similarity will happily rank a card about "background job queues" above it.

Attempt 3: both, fused. This is where it started working. Run FTS and vector search separately, take the top-k from each, and combine with Reciprocal Rank Fusion:

// score(doc) = Σ over lists of 1 / (k + rank_in_list)
// k=60 is the conventional constant; it dampens the top rank's dominance.
func rrf(lists [][]docID, k float64) map[docID]float64 {
    out := map[docID]float64{}
    for _, list := range lists {
        for rank, id := range list {
            out[id] += 1.0 / (k + float64(rank+1))
        }
    }
    return out
}

RRF is almost stupid, and that's the appeal: no tuning of score scales between two systems that produce incomparable numbers. A document that appears in both lists gets rewarded; a document that tops one list and is absent from the other still surfaces.

But two more problems showed up immediately.

Old summaries outranked new ones. A summary from three months ago about the auth migration matched the query beautifully — and was wrong, because the migration was later reverted. So: temporal decay. Each session summary's score is multiplied by exp(-λ · age_in_days). Knowledge cards do not decay (a fact about the pooler port is as true today as in March); only summaries do. Getting that distinction right — decay by layer, not globally — fixed more bad injections than anything else.

The top 5 results were the same fact five ways. After a week of sessions on one feature, five summaries all say roughly "worked on the payments webhook". Injecting all five wastes the budget. Maximal Marginal Relevance reranking fixes this: after scoring, pick greedily, and penalise each candidate by its similarity to what's already been picked. The injected set gets diverse, which is what a human would do when briefing a colleague — they don't say the same thing five times.

Final pipeline, per query:

FTS5 top-20 ──┐
              ├─► RRF ─► temporal decay (summaries only) ─► MMR ─► top-N by token budget
vector top-20 ┘

What "working" looks like

The test I trust is boring: open a project I haven't touched in two weeks, ask Claude Code something that depends on a decision we made back then, and see if it answers without me explaining. Before OpenMem, the answer was usually a plausible guess. After, it's usually right, and when it isn't, openmem_transcript gets it there in one step.

The failure mode I still watch for is the opposite one: a card that ranks well, reads confidently, and is out of date. That's why openmem_forget exists, and why I'd rather the agent inject four cards it's sure about than ten it isn't.

Things I'd tell past me

• Build the retrieval eval before the ingestion pipeline. A folder of 30 hand-written cards and 20 queries with expected answers would have saved me two weeks.
• Decay by layer. Facts don't age like events do.
• The redaction step is not optional and should be the first thing you write, not the last.
• Seven MCP tools is about right. I started with three (search, add, list) and the agent couldn't correct its own memory. Adding update, forget, transcript and stats turned it from a cache into something the agent treats as its own.

OpenMem is open source, and I'd love issues — especially "it injected something wrong" issues. Those are the ones that make the ranking better.