Database and Vector-Store Memory: When Files Aren't Enough

I asked an agent with vector-backed memory what I'd told it about a project's launch date. It came back with the right topic and the wrong date: a similar-sounding memory from a different conversation, ranked as the closest match. That's the moment I stopped assuming "add a vector database" meant "the agent now has memory."

Long-term memory for AI agents means storing facts outside the context window, in a file, database, or vector store, and retrieving them on a real trigger, not just bolting on semantic search and hoping. Which storage you pick depends entirely on what kind of fact you're storing and how you need to find it again.

Key Takeaways

• Files, databases, and vector stores sit on one spectrum: most agents don't need to reach the far end of it
• A vector store finds text that's similar to your query, not necessarily the fact that's correct
• Structured facts (settings, decisions, plan tiers) belong in database rows, not embeddings
• Memory that's never queried on a real trigger might as well not exist

When a File Stops Being Enough

A file stops being enough for agent memory once you can't reliably find a fact in it without re-reading the whole thing. Up to that point, a flat file, the kind you built in progress files and shift logs, is genuinely the right tool. Reaching past it earlier is over-engineering.

The spectrum runs: a plain file, for a handful of facts an agent can scan in full. A database, once those facts are structured and you need to filter or update them by field. A vector store, once you need to find facts by meaning rather than exact match.

Each step adds real cost: more infrastructure, more failure modes, more latency. The right move is picking the lightest tool that actually solves the query you have, not the most impressive one.

Files, databases, and vector stores compared

Structured Facts Need a Database, Not a Vector Store

Structured facts need a database because the query you're actually asking ("what plan is this user on," "what did we decide about X") has one correct answer, and exact match beats similarity search for that. A vector store returns the closest embedding, which is a probabilistic answer to a question that should have a deterministic one.

Think about what breaks if you get this backwards. Store a user's subscription tier as an embedding, and a semantically similar-but-wrong record can outrank the real one when two support tickets happen to use similar phrasing. Store it as a row in a table with a user_id and a tier column, and the lookup is exact, fast, and cheap: no embedding cost per query, no similarity threshold to tune.

This is also where most of an agent's genuinely long-term memory lives in practice: past decisions, resolved preferences, things that were true once and are still true. A small SQLite file or a hosted Postgres table (the same kind of setup from Getting Started) handles this without any of the machinery a vector store needs.

Semantic Recall Needs a Vector Store

Semantic recall needs a vector store when the question is "did we ever talk about something like this," not "what's the exact value of this field." That's a fundamentally different query, and it's one a database can't answer well: SQL doesn't know that "the client wants faster onboarding" and "reduce time-to-first-value" mean roughly the same thing.

This is the mechanism you already built in embeddings, vector stores, and RAG: text gets converted into a vector, similar meanings land near each other in that vector space, and a query pulls back the nearest matches. Applied to agent memory instead of document retrieval, the same store now holds facts about this specific agent's history (things a user said, conclusions the agent reached, context from three sessions ago) instead of a fixed knowledge base.

The catch, and the one that caught me: similarity is not correctness. Architecture analyses of agentic memory put this plainly: a vector database is one component inside a memory layer, not a replacement for it, and it hits real ceilings on temporal reasoning, exactly the "find the one thing that happened in a specific order" problem. Two facts about the same project, weeks apart, can come back ranked as equally relevant with no signal about which one is current. If your agent needs to know sequence: what changed last, what superseded what, it needs a timestamp column sitting next to the vector, not a vector alone.

Recall Triggers: When Does the Agent Go Look?

A recall trigger is the specific moment an agent decides to query its memory store, and without one, even a well-built memory system never gets used. Storing facts is the easy half of this; the harder half is deciding when the agent stops and asks "have I seen something relevant to this before?"

Three triggers cover most real agents:

• Session start: before the first response, pull anything tagged as relevant to this project or user, so the agent doesn't re-ask questions it already has answers to.
• Explicit reference: the user says "like we discussed" or "the thing from last week," and that phrase itself is the signal to query memory instead of context.
• Tool-call gate: before certain actions (sending an email, changing a setting), the agent checks memory for a relevant past decision, the way you'd check a changelog before touching old code.

Skip this step and the failure mode isn't dramatic, it's quiet: the memory system technically works, gets queried maybe twice, and everyone assumes the agent "just doesn't remember things" when the real problem was never wiring up a trigger.

Giving an Agent Vector-Backed Memory

To test long term memory for AI agents honestly, I gave a Claude Code agent vector-backed memory using the exact store from lesson 17.07 and the fact I chose to test with was deliberately boring: a decision about which linter config to use, made in a session I closed and didn't reopen for two days. Boring facts are the honest test: the agent isn't primed to remember something dramatic, and a real production agent's memory is mostly boring facts like this one.

Even the best-scoring memory systems on this measure aren't perfect: the 2026 State of AI Agent Memory report shows leading systems still miss a meaningful share of long-horizon recall questions on the LongMemEval benchmark, which is why the similarity-threshold filter below isn't optional.

The setup is small on purpose. One collection in the vector store, one write path (store a fact with a short description and a timestamp), one read path (embed the current query, pull the top matches, filter by a similarity threshold before trusting the result). The filter step matters: without it, a low-confidence match gets treated the same as a strong one, which is exactly the wrong-date failure from the opening of this lesson.

A files-vs-database-vs-vector spectrum still assumes a human decides what gets stored and when. The research frontier on self-editing memory hands that judgment call to the agent itself, which the next lesson covers.

Your Lab