Every agent memory product promises that your agents will remember. This checklist tests how effective an agent memory product is. It is arranged around five ways agent context fails, with questions to ask the vendor and checks to run for each.
What you need to complete it
Ask each vendor to send its benchmark results ahead of time.
Sections 1 to 3 test how a product decides which of two disagreeing memories wins.
1
Section 1: Which one does the agent act on?
0/2
Ask the vendor
“If an agent recalls two things that both apply to the task, and those things disagree, which one does the agent act on and why?”
Use an example from your own business. A typical one is a contract price, a promotion on the same SKU, and a sales rep's approved exception, all valid on the same day.
✓A good answer points to a stored rule that says which item outranks the other, under what condition, and names the person who owns and approved it.
✕A weak answer is "the newest one," "the closest match," or "the model decides from context." In each case the store ranks items and the model still makes the call.
2
Section 2: Was it replaced, or are both still in force?
0/2
Some memories should expire when a new one arrives, such as a new price that replaces an old one. Others stay valid side by side, such as a contract price and a promotion that are both live the same day.
Ask the vendor
“How does your store tell an item that was replaced from two items that are both in force?”
✓A good answer uses validity dates for replacement and gives you a way to record which item governs when both apply.
✕A red flag is validity dates alone. They retire stale items, and they cannot tell you which of two live items wins.
3
Section 3: What does the decision trace leave out?
0/4
Many memory products store decision traces: what the agent saw, what it did, who approved it, and the reason given. A trace is evidence about one past case. Reused as precedent, it can end up applied to cases it never covered.
Run this audit on your own agent's history:
If most traces leave out the condition, the agent can reuse them in cases where they do not apply, and nobody reviewing the output will see that a rule was skipped.
Benchmarks to ask forMemoryAgentBench, conflict resolution track (facts that change or conflict), and LongMemEval knowledge-update questions (a fact that was replaced).
02
Failure 2 of 5
Context poisoning
A wrong fact gets saved, and the agent keeps building on it.
4
Section 4: Can a wrong memory be taken back?
0/1
A wrong fact, such as the model's own mistake, gets saved. The agent keeps building on it, task after task.
Ask the vendor
“If a wrong item gets into memory, how do we find it, remove it, and see which decisions used it?”
Benchmark to ask forLongMemEval abstention questions (does the agent say it does not know instead of inventing an answer?). No public benchmark tests removing a wrong memory, so run the check below.
✓Good sign: each item is traced to its source, can be corrected, and lists the decisions that used it.
✕Red flag: no record of where an item came from, so nobody can find it.
03
Failure 3 of 5
Context distraction
A long history crowds out the task in front of the agent.
5
Section 5: Does a long history crowd out the task?
0/1
As history grows, agents lean on what they did before and repeat it, instead of reading the new case.
Ask the vendor
“What does the agent load for a task, and what keeps old history from crowding out the instructions?”
Benchmarks to ask forLongMemEval (recall from a long, mostly unrelated history) and LoCoMo (recall across long conversations).
✓Good sign: it loads only what the task needs, and summarizes or clears old history on a schedule.
✕Red flag: it loads everything it has, on the idea that more context is better.
04
Failure 4 of 5
Context confusion
Memory from one customer, project or user leaks into another.
6
Section 6: Does memory from one place leak into another?
0/1
A note saved for one customer, project or user shapes a task for another. The output looks reasonable, so nobody notices.
Ask the vendor
“How is memory scoped, and can something saved for one customer change a task for another?”
BenchmarkNo public benchmark tests whether memory stays within one customer or project. Run the check below on your own data.
✓Good sign: memory is scoped by customer, project or user, with access rules your team sets.
✕Red flag: one shared pool, with the model deciding what is relevant.
05
Failure 5 of 5
Literal reading
A vague request is followed word for word, and may be saved as a rule.
7
Section 7: What does the agent do with a vague request?
0/2
People send short notes and quick prompts that leave things out. Newer models tend to follow instructions literally, so the agent does what was written, not what was meant. Saved to memory, that note becomes a standing instruction.
Ask the vendor
“When a request is vague, does the agent ask or act on its literal reading? Does a one-off request get saved as memory?”
Benchmark to ask forSTATE-Bench (whether agents follow business procedures when a simulated user makes requests).
✓Good sign: the agent flags what is unclear, checks stored rules first, and saves a request as a rule only after someone approves it.
✕Red flag: it acts on the literal reading and saves every message as memory.
Vendor scorecard
Write each vendor's answer in the last column, then compare.
Loads only what the task needs; old history summarized or cleared
Loads everything it has
Failure 4 · Context confusionBenchmark: No public benchmark
6
Can memory leak from one customer or project to another?
Scoped by customer, project or user, with rules your team sets
One shared pool; the model decides what is relevant
Failure 5 · Literal readingBenchmark: STATE-Bench
7
What happens with a vague request?
Agent asks, checks stored rules, and saves a rule only with approval
Acts on the literal reading and saves every message
Decide where precedence lives
Memory is worth funding, because agents that run for weeks need to keep what they learn. Before you pick a product, decide where precedence lives and who owns it. If it lives in the store's ranking or the model's judgment, the model settles each conflict as it comes. A rule layer owned by the business, with memory and decision traces kept beside it, gives you an answer you can audit.
Trail is the context layer your agents run on. It holds rules, exceptions and dependencies as a graph any agent can follow, with each rule traced to the document it came from. When two rules conflict and nobody has set precedence, Trail asks the owner. See it on your own policy documents.