AI Companion Memory Testing: How We Evaluate Recall Accuracy

Why we test memory systematically

Every AI companion app claims 'she remembers you,' but the actual recall accuracy varies enormously. Our testing methodology gives you concrete data instead of marketing promises — we plant specific facts, wait specific intervals, and measure what comes back.

This matters because memory quality directly affects how real the relationship feels. A companion that forgets your name after three days is a fundamentally different product from one that remembers your birthday six months later.

Our testing protocol

We establish a standardized set of 20 facts across five categories: personal details (name, age, job), preferences (favorite food, music, color), relationship events (first conversation topics, shared jokes), emotional context (stressful events mentioned, happy moments), and contradictory updates (changing a previously stated preference).

We then test recall at four intervals: immediately after the conversation, 24 hours later, one week later, and one month later. Each test uses natural conversation rather than direct quizzes — we bring up topics where the facts should naturally surface and see if the companion incorporates them.

How we score results

Each fact gets one of four scores: recalled accurately, recalled with errors (right concept, wrong details), recalled when prompted but not volunteered, or not recalled at all. We weight accuracy higher than prompted recall, since a companion that naturally weaves your details into conversation feels more real than one that only remembers when asked directly.

The final score for each app is a weighted average across all 20 facts and all four time intervals, giving you a single number that captures overall memory quality.

What we have found so far

The best-performing apps recall around 85 percent of facts accurately at the one-month mark. The worst recall fewer than 30 percent even at the 24-hour interval. The biggest differentiator is not the language model but the memory architecture — apps with dedicated fact-extraction and retrieval systems dramatically outperform those that rely solely on context window size.

Contradictory updates are the hardest category: most apps struggle to properly overwrite a previously stored fact. If you tell your companion you moved from Chicago to Denver, some apps will confidently reference both cities in the same conversation.