How Mama remembers what you tell her

Google Cloud recently published a video about long-term memory for AI agents. Watching it, I recognised many of the same decisions we'd made while building Mama's memory over the past year.

This week I watched a video from Google Cloud Tech about how AI agents should remember things about the people they talk to.

The example was a fashion assistant that remembers what you’ve bought, what’s on your wishlist and the styles you tend to like.

As Annie went through the design, I realised how closely it matched what we’d built for Mama. There’s nothing in it I disagree with. We hadn’t followed Google’s approach. We’d arrived at something similar through a year of changes, starting when Mama launched in June 2025.

The video explains it neatly, so I thought I’d use it to show how Mama’s own memory works.

Two kinds of things to remember

The video starts with a distinction that makes a lot of sense.

Some things need to be remembered exactly. If you have a dinner on Thursday, or you bought the blue jacket, the assistant should know exactly that. That information belongs in a database where it can be looked up.

Other things are less precise. Maybe you prefer simple clothes, or tend to choose darker colours. Those are things the assistant learns from talking to you, rather than individual records you’ve asked it to save.

Mama has the same two kinds of information.

The first is what users deliberately record with her: expenses, income, bill reminders and spending caps. These are stored as normal records in Postgres.

The second is what Mama learns through conversations. Someone might mention that they’re saving for a wedding, worrying about a car loan or planning to change jobs. Those details can help her give more relevant advice later.

We store that information as memories, with embeddings that let us search them by meaning using pgvector, and as shorter insights about the user. It’s still the same Postgres database, just a different way of storing and retrieving things.

Save the expense, don’t remember it

If you tell Mama “Lunch RM15”, she doesn’t try to remember that you spent RM15 and write it down later.

The classifier routes the message to expense mode, and Mama calls a tool that records the expense in the database. By the time she replies, the expense is already saved.

Reminders follow a similar process.

Tell her to remind you to pay rent on the 1st, and the reminder tool works out the next date and time in your timezone, checks for duplicates and saves it. When the reminder is due, a scheduler picks it up and, since we moved everyone to the app, sends it as a push notification.

The same principle applies when someone asks about information they’ve already recorded.

Mama isn’t allowed to answer spending questions using numbers she happens to remember from a previous conversation. Her coaching prompt makes that clear:

Never answer a spending question from memory or from context you were given: numbers you did not just read are numbers you are guessing at.

If you ask how much you spent on groceries, she needs to query your expenses. If you ask what reminders you’ve set, she calls the tool that lists them, even if she created one just two messages earlier.

Remembering that someone is worried about overspending is useful. Remembering an old total and presenting it as their current spending isn’t.

Learning from conversations

The other kind of memory is less straightforward.

Someone might casually mention that they’re considering buying a car. That doesn’t necessarily mean Mama should remember they’re buying one.

The Google video makes a similar point. A single comment about liking yellow shouldn’t immediately become a permanent preference for yellow clothing.

We also don’t want Mama to spend time analysing what to remember while someone is waiting for a reply, so we do that work in the background.

After Mama replies in general chat or coaching mode, the conversation is added to a small buffer for that user. Routine expense logging and other record-keeping messages are excluded.

When the buffer reaches five exchanges, or the conversation has been quiet for ten minutes, we send that block to a smaller model to look for anything worth remembering.

The ten-minute rule is our rough equivalent of a session ending. Unlike the live call in Google’s demo, which has a clear start and finish, Mama’s chat is one continuous conversation.

The model looks for things such as financial goals, decisions, commitments, worries and important life events. Greetings, routine logging and ordinary small talk are skipped.

If there’s something useful, it writes a short summary and gives it an importance score from 1 to 5. A major life event, such as having a baby or starting a new job, scores higher than a passing mention of a favourite brand.

That summary is embedded and saved as a memory.

We then run another pass to turn the memory into smaller insights. Each insight is one short statement, no more than 20 words, with a category and a confidence score.

An insight based on something the user explicitly said gets a higher confidence score than one the model inferred from the conversation.

Not remembering everything

We don’t want Mama saving the same thing repeatedly just because someone talks about it often.

Before storing a new insight, we compare it with the user’s existing insights of the same type. We first check for matching text, then ask the model whether the new insight means the same thing as something already saved.

The duplicate check has to be careful, though. If someone mentions a savings goal of RM3,000 and later talks about RM5,000, those are two different goals. The comparison is told that insights with different names, dates or amounts are not duplicates.

We also limit each user to 30 insights. When that limit is reached, the oldest one is removed to make room for a new one.

There’s a slower process as well.

Every 40 messages, we create a conversation summary. Every three summaries are combined into a longer one, and a daily job examines the new summaries for insights that the more immediate process might have missed.

Remembering at the right moment

Saving memories is only half the job. Mama also needs to know which ones are relevant when someone messages her again.

In Google’s example, the assistant loads information at the beginning of a session. It might greet someone by mentioning an upcoming dinner or a clothing preference.

Mama works a little differently because there’s no clear beginning to a session. Instead, she gathers relevant context whenever a new message arrives.

The message is embedded once, and several lookups happen together. Mama gets the recent conversation, relevant conversation summaries, up to five memories that are close in meaning, and up to eight insights about the user.

If none of the saved memories is a close enough match, she falls back to the five most recent.

The insights work differently from the memories. Rather than searching them by meaning, we take the 40 most recent insights and rank them using confidence and age.

We added the age weighting in July because older insights were becoming too persistent. Previously, every insight went into the prompt, and an old observation with a high confidence score could keep appearing long after it had stopped being particularly relevant.

Now insights lose weight as they get older, with a half-life of 45 days. An insight that’s three months old counts for roughly a quarter of an equally confident new one.

We keep the top eight, with the result cached for 30 minutes.

Not every message needs all this work, either. If someone just says “Thanks Mama”, there’s no reason to search through their memories before replying, so small talk skips the lookups.

And the rule about financial records still applies. None of these memories replaces a database query when someone asks for an actual amount.

There’s one more difference between Mama and the assistant in Google’s demo.

In the video, the assistant might open with something like “I know you like to keep it simple.” Mama is told not to do that. Her instructions say:

Use these insights silently to shape your advice. Do not announce that you remember them or recite them back to the user.

If Mama knows someone is saving for a wedding, that should help her give more appropriate advice. She doesn’t need to remind the person that she knows about the wedding every time they talk about money.

Where Google’s approach differs

Google’s video uses Memory Bank on its Agent Platform, which handles extracting and storing memories from previous sessions.

We built our own version on top of Postgres and pgvector.

The conversation buffer, importance scoring, insight extraction, duplicate checks and age weighting are all parts we’ve added over time.

The video also covers another kind of information: raw input such as camera frames and audio. The recommendation is to keep the meaning rather than store all the raw material as the agent’s memory.

Mama doesn’t watch videos, but she does read receipts.

When someone sends her a receipt, we extract the information needed to record the expense. The receipt becomes a financial record, not something for her long-term conversational memory.

Watching the video, what struck me was how tidy the design looked when explained all at once.

For us, it was a year of adding things as we needed them, from saving conversation summaries to deciding which memories were important enough to keep, and eventually making older insights less influential.

We ended up with much the same separation Google describes: record the things that need to be exact, and let the assistant learn the rest without interrupting the conversation.

It’s nice to see someone else arrive at a similar approach.