How Mama's knowledge base works, and what we changed this week

Mama answers money questions from our own library of Mr Money TV videos and articles. This is how the search works, why we added a reranker, and three changes we made this week.

When you ask Mama a money question, like whether to pay off a car loan early or what tax relief you can claim, she doesn’t answer from whatever the model happens to know. She searches our own library first.

Most of that library is Mr Money TV videos, with articles on the Budget, tax relief and EPF alongside them. The idea is that Mama gives the same advice and reasoning you’d get from the channel, in her own voice.

This is the part of the system people usually call RAG, retrieval-augmented generation. We changed a fair amount of it this week, so it seemed like a good time to write down how it works and how it got here.

The first version didn’t last a month

In July 2025 we built the first knowledge base on OpenAI’s hosted file search. You upload your documents, and OpenAI handles the chunking, the embeddings and the search.

It wasn’t great.

The text that came back had a lot of odd artifacts in it, and Mama ended up quoting them in her replies. The answers also sounded like a generic assistant rather than Mama, so we added a second model call whose only job was to rewrite each answer in her voice.

Two calls to produce one answer, and the first one was feeding the second bad material. By the end of that month we’d decided to build our own pipeline, and we had one.

Getting a video into the library

A video doesn’t go straight in. It gets transcribed, then rewritten and cleaned up: ringgit amounts written consistently, KWSP turned into EPF, the filler and the sponsor segments and the “like and subscribe” taken out. Then it’s fact-checked.

Then it’s chunked. A model can split the text into self-contained thoughts of under a hundred words each, or the code can cut it into 500-token windows with a 50-token overlap. Either way, someone reads the chunks and can edit them before they’re saved.

Each chunk is embedded with the entry’s title in front of it, so a search for “Budget 2026” finds every chunk of the Budget 2026 article, but only the chunk text itself is ever shown to Mama. The embeddings come from OpenAI’s text-embedding-3-small.

The library is about 430 chunks today. That’s small enough that we don’t use a vector index at all. Every search compares against everything.

Similarity wasn’t enough

Only the coach reads the library. Logging an expense or listing reminders never touches it.

For a long time, the search was simple. Embed the question, find the ten closest chunks by cosine similarity, keep the ones above 40%, and put them in front of Mama.

The problem was that the similarity scores couldn’t tell a good match from a bad one.

A question about the user’s own money, like “what are my finances like”, is something the library can’t help with. Its best chunks scored between 41% and 50%. A question the library exists for, like how big an emergency fund should be, scored between 44% and 62%.

Those ranges overlap. There was no cut-off that kept the second kind and dropped the first.

Adding a reranker

In December 2025 we added a reranker. Where an embedding search compares a question and a chunk separately, a reranker reads them together and scores how well the chunk actually answers the question.

We run bge-reranker-v2-m3 through a hosted provider. The vector search still finds the ten closest chunks, the reranker scores them, and we keep the top three.

That separated the two kinds of question cleanly. The own-money questions all scored under 17%. The library questions scored between 28% and 88%. So in September we started dropping anything the reranker scored under 20%, and the irrelevant chunks stopped reaching Mama.

The reranker introduced a new dependency, though, and the first provider started turning requests away with server overload errors. We added the same model on a second provider as a backup. If both fail, the top three from the vector search go through unscored, and we get an alert.

The prompt cache problem

The first of this week’s changes was about cost rather than quality.

OpenAI caches the start of a prompt, so that when the next turn begins the same way, the cached part is cheaper and faster. The catch is that the cache only works up to the first point where the prompt differs.

Our knowledge base chunks were going into the instructions, at the very top of the prompt, with their scores attached: “Vector: 47.7%, Rerank: 56.9%”. A different chunk, or the same chunk with a slightly different score, meant the prompt differed from the first line, and nothing was served from the cache. Not the instructions, not the personality, not the conversation history.

We could see it in the logs. Cache hits were either 0% or about 94%, and never anything in between.

The fix was to move the chunks out of the instructions and into the input, just before the user’s message, so the instructions stay identical from turn to turn. From the second turn of a conversation, coach turns now read between 84% and 98% of the prompt from the cache.

Searching without the router’s help

The second change came with the new message classifier.

The old router extracted the concepts in a question, like “emergency fund” or “EPF Account 2”, and those concepts were what we searched the library with. The new router, which runs on OpenAI’s Decisions API, doesn’t produce them. So the search would have been running on the bare message, and a follow-up like “how much?” finds nothing on its own.

What we search with now is the end of Mama’s previous reply plus the user’s message, so a follow-up carries its context with it.

We tested this on 120 real coach turns, with a model judging whether each passage found was useful. The first run said it was every bit as good as the old concepts.

It wasn’t. The reranker provider was turning requests away during the run, so a lot of the searches fell back to the raw vector results. Those have no rerank score, which meant the 20% floor let all of them through, more passages reached the judge, and some of them were counted as relevant by luck.

We ran it again with a pause between turns so nothing fell back. The new query found a useful passage on 22 of the 120 turns. The old concepts managed 24, and the bare message 19.

A bit behind the old router, but better than the bare message, so it shipped. The lesson was as much about testing as about search. A fallback that quietly changes the scoring can make an experiment look better than it is.

Knowing when not to search

The third change was the simplest. The router is now also asked whether a passage from the library would help with this message at all, and if it says no, we skip the search.

On the same 120 turns, it skipped about one in six searches. The only useful passage it lost was on a question about how to use the app, which the library wouldn’t have answered well anyway.

Put together, a coach question now goes through the router, maybe gets a search, has its chunks reranked and filtered, and arrives in front of Mama in a form that doesn’t break the cache. Most of that is invisible to the person asking. They ask about their car loan, and Mama answers the way Mr Money TV would.