---
title: "AI PromoPulse: the first GPT feature we shipped"
description: "We added GPT-3.5 to MailerTrack to write headlines and summaries of promotional data. It was our first feature using OpenAI's API, but getting useful analysis out of it wasn't as straightforward as generating the text."
url: "https://rayson.me/writing/mailertrack-ai-promopulse/"
category: "Engineering"
author: "Janarayson"
published: 2023-11-08
---
MailerTrack has a new report this month, and it's the first feature we've built using OpenAI's API.
It's called AI PromoPulse: Category Insights. You choose a product category, your own company as the supplier and a date range, and the report compares the promotions during that period with the same period a year earlier.
We've been doing this kind of year-on-year comparison for a while. Our [FMCG Promo Pulse](https://www.intrack.com.my/insights/) is an industry report covering the wider market, with the analysis written by our team.
AI PromoPulse takes a similar idea and makes it available for an individual category and supplier, with GPT-3.5 Turbo writing the headline and a short paragraph of commentary.
The numbers and charts come from our existing data. The new part is asking a language model to make sense of them.
## Giving GPT-3.5 something to work with
The report already has three sets of data: how the category's total number of promotions changed year on year, the top ten retailers by promotion count and the top ten suppliers.
We show these as a growth figure and two charts. For the AI commentary, we take the same three tables and send them to GPT-3.5 Turbo as text.
The system message gives the model a role as a marketing analyst, explains the tables and tells it what to produce. We want a headline and one short paragraph written for someone working at the supplier, covering what's happening in the category, how the supplier is doing and which retailers are involved.
We also ask it to use "previous year" and "current year" rather than mentioning the actual years, so the same prompt works across different reporting periods.
A second, shorter message provides two additional instructions. The figures are about promotions, not sales, and the supplier either appears in the top ten or it doesn't.
We use a temperature of zero to reduce variation between responses. The idea is to get reasonably consistent commentary when the underlying figures haven't changed.
The model returns a headline and summary in one response, which we split into two parts for the report.
We initially split the text using the word "Summary", but the model doesn't always follow the requested format. So there's a fallback that splits at the first blank line instead.
We also log the number of tokens used for each request against the user and company, so we can keep track of what generating these reports costs us.
Getting the text onto the page was fairly straightforward. Getting the model to write something worth reading took more work.
## Getting the commentary right
The prompt went through quite a few versions before we put the report into production at the end of October.
One of the earliest versions told the model to ignore rows labelled Others. It also ended with an instruction to "spin a positive vibe" if the analysis was negative.
We removed that one.
If a supplier had a bad year for promotions, there wasn't much point in asking the model to make it sound cheerful. The person reading the report would still be looking at the numbers beside it.
Another problem was how the model talked about the selected supplier.
Sometimes the supplier wasn't among the top ten in the data we'd provided, but GPT-3.5 would still mention it in the commentary as though it had enough information to assess its performance.
That's why we added the explicit instruction telling the model whether the supplier appears in the top ten. We needed it to work with the information it actually had, rather than fill in what was missing.
We also experimented with chain-of-thought prompting.
The idea comes from a Google paper last year showing that language models can perform better on some reasoning tasks when they're prompted to work through intermediate steps. There are a few ways to do it, from providing worked examples to simply asking the model to think step by step.
We tried getting GPT-3.5 to work through the three tables before writing its final paragraph.
The quality was still inconsistent.
Sometimes the model picked out something useful from the figures. Other times, the same data would produce a rather bland observation. GPT-3.5 wasn't particularly dependable when it came to analysing a set of numbers, even when those numbers were already organised into small tables.
That was the larger problem with the feature. We could get it to write the commentary, but couldn't rely on every report being insightful or even entirely accurate.
## The problem that wasn't GPT's fault
We also ran into a strange issue after putting the feature into production.
For some users, the panel containing the AI commentary didn't appear at all.
The HTML element was named `gpt-container`, which seemed reasonable when we built it. But GPT is also the name of Google's Publisher Tag, used for advertising.
Some ad blockers were matching the class name and hiding the entire panel as though it were an advert.
There was nothing wrong with the API response or the text the model generated. The browser simply wasn't showing it.
We renamed the CSS class, and that fixed the problem.
## Keeping it in beta
AI PromoPulse is available in beta, and that's where we're leaving it for now.
It works in the sense that you can choose a category and reporting period, get the figures and have GPT-3.5 write a summary of them.
But we're not comfortable charging clients for analysis that is only reliable some of the time. A convincing paragraph isn't particularly useful if the observations behind it aren't sound.
We've been looking at what else might be possible with LLMs and our data.
Pinecone was one of the technologies we considered, since vector databases were getting a lot of attention alongside language models. But our data is structured into tables rather than documents, and we didn't see much reason to turn it into embeddings.
A more interesting possibility is SQL.
GPT-3.5 can generate SQL, which raises the possibility of letting a model query the underlying data rather than giving it three tables we've prepared in advance.
That could make the analysis more flexible, but it would also introduce a different set of risks, particularly if we were letting a model decide what to query in a system used by paying clients.
For now, we're keeping the report as it is while we consider what to try next.
We eventually retired AI PromoPulse. It never made it out of beta.