Bringing OpenDOSM prices into Hargapedia

Hargapedia now shows prices from the Department of Statistics Malaysia's OpenDOSM portal, including wet markets and smaller shops we haven't been able to track ourselves.

Hargapedia has always relied on the prices we collect from retailers and online stores. That covers quite a few places, but there are plenty of others we don’t have prices for, particularly wet markets and independent grocers.

In January, the Department of Statistics Malaysia launched its new OpenDOSM portal, which makes a large collection of government data publicly available. One of those datasets contains prices of everyday items collected by government staff at shops and markets across Malaysia.

Each record tells you what the item is, where it was sold, how much it cost and when the price was collected. There are also separate lists identifying the items and the premises, including their names and addresses.

What interested us was where those prices came from.

Our own data already covers the retailers we track and a large number of online stores. OpenDOSM includes places we’ve never been able to collect from. A pasar price for garlic, for example, is something Hargapedia hasn’t been able to show before.

So rather than treat OpenDOSM as another price comparison service, we’re bringing its data into Hargapedia to fill some of those gaps.

Making their data work with ours

OpenDOSM doesn’t use the same product catalogue or retailer records as Hargapedia, so we couldn’t simply download the dataset and display everything.

Our team first went through OpenDOSM’s item list and matched the relevant items to products we already track. We also selected which premises to include.

With those mappings in place, a scheduled job downloads the latest OpenDOSM file every morning.

The job checks which records we haven’t already imported, filters them down to the items and premises we’ve selected, and converts each one into a Hargapedia price record. We keep the original collection date rather than treating it as a price collected on the day we imported it.

That lets the new prices appear alongside the ones we collect ourselves.

We also had to decide how to represent the premises.

If a price comes from a chain we already have in Hargapedia, such as 99 Speed Mart, we show that retailer’s logo. For wet markets and independent grocers, we use a generic shop icon instead.

Showing prices from nearby shops

The OpenDOSM prices appear in the existing price list on a product page, rather than in a separate section.

Each card shows the price, when it was collected and the premise it came from. Underneath is a Department of Statistics crest and a “Sourced from OpenDOSM” label, so you can tell it apart from the prices we collect ourselves.

If several branches of the same retail chain have reported prices, the card shows the first branch and indicates how many others are available.

You can tap the card to open the premise in Google Maps. When there are several branches, the app shows a list so you can choose which one to view, along with a reminder that the prices come from the Department of Statistics.

But we don’t want to show every OpenDOSM premise in Malaysia on every product page.

A price from a wet market in Serdang isn’t particularly useful if you’re shopping in Penang.

We already collect a postcode in Hargapedia, so we use that to work out an approximate location and only include premises within 10 km of it.

That keeps the OpenDOSM prices relevant to the area you’ve selected, rather than mixing prices from across the country into the same list.

The problem with older prices

There’s one big difference between OpenDOSM and our own price data.

OpenDOSM prices are collected daily, but the published dataset doesn’t necessarily receive new records every day. New dates tend to appear every week or so, and a month’s data may not be complete until that month has ended.

Our own retailer prices are updated daily, sometimes more than once a day.

That means an OpenDOSM price could be several days old by the time someone sees it in Hargapedia.

It’s why I’ve made the collection date part of the card, alongside the source label.

A price collected at a pasar on 8 February might still be useful as a guide, but you should know that it was recorded on 8 February rather than assume it’s today’s price.

The card follows the same general layout as the rest of Hargapedia’s price list, while keeping the source and date visible.