Skip to Content
Getting StartedCompetitor price comparison, end to end

Competitor price comparison, end to end

Every other page here covers one step. This one is the spine: what order the steps go in, and what each one has to produce before the next can start.

The example is public. bebo.ee  is a live grocery price comparison for Estonia, built on Scrapewise, and the figures below are the ones it publishes on its own front page:

Estonian grocery e-shops tracked12
Products tracked130,000+
Products comparable across chains36,300
Price history kept1 year

That third row is the one worth sitting with. 36,300 of 130,000+ products are comparable across chains — the rest exist in one shop only, or under a private label nobody else sells. The gap is not a failure of the scrape. It is the honest answer to “how much of this can actually be compared?” Any price-comparison project has a number like that, and finding it early is better than discovering it after you have promised a dashboard.

Step 1 — Decide what “the same product” means

Before any scraping: pick the column that identifies a product. A barcode (EAN/GTIN) if the market publishes one; your own article number if you are comparing against your own catalogue.

In groceries, barcodes are the reason a project like bebo.ee is possible at all. Where shops publish them, matching is a code comparison and the answer is certain. Where they do not, it falls back to names and pictures — see How product matching works.

Step 2 — Get the URLs

One scraper per shop, with that shop’s product URLs on its link list. Not one scraper per product.

Finding the URLs is its own job. If the shop publishes a sitemap, harvest it — see Sitemaps. If you already have the URLs in a spreadsheet, upload them — see File scrapers.

Step 3 — Scrape the raw columns only

Scrape what the page actually says: price, sale price, currency, stock, the identity column, and the pack size as written. Nothing derived.

This is the step people over-build. A column you calculate at scrape time is frozen; a column you calculate as a rule can be changed and re-applied to data you already have. So scrape 2 x 250 ml and let a rule turn it into a number later.

→ REST quickstart to run your first scraper.

Step 4 — Clean it into one shape

One currency, price per unit, one label per value. Twelve shops means twelve ways of writing the same thing, and this is where they become one.

For a grocery comparison the per-unit price is not a nice extra — it is the comparison. A 500 ml bottle against a 250 ml bottle is not a price difference until both are per litre.

→ Cleaning and group rules.

Step 5 — Match

Put the shops in one group with your list as the main scraper, draw the lines, run it. Codes first, then text, then pictures.

Then read the three buckets rather than the headline number. Matched is what you can compare. No match is the shop-exclusive and private-label long tail — in the bebo.ee shape, most of the distance between 130,000+ and 36,300. Review is the set a person still has to decide.

→ How product matching works · Matched data.

Step 6 — Read it, and keep reading it

A price comparison is only worth anything repeated. One scrape tells you today’s prices; a year of them tells you who moves first, who follows, and who was never really on promotion.

  • Pull the comparable rows over the API — Matched data, paged.
  • Export for a spreadsheet or a BI tool — Scraped data.
  • Shops redesign. AI control tests a fix on live pages before saving it, so a layout change does not quietly become a month of missing prices.

What a project like this needs from you

Being blunt about it, because it is the part no API reference covers:

  • An identity column the market actually publishes. No barcode anywhere means text matching, which means a review queue.
  • A decision about the long tail. Most catalogues have a large shop-exclusive share. Decide up front whether those rows are out of scope or a separate report.
  • Per-unit rules per shop. Each shop lists in its own unit; that is one rule with branches, not one rule per shop.
  • A tolerance for review. The honest alternative to a review queue is a wrong price presented as a fact.

Want this as a finished result?

If you would rather receive the comparison than build it, that is Competitor price tracking  — your products next to every competitor’s price, delivered by CSV, Excel or API.

What’s next

Cleaning and group rulesOverview