Cleaning and group rules

A scraped price is rarely comparable as-is. Across twelve shops the same shampoo arrives as 9,99 €, £8.50, 12.49 USD, 9.99 with the currency in a different column, and 2 x 250 ml versus 500ml versus 0,5 l. Every one of those is correct on its own page and useless next to the others.

Cleaning is the step that makes them one shape. It runs after scraping and before matching, and in Scrapewise it is configuration, not code.

The three things that actually block a comparison

One currency. Two prices in different currencies cannot be ranked. Conversion needs a rate, and a rate needs a date — a price converted at today’s rate is not the price that was charged last month. So the converted value and the rate it used are kept together.

One unit. 9,99 € for 500 ml and 6,49 € for 250 ml: the cheaper number is the more expensive product. The only honest comparison is price per unit, which means the pack size has to be read out of whatever the shop wrote — 2 x 250 ml, 0.5L, 500 g — and turned into a number.

One label per value. One shop says In stock, another Available, a third 1. Until they are the same word you cannot filter on it.

That is the whole job. The five kinds of rule Scrapewise gives you map onto it directly: convert a currency, map one label onto another, strip junk out of a text field, normalise a quantity, and pull a number out of a string.

Why the rules live on the group, not the scraper

The obvious place to put a cleaning rule is on the scraper that needs it. That works until you have twelve scrapers, and then you have twelve copies of the same “convert to EUR” rule, which drift.

So the rules live once on the group and apply to every scraper in it. A single scraper can still override one — shops do differ — and the override is explicit, so you can see which scraper is not following the group.

This matters most for the thing that varies: pack size. Every shop sells the same shampoo, but one lists it per bottle, one per 100 ml and one per 50 g. The rule that divides price by pack size is the same rule everywhere; only which unit changes per shop. That is what choice groups are for — one rule with several branches, and each scraper picks the branch that fits it. There is deliberately no default branch: if a shop has not been told which unit it sells in, the platform does not guess.

It will not quietly invent a value

Cleaning rules are allowed to add columns. They are not allowed to overwrite what you scraped, and they are not allowed to write a value they had to assume.

When a rule cannot run — the input column was empty, no branch was chosen, two rules wanted to set the same currency rate — the row gets a marker saying so instead of a number. An empty cell with a marker is a worse-looking result and a more useful one than a plausible figure nobody can trace. The markers are listed under Row markers, and the guarantees under Row safety.

When to set this up

Before matching, and before you look at any price. Matching uses price as a sanity check on a candidate, so a price in the wrong currency makes a correct match look suspicious. And a price-per-unit column that appears after you have already drawn conclusions from the raw prices tends to reverse them.

One ordering rule is worth knowing: a scraper’s own rules run first, then the group’s. So the group rule sees the already-cleaned value — see Order.

Set it up

Want this as a finished result?

If you would rather receive data that is already one currency, one unit and one label set, that is Product data cleaning  — we build and maintain the rules and deliver the cleaned result.

What’s next

• Every rule field → Rules for all scrapers in a group
• Match the cleaned data → How product matching works
• See it end to end → Competitor price comparison, end to end