Groups

A group is the level-1 organisational unit: every scraper belongs to exactly one group, and the group’s dataTable field names the MongoDB collection where its scrapers’ rows land. Use groups to:

• Apply uniform scheduling to a set of scrapers (startType = MANUAL, DAILY, WEEKLY, NONE).
• Share a coherent dataset between customers (Shared groups).
• Match your list against the other scrapers of the group (Product matching) and merge data between them (Merge jobs).
• Share one list of post-process rules across all scrapers in the group (Rules for all scrapers in a group).
• Let AI check and repair the group’s scrapers on a schedule (AI control).

Endpoint summary

List groups — GET /api/scraper/group/list

curl -H "Authorization: Bearer $KEY" \
  https://portal.scrapewise.ai/api/scraper-api/api/scraper/group/list

Returns the groups you OWN. For groups shared TO you, use GET /api/scraper/shared/group/list.

Response (200) — List<GroupDTO> with id, name, dataTable name, startType, schedule metadata, member-scraper count. Empty list if you have no groups.

Errors — 400 (N/A) / 401 / 403 (N/A) / 404 (N/A) / 429 / 500.

Create or update a group — PUT /api/scraper/group

PUT /api/scraper/group
Authorization: Bearer <key>
Idempotency-Key: <uuid>
Content-Type: application/json
 
{
  "id": null,
  "name": "competitor-prices",
  "dataTable": "competitor_prices",
  "startType": "DAILY",
  "shouldMatchProducts": false
}

Upsert semantics: omit id to create; set id to update. Creating a new group is plan-feature gated by MAX_GROUPS; updating is not.

Response (200) — the persisted GroupDTO including the assigned id and any server-stamped fields.

Errors — 400 (validation) / 401 / 402 (MAX_GROUPS hit on create) / 403 (N/A) / 404 (N/A) / 429 / 500.

Set group-wide start type — PUT /api/scraper/group/{id}/start-type/{startType}

PUT /api/scraper/group/5f9a.../start-type/DAILY
Authorization: Bearer <key>
Idempotency-Key: <uuid>

Updates EVERY scraper in the group to share the same StartType. Values: MANUAL, DAILY, WEEKLY, NONE (paused).

Each non-NONE value is plan-feature gated:

• MANUAL → MANUAL_RUN
• DAILY → DAILY_SCHEDULER
• WEEKLY → WEEKLY_SCHEDULER

Response (200) — List<StartTypeForGroupDTO> — per-scraper records reflecting the new state.

Errors — 400 (group not found) / 401 / 402 (plan lacks the chosen feature) / 403 (N/A) / 404 (N/A — 400 used instead) / 429 / 500.

Product matching — PUT /api/scraper/group/{id}/match

Matching links rows of your list (the main scraper of the group) to rows of the sources you compare it with. It works the same for any kind of data: products, leads or locations. You say which column of your list is compared with which column of each source. Scrapewise saves that setup and runs the matching in the background.

Save or run a setup

PUT /api/scraper/group/5f9a.../match
Authorization: Bearer <key>
Idempotency-Key: <uuid>
Content-Type: application/json
 
{
  "masterData": {
    "jobId": "65b0...",
    "targetField": "sku",
    "matchers": [
      { "field": "name", "threshold": 80 },
      { "field": "brand", "threshold": 90, "kind": "exact" }
    ],
    "matchersByScope": {
      "65c1...": [
        { "field": "name", "scopeField": "title", "threshold": 75 }
      ]
    }
  },
  "scope": ["65c1...", "65c2...", "65c3..."],
  "draft": false
}

Who may save. The group’s owner saves the setup with a Full key or in the portal. On a group on the newer matching engine the owner saves drafts of the text settings (lines, levels); the runs start after new scrapes or are started by Scrapewise, and picture settings are kept by Scrapewise. See Start a matching run by hand.

A per-source map that is left out of the body keeps what is stored. An empty map {} clears it.

Response (200) — the saved setup (MatcherJobDTO) with its id and status. Matching runs in the background. Read the result in the portal or with GET …/match.

Errors

Read a setup — GET /api/scraper/group/{id}/match

Returns the saved setup of the group in the same shape as the PUT body. 404 when the group has no matching setup yet.

Read the setup as the canvas draws it — GET /api/scraper/group/{id}/match/setup

Read only (operation ID and MCP tool scrapewise_get_matcher_setup_view). Returns the matching setup of the group as it is now, for the same setup GET …/match returns.

flags says which matching switches are on for the group (number lines, pictures). It is null when the matching service has not reported for more than 5 minutes; the portal then makes no claim about them.

Run matching on a schedule — PATCH /api/scraper/group/{id}/match/schedule

Sets when the group’s matching runs again by itself, for example every Monday at 06:00. It works for classic groups and for groups on the newer matching engine. Also the MCP tool scrapewise_update_matcher_schedule. Full key only; a Read key is refused with 403.

PATCH /api/scraper/group/5f9a.../match/schedule
Authorization: Bearer <key>
Idempotency-Key: <uuid>
Content-Type: application/json
 
{ "type": "WEEKLY", "daysOfWeek": ["MONDAY"], "time": "06:00", "zone": "Europe/Tallinn" }

• time is HH:mm (24 h). zone is a time zone name; default Europe/Tallinn.
• Send the body null to clear the schedule: matching then runs only when you start it.
• A new or changed schedule starts from now. Missed earlier times are not run.
• The PATCH never starts a run and does not change the setup or its results.
• The answer is the setup with schedule, lastScheduledRun, nextRunAt (the next time it runs) and a new version.

Errors — 400 (a bad schedule field; the message names it) / 401 / 403 (a Read key, or a plan without matching) / 404 (the group has no matching setup yet) / 409 (the setup was replaced meanwhile — read it again and retry with a new key) / 429 / 500. Nothing was changed in each.

How the portal draws your setup

Every matching setup is drawn on the match canvas in the portal, whoever made it: the canvas itself, this API, or Scrapewise for a group whose matching it sets up. Enrichment saved with PUT …/merge (also the MCP tool scrapewise_run_scraper_data_group) is drawn too. You never need to paste or copy anything for the lines to appear.

An open canvas picks up a change made elsewhere by itself, within about a minute. If you have unsaved edits on the canvas, nothing is overwritten: the note “This setup was changed elsewhere.” appears with a Reload button. Saving while the note shows asks first: “This setup was changed elsewhere. Saving replaces it. Save anyway?”

• Why each box is there. Every box has one short label: Your list, Compared, Adds data · key …, Not used or In this group only.
• All / Matching / Enrichment. A switch above the canvas shows all lines, only the matching lines, or only the enrichment lines. Each choice shows how many lines it has, for example All (12). It only changes the view, never what is saved.
• Full screen. The canvas fills the screen when it opens, and boxes never overlap: the compared sources stack next to your list. Full screen shows the canvas on the whole page; press Esc or Exit full screen to leave.
• Order used in this group. A strip shows the steps 1 Codes → 2 Text → 3 Pictures, each with On or Off (or on for some sources). Step 1 decides first. The order is fixed; only the on/off switches differ per group or source. Each matching line shows the number of its step; price and stock checks belong to step 2. Click a step to see how it is set up. On older setups the strip shows one step, All lines together, and each line keeps its threshold label.
• Columns that only carry a value get no line. For example the column that groups rows into one product: its hover says “This column groups your file’s rows into one product. It is not compared.”
• Rules for the whole group. A panel above the canvas lists the settings that are not lines: levels, how many days of source data are used (Uses source data from the last N days), codes, text weights, pictures and number lines. A source with its own setting shows it on its box, for example Codes: sku or Needs a code to match.
• Pictures and number lines. They are drawn as full lines. When they are switched off for the group, the line keeps its style and gets a small off badge; hover it to see why.
• Skipped sources. A source that the latest matching run could not use says why on its box, for example No finished scrape yet (in the latest run). The label is about the latest run only and goes away after the next run that can use the source.
• Columns not in the latest scrape. When a line needs a column the box’s latest scrape does not have, the box shows it as an extra row so the line is still drawn: Built during matching (Scrapewise builds the column while matching) or Not in the latest scrape (the setup uses it, but the latest scrape of this source does not have it).
• Move the boxes. Drag a box to where you want it. The canvas saves the spot for this group, so the box is in the same place after a reload and on your other devices (spots belong to your account). A new source gets a free spot and never covers a box you placed. Reset layout (shown once you moved a box) puts every box back where the canvas places it. If a spot cannot be saved, it stays in this browser and the toolbar shows Layout not saved — retry.
• Only connected rows. This switch in the toolbar hides the rows of each box that have no line in the current view. A box with no line left shows No lines in this view. The switch is remembered in this browser and only changes the view.
• Show one type of line. The legend items are filters. Click one, for example Price and stock checks, to see only those lines. Click more items to add them; click an active item to remove it. Show all brings every line back. Own setting for this source marks a setting, not a type of line, so it is not a filter. The filter is for this visit only and never changes what is saved. With Only connected rows on, rows of hidden lines are hidden too.
• Lines you can add. In edit mode, a list under the canvas shows the sources that have no lines of their own yet but could get useful ones, with the lines each would get (for example category → title). A source is listed only when it has a finished scrape that has the column and the line would change its matching. Add lines adds them to the canvas; + key line sku → sku means the click also adds the source’s key line (★). More than 5 sources: Show all N. The list is hidden when no source qualifies.
• Unsaved lines and Undo. Lines added with Add lines are not saved yet. Each shows Unsaved · Undo on the line, and the bar above Save shows Unsaved changes with Undo (removes the lines you added last). Undo removes every line of that click. It is not offered when you drew more lines on the same source afterwards; remove those by hand. Nothing is stored until you press Save or Run.
• The key line (★) stays while a source has other lines. Deleting it then is refused with Mark another key line first. Mark another line as the key, or delete the other lines first. A source’s only line can always be deleted.
• Run and Stop. While a matching run of the group is waiting or running, the Run button shows Running and a Stop button sits next to it. Stop asks Stop this matching run?; Stop run stops it.

Matching in the portal: the tabs

The group’s Matching page has up to four tabs: History · Match the data · Results · Match quality.

• Match the data (the canvas) is always there.
• History and Results appear after the group’s first finished run. Match quality appears when that run was checked.
• While a run is going, the tabs of the last finished run stay. Until the first run has finished, stop a run with Stop next to Run on the canvas.
• Opening Matching shows Results when it is there, else Match the data. After Run you land on History when it is there.

History

One table, one row per run, newest first:

A column that is empty on every row of the page is left out. Tick two rows and press Compare 2 runs to compare them. Show deleted runs also lists deleted runs. The Danger zone below the table deletes matching data.

Results

The matched rows of the active run, in three pills with their counts: Matched, No match and Review. The Review count is the number of pairs waiting for review. When the side-by-side (wide) view cannot be built for the run, Results opens straight in the long view (one row per match). Rank by (Total, Name, Photo, Description) orders the pairs; Show system fields shows the matcher’s own columns.

Match quality

Cards with the numbers of the active run. A card is shown only when the run has a real number for it:

The line above the cards says when the run ran and which days the data was scraped. By source lists every compared source: Items in scope, Matched, Waiting for review and Match rate (more columns when the run has those numbers). Problems in this run, How right is each way of matching?, Right item, right variant? and Settings used for this run stay closed until you open them.

Held. On a group on the newer matching engine, a run whose result looks wrong in size (for example far fewer or far more rows than the results you have now) is held: it is not delivered. History shows Held · N vs M rows · previous results kept, and your current results stay as they are. When nothing was delivered before, it shows Held · N rows · no results yet. Over the API, match/history shows such a run with status FAILED and a guardHold block (code, rows, deliveredRows).

The Review count in Results is live: it goes down while people decide pairs. After you decide pairs, Match quality checks the run again by itself; the old numbers stay on screen until the new ones are ready.

Review

Review shows one pair per card: your item on the left, the source item on the right, each with its picture.

• Fields chooses which values show under both pictures. The list has four groups: Both sides (columns that exist on both sides, by the same name or paired by your matching lines, shown as title ↔ name, plus photo description, the text read from each photo), Left only, Right only, and Technical last (scores, status, ids). A Both sides field sits on the same line under both pictures, so the values are side by side; a missing value shows —. Nothing technical is chosen for you. The choice is saved per group.
• Pairs narrows the review to some source pairs, each shown as your list ↔ the source with the number of pairs to decide, for example Products ↔ Shop A · 1,234. Pick one or more; the list and its count follow. The filter runs on the server, so it works for any number of pairs. Show all clears it. Saved per group. The control is hidden when the review has only one source.
• Rank by and Pairs cannot be changed while you have answers that are not sent yet.

Delete a group (destructive — two-call protocol) — DELETE /api/scraper/group/{id}

Steps

1. POST /api/scraper/group/{id}/preview-delete[?withData=true] — mints a 5-minute token + preview summary.
2. DELETE /api/scraper/group/{id}[?withData=true] — commits the delete (idempotency-key required).

Skipping the preview step deletes rows without confirmation.

Step 1 — Preview

POST /api/scraper/group/5f9a.../preview-delete?withData=true
Authorization: Bearer <key>

Response (200)

{
  "token": "9b2d-...",
  "opName": "scrapewise_delete_scraper_group",
  "targetEntityId": "5f9a...",
  "previewSummary": {
    "entityName": "5f9a...",
    "entityType": "scraper_group",
    "cascadeCounts": {},
    "warnings": [
      "withData=true: the group's entire MongoDB data collection will be dropped (all historical scraped rows lost, irreversible)."
    ]
  }
}

When withData=true, the warning array surfaces the irreversible-data-loss notice — the user must see this before committing.

Step 2 — Commit

DELETE /api/scraper/group/5f9a...?withData=true
Authorization: Bearer <key>
Idempotency-Key: <uuid>

withData query param semantics:

• false (default) → drops the group + all its scrapers; preserves the MongoDB data collection.
• true → also drops the data collection (irreversible).

Response — 204 No Content.

Errors (both steps)

Rules for all scrapers in a group

A group can have one shared list of post-process rules. The list runs on every scraper in the group, so you write a rule once instead of copying it into each scraper. In the portal: group card → ”…” → Rules for all scrapers.

Group rules use the same rule kinds as scraper rules (CURRENCY_CONVERT, ENUM_MAP, REGEX_CLEAN, QUANTITY_NORMALIZE, NUMBER_EXTRACT — see Scrapers → per-kind reference). On top of that, a group rule can have conditions and can sit in a choice group.

Order: scraper rules first, then group rules

For each row:

1. The scraper’s own postProcessRules run first, top to bottom, as before.
2. Then the group rules run, top to bottom.

A group rule may read a column that a scraper rule wrote (for example price or pack_grams). A group rule may read the output of an earlier group rule, with the same chaining limits as scraper rules (Rule order and chaining). A scraper rule must not read a group rule’s output — it does not exist yet when scraper rules run.

Each row gets the group rules once, even on run paths that save rows in two steps.

Scraper override

If a scraper has its own rule for the same outputField, the scraper’s rule wins on all rows of that scraper. The group rule does not run for that scraper.

If a scraper overrides any rule inside a choice group, the whole choice group is skipped for that scraper. That scraper then owns that calculation.

The GET response shows, per scraper in scrapers, which outputs it overrides (overriddenOutputs) and which choice groups it owns (ownedSets).

Conditions (“only when …”)

A group rule can have up to 3 conditions. All must be true (AND). If they are not all true, the rule does not run on that row and its output is left out of the row.

• Comparisons ignore upper and lower case, and leading and trailing spaces.
• Numbers compare in plain form: 50.0 is "50", 0.1 is "0.1". true / false compare as text.
• A list or object value only counts for notBlank.
• value is 1–200 characters; each item of values too. No regex, no expressions, no OR, no NOT.
• field follows the column name rules: not blank, no . or $, and it does not start with _sw_ or ctx_.

{ "field": "product_type", "op": "equals", "value": "yarn" }
{ "field": "product_type", "op": "in", "values": ["yarn", "mixed"] }
{ "field": "name", "op": "contains", "value": "mat" }
{ "field": "pack_grams", "op": "notBlank" }
{ "field": "pack_grams", "op": "blank" }

Where the value comes from: the row’s column at the moment the group rule runs, if it is not empty. If the row has no value there, the scraper’s group value of the same name is used (see below). If neither exists, the value is missing.

Conditions exist only on group rules. A scraper rule with conditions, branchSet or branch is refused with 400.

Choice groups (branch sets)

Sometimes one calculation has several versions, and each row must use exactly one. For example: price per 50 g for yarn, per 100 g for fillings, per piece for mats.

Put those rules in one choice group: give each rule a branchSet (the group’s name, e.g. basis) and a branch (the choice, e.g. per_50g).

• branchSet and branch are both set or both empty. Each is 1–50 characters.
• Every rule in a choice group has at least 1 condition. Rules with the same branchSet and branch have identical conditions (else 400).
• A choice group has 2–10 choices. A rule list has at most 10 choice groups.

Choices may share an output column. Rules in different choices of the same choice group may write the same outputField, so every row gets its result in one column whatever choice it took. Example: per_50g, per_100g and per_piece all write unit_price, unit_price_eur and unit (a fixed value, for example 50 g, 100 g or piece).

• All writers of a shared column are in the same choice group, each in a different choice, with the same kind and outputType (and, for CURRENCY_CONVERT, the same to). Any other repeated outputField → 400, as before.
• A rule inside a choice reads a shared column only after a rule of its own choice wrote it. A rule outside the choice group reads it only after all its writers (400 otherwise).

{ "kind": "QUANTITY_NORMALIZE", "sourceField": "price", "outputField": "unit_price", "outputType": "NUMBER",
  "params": { "quantityField": "pack_grams", "targetQuantity": 50 },
  "branchSet": "basis", "branch": "per_50g",
  "conditions": [{ "field": "product_type", "op": "equals", "value": "yarn" }] }
{ "kind": "QUANTITY_NORMALIZE", "sourceField": "price", "outputField": "unit_price", "outputType": "NUMBER",
  "params": { "quantityField": "pack_pieces", "targetQuantity": 1 },
  "branchSet": "basis", "branch": "per_piece",
  "conditions": [{ "field": "product_type", "op": "equals", "value": "accessory" }] }

To test one rule of a choice group with POST /api/scraper/preview-rule, remove conditions, branchSet and branch first. The preview refuses them with 400. It never evaluates conditions.

For each row, the platform counts the choices whose conditions are true:

There is never a default choice. The platform does not guess. Write the conditions so that the choices exclude each other.

Group values per scraper (labels)

Some scrapers cannot carry a fixed column (for example AI scrapers). So the rule list can store group values: short labels per scraper, kept on the group, not on the scraper. Example: product_type = yarn on one scraper, accessory on another.

• scraperValues maps scraper id → name → value, for example "6a01...": { "product_type": "yarn" }.
• At most 5 names per group. Each value is 1–50 characters.
• Group values are only read by conditions. They are never written into the rows.
• A row column with the same name wins when the row has a value (see “Where the value comes from”). This lets an uploaded file carry the label per row.
• A fallback scraper without its own value uses the value of its main scraper.
• When a scraper leaves the group, its values are dropped. The GET does not return them, and the next PUT drops them with a warnings entry.

Row safety

Group rules never overwrite your data and never write a guessed value.

• An input column (sourceField, quantityField or fromField) is missing, null or blank → the rule is skipped on that row. Its output is left out. For CURRENCY_CONVERT, the shared <to>_rate / <to>_rate_date pair stays only if another conversion to that currency gave a value on the row (see Rate columns).
• The row already has a column with the rule’s output name (from the page or the file) → the rule is skipped and the existing column is kept as it is.
• A group ENUM_MAP with a missing source writes nothing (no default).
• A row that already has its own eur_rate / eur_rate_date (scraped, API or uploaded file) keeps it in both the scraper and the group phase. A scraper CURRENCY_CONVERT rule still writes its value, and a different computed rate adds RULE_UNUSABLE:eur_rate. A group CURRENCY_CONVERT rule to that currency writes nothing on that row and adds RULE_UNUSABLE:<output>, the same as when its output name is already a column.
• overwritesRawField cannot be used on group rules (400).
• Rows that failed to scrape (_sw_status other than OK) get no group rules.

Row markers

When a group rule cannot give a value where one is expected, the row gets a marker in its _sw_warnings column (see Scraped data). Most markers come from group rules; the two rate markers come from any CURRENCY_CONVERT rule (scraper or group).

A missing sourceField gives no marker. Example: a product with no sale_price simply gets no sale_price_… columns.

Uploaded files

Group rules also run on uploaded files (File scrapers): on a replace import and on the new rows of an append import. So an uploaded customer file can get the same shared columns (for example price_eur) as the scraped rows. File rows get the same markers.

• Rows that an append import fills in (rows matched to an existing row) get no rules. The import summary says how many rows that was.
• A file column with the same name as a group rule’s output is kept; the rule skips it with RULE_UNUSABLE.
• Numbers stored as text in a CSV (for example "16.95") are read as numbers.

When changes apply

A run reads the group rules once, when it starts. A change applies to runs and samples that start after the save. A run that is already going keeps the rules it started with. Samples use the group rules too, so a sample shows the same columns as a run.

Group rules are separate from the scraper’s own settings. Changing them does not restart a resumable run.

Recommended price-compare layout

A typical group that compares competitor prices needs only this rule list. Every scraper writes the same input columns (price, sale_price, currency, pack_grams or pack_pieces); the group adds the rest.

• Outside any choice: CURRENCY_CONVERT price → price_eur and sale_price → sale_price_eur, both with "fromField": "currency".
• One choice group basis, one choice per unit (per_50g, per_100g, per_piece). Each choice has 5 rules that write the same columns: QUANTITY_NORMALIZE price → unit_price and sale_price → sale_unit_price, CURRENCY_CONVERT unit_price → unit_price_eur and sale_unit_price → sale_unit_price_eur, and an ENUM_MAP fixed value price → unit (50 g, 100 g or piece).
• The fixed value is written only where its sourceField is filled. Read it from a column every row has (price), never from a label: labels are not in the row.
• Each scraper gets a label in scraperValues (for example product_type: yarn, accessory or mixed). Labels only steer the conditions; they are never written to rows. To show the kind in the data, add an ENUM_MAP fixed value inside each choice.
• Conditions that exclude each other: per_50g = product_type in [yarn, mixed] AND pack_grams notBlank; per_100g = product_type equals accessory AND pack_grams notBlank; per_piece = product_type in [accessory, mixed] AND pack_pieces notBlank AND pack_grams blank. A row with no pack size gets RULE_NO_BRANCH:basis and no unit price; that is expected, not an error.

The per_50g choice in full (the other two choices differ only in branch, conditions, quantityField, targetQuantity and the unit default):

[
  { "kind": "QUANTITY_NORMALIZE", "sourceField": "price", "outputField": "unit_price", "outputType": "NUMBER",
    "params": { "quantityField": "pack_grams", "targetQuantity": 50 },
    "branchSet": "basis", "branch": "per_50g",
    "conditions": [{ "field": "product_type", "op": "in", "values": ["yarn", "mixed"] },
                   { "field": "pack_grams", "op": "notBlank" }] },
  { "kind": "QUANTITY_NORMALIZE", "sourceField": "sale_price", "outputField": "sale_unit_price", "outputType": "NUMBER",
    "params": { "quantityField": "pack_grams", "targetQuantity": 50 },
    "branchSet": "basis", "branch": "per_50g",
    "conditions": [{ "field": "product_type", "op": "in", "values": ["yarn", "mixed"] },
                   { "field": "pack_grams", "op": "notBlank" }] },
  { "kind": "CURRENCY_CONVERT", "sourceField": "unit_price", "outputField": "unit_price_eur", "outputType": "NUMBER",
    "params": { "fromField": "currency", "to": "EUR" },
    "branchSet": "basis", "branch": "per_50g",
    "conditions": [{ "field": "product_type", "op": "in", "values": ["yarn", "mixed"] },
                   { "field": "pack_grams", "op": "notBlank" }] },
  { "kind": "CURRENCY_CONVERT", "sourceField": "sale_unit_price", "outputField": "sale_unit_price_eur", "outputType": "NUMBER",
    "params": { "fromField": "currency", "to": "EUR" },
    "branchSet": "basis", "branch": "per_50g",
    "conditions": [{ "field": "product_type", "op": "in", "values": ["yarn", "mixed"] },
                   { "field": "pack_grams", "op": "notBlank" }] },
  { "kind": "ENUM_MAP", "sourceField": "price", "outputField": "unit", "outputType": "TEXT",
    "params": { "mapping": {}, "default": "50 g" },
    "branchSet": "basis", "branch": "per_50g",
    "conditions": [{ "field": "product_type", "op": "in", "values": ["yarn", "mixed"] },
                   { "field": "pack_grams", "op": "notBlank" }] }
]

The platform writes eur_rate and eur_rate_date on each converted row; never name a column that. After the save, read warnings.

Get the group rules — GET /api/scraper/group/{id}/post-process-rules

curl -H "Authorization: Bearer $KEY" \
  "https://portal.scrapewise.ai/api/scraper-api/api/scraper/group/5f9a.../post-process-rules?page=0&size=200"

Query (all optional; without them the answer is the same as before):

Response (200)

In counts:

• overridden — scrapers whose own rules override at least one group output;
• missingDeclaredInput — scrapers with at least one entry in undeclaredInputs;
• missingValue — scrapers that have neither a column nor a group value for a name that a condition reads (the names are in each row’s missingLabels);
• withRowWarnings — scrapers with at least one entry in rowWarnings.

rowWarnings counts the rows of the scraper’s latest run that were stored with a choice-group marker: {code, choiceGroup, rows} where code is RULE_NO_BRANCH (no choice fits the row) or RULE_MULTI_BRANCH (more than one choice fits). Only choice groups of the saved rules count: rename or delete a choice group and its warnings go away. The latest run is the newest run that stored rows (with its continuation runs); a run that is still going counts its rows so far. A run that stored no rows does not hide the run before it. Deleted rows (a deleted run, removed duplicates, data older than your retention) are not counted. Scrapers show no row warnings until their next run after this feature.

undeclaredInputs lists inputs (sourceField, quantityField, fromField) of group rules that run for this scraper, which the scraper does not have. For uploaded-file scrapers it is always empty: their columns come from the file.

{
  "groupId": "6abbc37b049426c568144f5e",
  "rules": [],
  "scraperValues": { "6a01...": { "product_type": "yarn" } },
  "version": 3,
  "updated": "2026-09-30T10:00:00Z",
  "updatedBy": "someone@example.com",
  "counts": { "scrapers": 33, "overridden": 1, "missingDeclaredInput": 0, "missingValue": 0, "withRowWarnings": 1 },
  "scrapers": [
    {
      "scraperId": "6a01...",
      "name": "rito.dk",
      "type": "MULTIPLE_PRODUCTS",
      "values": { "product_type": "yarn" },
      "overriddenOutputs": ["price_eur"],
      "ownedSets": [],
      "undeclaredInputs": [],
      "missingLabels": [],
      "rowWarnings": [{ "code": "RULE_NO_BRANCH", "choiceGroup": "basis", "rows": 4 }]
    }
  ],
  "page": 0,
  "size": 200,
  "totalScrapers": 33,
  "matchingScrapers": 33,
  "runWarningsAvailable": true
}

Errors

Save the group rules — PUT /api/scraper/group/{id}/post-process-rules

PUT /api/scraper/group/5f9a.../post-process-rules
Authorization: Bearer <key>
Idempotency-Key: <uuid>
Content-Type: application/json
 
{
  "expectedVersion": 3,
  "scraperValues": {
    "6a01...": { "product_type": "yarn" },
    "6a02...": { "product_type": "accessory" }
  },
  "rules": [
    {
      "kind": "CURRENCY_CONVERT",
      "sourceField": "price",
      "outputField": "price_eur",
      "outputType": "NUMBER",
      "params": { "fromField": "currency", "to": "EUR" }
    },
    {
      "kind": "QUANTITY_NORMALIZE",
      "sourceField": "price",
      "outputField": "price_per_50g",
      "outputType": "NUMBER",
      "params": { "quantityField": "pack_grams", "targetQuantity": 50 },
      "branchSet": "basis",
      "branch": "per_50g",
      "conditions": [
        { "field": "product_type", "op": "equals", "value": "yarn" },
        { "field": "pack_grams", "op": "notBlank" }
      ]
    },
    {
      "kind": "QUANTITY_NORMALIZE",
      "sourceField": "price",
      "outputField": "price_per_piece",
      "outputType": "NUMBER",
      "params": { "quantityField": "pack_pieces", "targetQuantity": 1 },
      "branchSet": "basis",
      "branch": "per_piece",
      "conditions": [
        { "field": "product_type", "op": "in", "values": ["accessory", "mixed"] },
        { "field": "pack_pieces", "op": "notBlank" },
        { "field": "pack_grams", "op": "blank" }
      ]
    }
  ]
}

The PUT replaces the whole rule list and all group values. expectedVersion is the version from your last GET (0 when the group has no rules yet). To remove all group rules, send "rules": [] and "scraperValues": {} with the current expectedVersion.

Response (200) — the same shape as the GET (page 0, size 200), plus warnings (a list of plain sentences; only on PUT). A warning is not an error; the save succeeded. You get one when:

• a rule reads a column that no scraper in the group has;
• a condition field is both a column of some scraper and a group-value name (the row column wins there);
• a group value was dropped because that scraper is no longer in the group.

Limits (400 when broken): at most 50 rules; the whole document at most 64 KB; at most 3 conditions per rule; in with 1–50 values; value 1–200 characters; at most 10 choice groups with 2–10 choices each; at most 5 group-value names; group values 1–50 characters; every scraper id in scraperValues must be one of your scrapers; each CURRENCY_CONVERT has exactly one of from / fromField; overwritesRawField must be false or absent. The normal per-kind rule checks also apply.

Errors

MCP tools

The two routes are also MCP tools: scrapewise_get_scraper_group_post_process_rules and scrapewise_update_scraper_group_post_process_rules. They take the same fields as the REST routes.

GET /api/scraper/config/parameters lists the condition ops as postProcessConditionOps and the limits above as groupRuleLimits (maxRules, maxConditionsPerRule, maxInValues, maxConditionValueLength, maxBranchSets, maxBranchesPerSet, maxBranchNameLength, maxValueNames, maxScraperValueLength, maxDocumentBytes, maxPageSize). If these fields are missing, the API does not support group rules yet.

Deleting or moving a group

Deleting a group also deletes its group rules. Group rules stay with the group when the group is moved to another account.

AI control — keep a group’s scrapers working

Websites change, and a scraper that worked last week can return empty or wrong rows today. AI control watches the scrapers of one group for you. It is switched on per group.

What it does when it is on:

• On the schedule you choose (DAILY, TWICE_WEEKLY or WEEKLY), it checks the latest run of the scrapers in the group.
• It looks for failed runs, empty runs and values that went missing.
• When it finds a problem, it proposes a fix and tests the fix on live pages before it saves it.
• It undoes a fix when the next real run is worse.
• Each check looks at up to scrapersPerCheck scrapers (read the number from the status below), the ones checked longest ago first.

Which scrapers it can repair:

• It can repair scrapers that read the page with selectors (built automatically or by hand), and JSON-LD and API scrapers. We switch each kind of repair on.
• AI-schema scrapers are checked and you get an alert. They are not repaired yet.
• Supplier endpoints and uploaded files are not checked.

Good to know:

• We are in a test phase. Some problems are only recorded, not repaired, for now.
• After you switch it on, the first check usually starts within minutes (when the group is due for a check).
• Switching on needs your first wallet top-up. Switching off is always allowed.
• In the portal, open the group’s AI control dialog to do the same.

Switch AI control on or off — PUT /api/scraper/group/{id}/ai-control

PUT /api/scraper/group/{id}/ai-control
Authorization: Bearer <key>
Idempotency-Key: <uuid>
Content-Type: application/json
 
{
  "enabled": true,
  "cadence": "WEEKLY"
}

Changing the cadence, or switching off and on again, never brings the next check earlier than the chosen cadence allows.

Response (200) — the group (GroupDTO) with aiControlEnabled and aiControlCadence.

Errors

Read the AI control status — GET /api/scraper/group/{id}/ai-control/status

GET /api/scraper/group/{id}/ai-control/status
Authorization: Bearer <key>

Response (200)

{
  "enabled": true,
  "cadence": "WEEKLY",
  "phase": "SCHEDULED",
  "nextCheckAt": "2026-10-22T06:10:00Z",
  "lastCheckAt": "2026-10-15T06:12:41Z",
  "lastResultAt": "2026-10-15T06:12:41Z",
  "nothingNewSinceResult": false,
  "results": [
    { "kind": "REPAIRED", "count": 1, "reasons": ["VERIFIED"] },
    { "kind": "NOTHING_TO_REPAIR", "count": 11, "reasons": ["NO_ACTION"] }
  ],
  "checkedCount": 12,
  "scrapersPerCheck": 50
}

reasons are internal codes; read kind.

Phases

Result kinds

Why a scraper can be BLOCKED:

• Our own daily limit for AI checks was reached, or the AI service was busy. This is not your wallet.
• The scraper was busy (for example a run was going on).
• The website or the network failed.
• The test did not give a clear answer.
• That kind of repair is switched off.
• The scraper was changed at the same time.

You do not need to ask again and again: the next check time is in the answer.

Errors

MCP tools

Both routes are also MCP tools, in the setup tools:

• scrapewise_update_scraper_group_ai_control switches AI control on or off. It needs an LLM_FULL key.
• scrapewise_get_scraper_group_ai_control_status reads the status. An LLM_READ key can call it.

They take the same fields as the REST routes.

See also

• Scrapers — scrapers belong to a group
• Scraper jobs — load-history + data merge are group-scoped
• Shared groups — share a group’s read access cross-customer
• Scraped data — read group data
• Scrapers — per-kind rule reference — the rule kinds a group rule can use