Scrapers

The full scraper CRUD + run + watch lifecycle. A scraper is a reusable extraction recipe; runs of a scraper produce jobs; jobs produce rows of data (see Data).

Create or update — PUT /api/scraper

Upsert semantics: omit id in the body to create a new scraper, set id to update an existing one. The same endpoint covers both — there is no separate PUT /api/scraper/{id} (try it and you get 405). The body is the full ScraperDTO; partial updates are not supported — always fetch the current state via GET /api/scraper/{id}, mutate, then PUT the whole thing back.

The easiest way to discover a valid ScraperDTO for a given site is to call the Preview endpoint first — it returns one or more candidate scraperDTO payloads you can copy into your PUT body.

PUT /api/scraper
Authorization: Bearer <key>
Idempotency-Key: <uuid>
Content-Type: application/json
 
{
  "id": null,
  "name": "competitor-prices",
  "groupRef": "grp_xyz",
  "type": "MULTIPLE_PRODUCTS",
  "sourceConfig": {
    "url": "https://competitor.com/products",
    "pagination": { "type": "URL_PAGE_PARAM", "param": "page", "max": 50 }
  },
  "itemsConfig": [
    { "name": "title",    "selector": "h2.product-title",    "kind": "TEXT"   },
    { "name": "price",    "selector": ".price",              "kind": "TEXT"   },
    { "name": "in_stock", "selector": ".add-to-cart",        "kind": "EXISTS" }
  ],
  "schema": { "id": "sch_..." },
  "postProcessRules": []
}

Response — the persisted ScraperDTO with the assigned id. On a create it may also carry warning: you created more than 5 scrapers of the same endpoint in this group in the last 10 minutes. The create still succeeded — the warning points you to the list method below.

When the scraper’s group has group rules, the response may also carry ruleWarnings (a list of plain sentences; absent when there are none). This applies to PUT /api/scraper and to PUT /api/scraper/v2 (inside scraper). You get one when a group rule that runs for this scraper reads a column this scraper does not have, or when one of this scraper’s own rules reads a column that only a group rule writes (scraper rules run first, so that column is not there yet). The save still succeeded.

Many URLs? Do NOT create one scraper per URL. Create ONE scraper with the whole list — see Many URLs: one scraper, one list.

Errors — 400 (validation failed / cross-tenant id forge / schema_validation_failed) / 401 / 402 (plan MAX_SCRAPERS limit on create) / 403 (N/A) / 404 (N/A) / 429 / 500.

List — GET /api/scraper/list

curl -H "Authorization: Bearer $KEY" \
  "https://portal.scrapewise.ai/api/scraper-api/api/scraper/list?limit=50"

Paginated (see REST overview — pagination).

Filters:

Errors — 400 (N/A) / 401 / 403 (N/A) / 404 (N/A — empty list returned if none) / 429 / 500.

Get — GET /api/scraper/{id}

Single scraper by id. Returns the full ScraperDTO definition (sourceConfig, itemsConfig, postProcessRules, schema reference, etc.).

Errors — 400 (N/A — invalid id format surfaces as 404 envelope) / 401 / 403 (N/A) / 404 (scraper_not_found) / 429 / 500.

Updating an existing scraper: there is no separate PUT /api/scraper/{id} endpoint. Update is via PUT /api/scraper with id set in the body — same endpoint that handles create.

V2: enriched create/update + get — PUT /api/scraper/v2 + GET /api/scraper/v2/{id}

V2 endpoints have identical semantics to their V1 counterparts (PUT for upsert, GET for read by id) but return the richer ScraperSuperDTO shape instead of ScraperDTO: configValid flag, full inlined schema body, full inlined group body, and rolled-up runStats from the most-recent N runs.

Use V2 when you need the full picture in one round-trip instead of composing V1-get + a separate group fetch + a separate schema fetch. V2 is the preferred shape for new tooling — V1 is kept for portal back-compat.

PUT /api/scraper/v2
Authorization: Bearer <key>
Idempotency-Key: <uuid>
Content-Type: application/json
 
{ ... same ScraperDTO body as V1 ... }

curl -H "Authorization: Bearer $KEY" \
  https://portal.scrapewise.ai/api/scraper-api/api/scraper/v2/scr_abc123

Response (200) — ScraperSuperDTO (strict superset of ScraperDTO):

{
  "id": "scr_abc123",
  "name": "competitor-prices",
  "configValid": true,
  "schema": { "id": "sch_...", "content": { /* full schema document */ } },
  "group": { "id": "grp_...", "name": "competitors", "dataTable": "competitor_prices" },
  "runStats": { "lastNRuns": 5, "successes": 5, "failures": 0, "avgRows": 184 },
  "...": "all V1 fields preserved"
}

Errors (both PUT v2 and GET v2) — same as the V1 equivalents: 400 (validation / cross-tenant id forge) / 401 / 402 (MAX_SCRAPERS on create) / 403 (N/A) / 404 (scraper_not_found on GET) / 429 / 500.

Many URLs: one scraper, one list

Do NOT create one scraper per URL. When you have many addresses of one kind — 300 Amazon product pages, 50 Google News searches — create ONE scraper and give it the whole list as its link list. One scraper runs once over every address on its list. One scraper per address means hundreds of scrapers, each billed, scheduled and listed on its own.

There are three ways to do it. Pick the first one unless the scraper already exists.

For the no-code way, see scrape a list of URLs .

Storing a list switches the scraper to its list by itself — no follow-up PUT /api/scraper is needed. Then run it with GET /api/scraper/{id}/run (or schedule it).

Create, fill and run in one call — POST /api/scraper/url-list-run

The bulk route. Also the MCP tool scrapewise_run_scraper_url_list. Two body types on the same path:

Up to 5,000 addresses — JSON:

POST /api/scraper/url-list-run
Authorization: Bearer <key>
Content-Type: application/json
 
{
  "groupId": "65f1c2a9e4b0a1b2c3d4e5f6",
  "name": "News searches - daily",
  "urls": [
    "https://news.google.com/search?q=solar",
    "https://news.google.com/search?q=wind"
  ]
}

With JSON, every address must be usable: one bad address refuses the whole call (400 INVALID_URL, the address is named) and nothing is saved.

More than 5,000 addresses — JSON Lines (REST only, up to 100,000 lines):

curl -X POST "https://portal.scrapewise.ai/api/scraper-api/api/scraper/url-list-run" \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/x-ndjson" \
  --data-binary @list.jsonl

list.jsonl — the first line is the header, then one address per line:

{"groupId":"65f1c2a9e4b0a1b2c3d4e5f6","name":"Amazon DE - all products","clientRequestId":"list-2026-09-24"}
{"url":"https://www.amazon.de/dp/B0C1234567","title":"Product 1"}
{"url":"https://www.amazon.de/dp/B0C7654321","title":"Product 2"}

The header takes the same fields as the JSON body, plus a required clientRequestId (at most 128 characters). Each line is at most 4 KB. Here a line the endpoint cannot read is dropped and counted (linksDropped, also the X-Links-Dropped header; up to 10 are named); the call is refused only when no line is usable.

Response (200):

{
  "scraperId": "66a0c1d2e3f4a5b6c7d8e9f0",
  "groupId": "65f1c2a9e4b0a1b2c3d4e5f6",
  "tag": "gnews",
  "linksAccepted": 2,
  "linksDropped": 0,
  "linksQueued": 2,
  "jobQueued": true,
  "reused": false
}

linksQueued is how many addresses this run fetches; estimatedMicroCents is linksQueued times the endpoint’s price per address (null when unknown). resumedFrom appears on a resume, runSkippedReason: "NOTHING_TO_RESUME" when a resume had nothing left. Rows appear under scraperId while the run goes.

The run has no time limit. It stops only when the list is done, the wallet is empty, or you stop it. A run that makes no progress for 30 minutes while waiting on nothing is stopped with stoppedReason: "NO_PROGRESS" (see Jobs).

Billing: a resume (rerun: true) charges only the addresses that were never attempted. A force (force: true) charges every address again. Neither ever charges an address twice inside one run.

Retries are safe. The same name + the same addresses is the same scraper: a repeat of a finished call does not scrape again — it answers 409 PREVIOUS_RUN_ALREADY_CHARGED with rowsAlreadyStored and an options block saying what rerun and force would queue and cost. No Idempotency-Key header is needed.

Errors — 400 (INVALID_URL, LIST_TOO_LARGE, TAG_NOT_RESOLVED, TAG_NOT_SUPPORTED, HEADER_INVALID, NAME_INVALID) / 401 / 402 (WALLET_INSUFFICIENT_BALANCE) / 403 (read-only API key) / 404 (GROUP_NOT_FOUND, SCRAPER_NOT_FOUND) / 409 (PREVIOUS_RUN_ALREADY_CHARGED, PREVIOUS_RUN_TRUNCATED, NAME_TAKEN_WITH_DIFFERENT_LIST, AMBIGUOUS_SCRAPER_NAME, ALREADY_RUNNING, BULK_RUN_IN_PROGRESS, BULK_RUN_LEASE_HELD — wait Retry-After and send the same request again) / 413 (JSON body above 12 MB) / 429 (INGEST_CAPACITY* — too many uploads at once) / 500.

A list for a scraper you already have

Pasted or uploaded list — send the whole list; it replaces the stored one:

PUT /api/scraper/site
Authorization: Bearer <key>
Idempotency-Key: <uuid>
Content-Type: application/json
 
{
  "scraperId": "66a0c1d2e3f4a5b6c7d8e9f0",
  "links": [
    { "url": "https://news.google.com/search?q=solar", "title": "solar" },
    { "url": "https://news.google.com/search?q=wind", "title": "wind" }
  ]
}

Fed list — the addresses come from another scraper’s result column, and refill after each of its runs. You do not fetch those rows yourself, and there is no /urls call to make — the server copies them:

PUT /api/scraper/site
Authorization: Bearer <key>
Idempotency-Key: <uuid>
Content-Type: application/json
 
{
  "scraperId": "66a0c1d2e3f4a5b6c7d8e9f0",
  "linksSourceScraperId": "66a0c1d2e3f4a5b6c7d8e9aa",
  "urlFieldName": "product_url",
  "titleFieldName": "title",
  "updateLinksBySourceScraperAutomatically": true
}

Addresses this scraper cannot read are not stored; the response’s linksDropped counts them. Full details on the Sites page.

Delete a scraper (destructive — two-call protocol) — DELETE /api/scraper/{id}

Steps

1. POST /api/scraper/{id}/preview-delete[?withData=true] — mints a 5-minute DestructiveOpToken + preview summary listing what will be removed (scraper config + groups + optional data cascade).
2. DELETE /api/scraper/{id}[?withData=true] — commits the deletion (idempotency-key required).

Skipping the preview step deletes rows without confirmation.

Step 1 — Preview

POST /api/scraper/scr_abc123/preview-delete?withData=true
Authorization: Bearer <key>

Response: DestructivePreviewResponseDTO (token, opName, targetEntityId, previewSummary with cascade counts + warnings). Render to the user before committing.

Step 2 — Commit

DELETE /api/scraper/scr_abc123?withData=true
Authorization: Bearer <key>
Idempotency-Key: <uuid>

withData=false (default) → preserves the scraped rows; withData=true → also removes them from the group’s data collection (irreversible).

Response — 204 No Content.

Errors (both steps)

A scraper with a run in progress

• Not stopped: the delete is refused with 409 SCRAPER_RUN_ACTIVE and nothing is deleted. Stop the run first, then delete.
• Stopped, still ending: the delete ends the run first and then deletes. If the run has not ended within a few seconds, the answer is 409 RUN_STILL_ENDING with Retry-After: 5 and nothing is deleted. Wait 5 seconds and send the delete again.

Deleting only the data, not the scraper: use DELETE /api/scraper/data?scraperJobStatusId=..., which also follows the two-call protocol.

Auto-refresh values — keeping a cURL alive across the site’s deploys

Some sites rotate pieces of their URLs or headers on every deploy — Next.js sites change buildId in the _next/data/<id>/… path, Vercel rotates x-deployment-id, anti-bot middleware mints fresh tokens. A cURL captured from DevTools on Tuesday breaks on Wednesday’s deploy, and you’d otherwise have to re-paste a fresh cURL every time.

Two ways the platform solves this — both opt-in per scraper:

1. Zero-config Next.js auto-recovery (no setup needed)

When an API/CURL scraper returns 404 AND the URL matches /_next/data/<id>/, the engine automatically:

1. Fetches the host root (e.g. https://www.ermitazas.lt/)
2. Reads <script id="__NEXT_DATA__"> for the fresh buildId
3. Reads <meta name="next-deployment-id"> for the fresh deployment id (if present)
4. Rewrites the cURL with the fresh values and retries once

Works for ermitazas.lt (Vercel), karkkainen.com (Next.js on CloudFront), and any standard Next.js site without any per-scraper config. If discovery fails or returns the same tokens (genuine 404), the original error propagates unchanged.

2. Explicit sourceConfig.prelaunch (for custom rotating tokens)

For sites with non-standard rotating tokens (anti-bot middleware, JWT-shaped nonces, custom markers), declare extraction rules on sourceConfig.prelaunch. The cURL template uses {{NAME}} placeholders that get substituted with fresh values before each fetch.

{
  "sourceConfig": {
    "curl": "curl 'https://www.ermitazas.lt/_next/data/{{BUILD_ID}}/lt/search.json?q=home4you' -H 'x-deployment-id: {{DEPLOYMENT_ID}}'",
    "prelaunch": {
      "discoveryUrl": "https://www.ermitazas.lt/",
      "ttlMinutes": 360,
      "tokens": [
        { "name": "BUILD_ID",      "source": "NEXT_DATA", "expression": "/buildId" },
        { "name": "DEPLOYMENT_ID", "source": "META",      "expression": "next-deployment-id" }
      ]
    }
  }
}

Behaviour notes:

• Self-healing on rotation: if the substituted cURL returns 404/410/451, the engine invalidates the cached tokens, rediscovers, and retries once. If the fresh tokens are byte-identical to the stale ones (i.e. a genuine product 404), the retry is short-circuited.
• Cache + concurrency: resolved tokens are cached per (scraperId, configHash) per pod. A 500-URL batch hitting the same scraper triggers exactly one discovery fetch — concurrent callers coalesce via a per-scraper mutex.
• Cross-scraper throttle: discovery requests share a 3-permit semaphore so a cold-start storm (e.g. 50 scrapers waking up after a pod restart) doesn’t saturate the upstream proxy.
• Failure modes: if discovery fails (network, missing markers, regex no-match, value fails validatePattern), the scrape fails with a typed prelaunch_refresh_failed exception — the system never silently sends a literal {{BUILD_ID}} to the upstream.

Trigger a run — GET /api/scraper/{id}/run

Yes, GET for run is intentional — runs are idempotent at the Idempotency-Key level (see below). Triggers an async job.

curl -H "Authorization: Bearer $KEY" \
  "https://portal.scrapewise.ai/api/scraper-api/api/scraper/$SCRAPER_ID/run"

Response:

{
  "jobId": "job_xyz789",
  "scraperId": "scr_abc123",
  "status": "PENDING",
  "queuedAt": "2026-05-19T12:30:00Z"
}

The job runs async. To watch progress, use the SSE stream (next section).

Idempotency for runs

Pass an Idempotency-Key header to make retries safe:

curl -H "Authorization: Bearer $KEY" \
  -H "Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000" \
  ".../api/scraper/$SCRAPER_ID/run"

If you retry with the same idempotency key within 24h, Scrapewise returns the original job (no double-run).

Do not loop this call over a list of scrapers you just created, one per URL. Give ONE scraper the whole list instead — see Many URLs: one scraper, one list.

Errors — 400 (scraper_already_running / start_urls_unreachable) / 401 / 403 (N/A) / 404 (scraper_not_found) / 429 / 500.

Stream job status — GET /api/scraper-job-status/{id}/stream

Server-Sent Events stream of an individual job’s progress. The {id} is a scraper job status id (ScraperJobStatusDTO.id), not the scraper id — obtain it from GET /api/scraper/load-history filtered by scraperId.

# 1) Find the latest job for this scraper
JOB_ID=$(curl -s -H "Authorization: Bearer $KEY" \
  "https://portal.scrapewise.ai/api/scraper-api/api/scraper/load-history?scraperId=$SCRAPER_ID&size=1" \
  | jq -r '.content[0].id')
 
# 2) Stream its progress
curl -N -H "Authorization: Bearer $KEY" \
  "https://portal.scrapewise.ai/api/scraper-api/api/scraper-job-status/$JOB_ID/stream"

The -N flag disables curl’s buffering so events appear as they arrive.

You’ll see events like:

event: status
data: {"state":"RUNNING","itemsQuantity":0}

event: status
data: {"state":"RUNNING","itemsQuantity":42}

event: status
data: {"state":"RUNNING","itemsQuantity":140}

event: status
data: {"state":"SUCCESS","itemsQuantity":175,"duration":47213}

Field names follow ScraperJobStatusDTO — state, itemsQuantity, totalRequests, duration, errorMessage. The stream closes when the job reaches a terminal state (SUCCESS / FAILED / CANCELLED).

Node.js SSE example

import { fetchEventSource } from '@microsoft/fetch-event-source';
 
// jobStatusId obtained from /api/scraper/load-history beforehand
await fetchEventSource(
  `https://portal.scrapewise.ai/api/scraper-api/api/scraper-job-status/${jobStatusId}/stream`,
  {
    headers: { Authorization: `Bearer ${process.env.SCRAPEWISE_KEY}` },
    onmessage(ev) {
      const data = JSON.parse(ev.data);
      console.log(`${data.state} items=${data.itemsQuantity ?? '-'}`);
    },
  }
);

Python SSE example

import json, os, requests
 
# job_status_id obtained from /api/scraper/load-history beforehand
with requests.get(
    f'https://portal.scrapewise.ai/api/scraper-api/api/scraper-job-status/{job_status_id}/stream',
    headers={'Authorization': f"Bearer {os.environ['SCRAPEWISE_KEY']}"},
    stream=True,
) as r:
    for line in r.iter_lines():
        if line and line.startswith(b'data: '):
            event = json.loads(line[6:])
            print(event['state'], event.get('itemsQuantity'))

Errors — 400 (N/A) / 401 / 403 (N/A) / 404 (N/A — invalid {id} surfaces as a 401 envelope) / 429 (N/A) / 500. The SSE connection closes with the terminal event when the job ends; long-lived 5xx errors break the connection (reconnect from your client).

Stop a running scraper — GET /api/scraper/{id}/stop

Stops the scraper’s run that is waiting (PENDING) or going (RUNNING). A waiting run ends at once. A running run ends after the page it is reading. A scraper with no live run is left as it is. Partial scraped data already written to the group’s data collection is kept — clear it with DELETE /api/scraper/data?scraperJobStatusId=... if needed.

POST /api/scraper/{id}/stop does the same as GET. Both need the Idempotency-Key header.

curl -H "Authorization: Bearer $KEY" \
  -H "Idempotency-Key: $(uuidgen)" \
  "https://portal.scrapewise.ai/api/scraper-api/api/scraper/$SCRAPER_ID/stop"

Response — 200 OK (empty body). State transitions complete synchronously before the response is returned.

Errors — 400 (scraper_not_found) / 401 / 403 (N/A) / 404 (N/A — 400 envelope used) / 429 / 500.

To stop ALL running scrapers in a group at once, see Stop every running scraper in a group.

Sample data — GET /api/scraper/{id}/get-sample-data

Runs the scraper synchronously against its source URLs (capped to a FIRST batch / FIRST page — NOT a full run, no pagination expansion) and returns the parsed rows immediately. Use to validate a scraper config visually before scheduling a full async run.

curl -H "Authorization: Bearer $KEY" \
  "https://portal.scrapewise.ai/api/scraper-api/api/scraper/$SCRAPER_ID/get-sample-data?useCache=true"

Response (200) — List<Map<String, Any>> keyed by the scraper’s configured field names. Values are post-processed if rules are set.

Errors — 400 (scraper_not_found) / 401 / 403 (N/A) / 404 (N/A — 400 used instead) / 429 / 500. Also 502 if the source site is unreachable.

Preview a post-process rule — POST /api/scraper/preview-rule

Apply a single PostProcessRule to a user-provided sample value and return the derived output without persisting anything or running a real scrape. Use to validate rule parameters before saving the rule onto a scraper config.

Request body shape

Response (200): { "output": <derived value> } (plus rate + rateDate for CURRENCY_CONVERT). When the rule is valid but cannot produce a value for this sample, the response has no output and carries skipped with the reason instead (only QUANTITY_NORMALIZE and NUMBER_EXTRACT do this).

The preview takes one scraper rule. A rule with conditions, branchSet or branch is refused with 400. To preview a group rule, remove those three fields first; conditions are never evaluated here.

Per-kind reference

Each rule kind has its own params shape and its own null-source behaviour. The five kinds differ deliberately — see the per-kind note below.

CURRENCY_CONVERT — convert one currency to another using ECB rates

Set exactly one of from and fromField. Both or neither → 400 on save.

Currency from a column (fromField): the rule reads the currency from the row. The value is trimmed and upper-cased, so dkk works. It must be a 3-letter code that has an ECB rate.

• The column is missing, null or blank → the rule is skipped for that row. No output is written.
• The value is not a known currency (for example XYZ, € or kr.) → the rule is skipped for that row. No output is written.
• overwritesRawField cannot be used together with fromField (400).
• On a scraper rule, fromField must be one of the scraper’s own columns (400 otherwise).

{
  "kind": "CURRENCY_CONVERT",
  "sourceField": "price",
  "outputField": "price_eur",
  "outputType": "NUMBER",
  "params": { "fromField": "currency", "to": "EUR" }
}

A scraper-level rule with fromField needs a current API version. An older version does not know fromField: the rule fails and writes nothing (it never writes a wrong value). In a group rule set, a skipped fromField conversion also puts a marker on the row (see Row markers).

Null source: no-op — the output field is not written.

Precision: the converted value is not rounded. It is amount ÷ rateFrom × rateTo using ECB rates, stored at full precision. Converting a currency to itself returns the amount unchanged with rate 1.0.

Rate columns: one shared pair per target currency

A CURRENCY_CONVERT rule writes its converted value to outputField. The rate is stored once per row and target currency, in two fixed columns:

<to> is the rule’s to code in lower case (EUR, eur and Eur all give eur_rate). All conversions to the same currency on one row share this pair: three rules that write price_eur, sale_price_eur and unit_price_eur give one eur_rate and one eur_rate_date, in both scraper rules and group rules. The rule writes no other side columns.

• One rate table per run. All conversions of one run use the same ECB rate table, so a new ECB publication or midnight during a run never mixes two dates.
• Cleaning in place. A rule with overwritesRawField: true and the same fixed from and to (for example DKK → DKK) only cleans the number. It writes no rate pair.
• Same currency through fromField. A row whose fromField column already holds the target currency gets <to>_rate = 1.0 and the date of the run’s rate table.
• Two different rates on one row. When two conversions to the same currency on one row need different rates (for example one reads DKK from fromField and another has a fixed from: "SEK"), both converted values are written, the pair is left out, and the row gets RULE_RATE_CONFLICT:<to>_rate in its _sw_warnings column. The platform never shows a rate that is wrong for one of the values.
• The row already has a rate column. If the row arrives with its own <to>_rate or <to>_rate_date column (for example eur_rate, scraped from the page, sent by an API source or in an uploaded file), that column is kept as it is and never removed. The converted value is still written. When the rate the platform computed differs from it, the row gets RULE_UNUSABLE:<to>_rate (for example RULE_UNUSABLE:eur_rate). Group rules are stricter: a group conversion to that currency writes nothing on such a row (see Row safety).
• No converted value, no pair. If no conversion to a currency gave a value on a row (all skipped), that row has no pair for that currency.
• A rule’s outputField cannot be a pair name (eur_rate, eur_rate_date, …), and on a scraper rule the pair name cannot be one of the scraper’s own columns (400).

Rows saved before this change keep their old per-rule columns ({outputField}_rate and {outputField}_rateDate); they are not renamed. Internal columns that start with __cc_ are no longer written and are never returned (grid, REST, MCP, exports, samples).

POST /api/scraper/preview-rule
Authorization: Bearer <key>
Content-Type: application/json
 
{
  "rule": {
    "kind": "CURRENCY_CONVERT",
    "sourceField": "priceSek",
    "outputField": "priceEur",
    "outputType": "NUMBER",
    "params": { "from": "SEK", "to": "EUR" }
  },
  "sampleValue": 2490
}

Response: { "output": 215.85, "rate": 0.0867, "rateDate": "2026-04-17" }. rate and rateDate are the values the rule would write to the shared pair (eur_rate, eur_rate_date). A cleaning-in-place rule (same from and to, overwritesRawField: true) returns no rate.

ENUM_MAP — look up the source value in a mapping table

Null source — the null-sentinel: a null source value looks up the empty-string "" key. The same applies to an empty-string source ("") — null and "" are intentionally treated as the same “no meaningful value” for ENUM_MAP lookup purposes. This is the canonical pattern for deriving a True/False column based on whether a scraped field is present or absent on the page.

Null mapped value: if a key maps to null (e.g. {"": null}), that’s treated as “no real mapping” and falls through to default. Closes the silent-strip footgun where a null output value would be removed from the row by the downstream null-filter.

Fixed value: leave mapping empty ({}) and set default to write the same text on every row. An empty mapping without default → 400. Example: write unit = 50 g:

{
  "kind": "ENUM_MAP",
  "sourceField": "price",
  "outputField": "unit",
  "outputType": "TEXT",
  "params": { "mapping": {}, "default": "50 g" }
}

As a scraper rule, a fixed value is written on every row, also when sourceField is null or missing. As a group rule, it is written only on rows where sourceField has a value — so unit appears exactly on the rows that have a price.

POST /api/scraper/preview-rule
Authorization: Bearer <key>
Content-Type: application/json
 
{
  "rule": {
    "kind": "ENUM_MAP",
    "sourceField": "promotionalBanner",
    "outputField": "isPromoted",
    "outputType": "TEXT",
    "params": {
      "mapping": {
        "":     "False",
        "Sale": "True",
        "New":  "True"
      },
      "default": "Unknown"
    }
  },
  "sampleValue": null
}

Response: { "output": "False" } — the empty-string mapping key matched the null source.

Worked behaviour table for the rule above:

The same null-sentinel works in the portal’s rule builder — leave the “When scraped value is” cell blank and type the desired output in “Save as”. The empty-key row shows a (matches missing / null) placeholder so it’s visually distinguishable from a half-finished add.

REGEX_CLEAN — keep only characters matching a safe character class

Null source: no-op — the output field is not written.

Safety: the keep pattern is restricted to simple character classes — groups (), quantifiers *+?, alternation |, and curly braces {} are all rejected to prevent catastrophic backtracking. Unsafe pattern → 200 { "error": "Unsafe REGEX_CLEAN 'keep' pattern" }.

POST /api/scraper/preview-rule
Authorization: Bearer <key>
Content-Type: application/json
 
{
  "rule": {
    "kind": "REGEX_CLEAN",
    "sourceField": "price",
    "outputField": "priceClean",
    "outputType": "TEXT",
    "params": { "keep": "[0-9.]" }
  },
  "sampleValue": "$1,234.56"
}

Response: { "output": "1234.56" }.

QUANTITY_NORMALIZE — price per unit

Shown as Price per unit in the portal. Rescales a per-pack value to a fixed reference quantity, so products sold in different pack sizes can be compared:

output = sourceField ÷ quantity × targetQuantity

The quantity comes from a column (quantityField) or is a fixed number (quantityValue). Set exactly one of the two — both or neither returns 400.

The rule is unit-agnostic: grams, millilitres, metres or pieces all work, because it only divides and multiplies numbers. It never reads a unit symbol, so make sure the quantity and targetQuantity use the same unit.

With quantityValue the same rule does simple fixed arithmetic:

Rounding: the result is rounded half-up to 4 decimals and stored as a number.

Skips — the output column is left out of the row (never written as 0 or null) when:

• the source or the quantity is missing;
• the source or the quantity is not an unambiguous number — plain numbers and strings like "100", "12.5" or "12,5" are read, but a string such as "1.000" is refused because it could mean 1 or 1000;
• the quantity is zero or negative;
• the result is out of range, or rounds to 0 at 4 decimals while the source is not 0.

A skip never removes a scraped field. If the row already carried a value under outputField, that value is removed, so a stale per-unit price is never left behind.

Not allowed: overwritesRawField: true — the output is always a new derived column. Saving such a rule returns 400.

POST /api/scraper/preview-rule
Authorization: Bearer <key>
Content-Type: application/json
 
{
  "rule": {
    "kind": "QUANTITY_NORMALIZE",
    "sourceField": "price",
    "outputField": "price_per_50g",
    "outputType": "NUMBER",
    "params": { "quantityField": "pack_grams", "targetQuantity": 50 }
  },
  "sampleValue": 7.5,
  "sampleQuantity": 100
}

Response: { "output": 3.75 }.

Without sampleQuantity (or with "sampleQuantity": "1.000") the rule cannot produce a value, and the response carries the reason instead: { "skipped": "quantity field 'pack_grams' is absent" }. With quantityValue, sampleQuantity is ignored:

{
  "rule": {
    "kind": "QUANTITY_NORMALIZE",
    "sourceField": "price_minor",
    "outputField": "price",
    "outputType": "NUMBER",
    "params": { "quantityValue": 100, "targetQuantity": 1 }
  },
  "sampleValue": 2500
}

Response: { "output": 25.0 }.

NUMBER_EXTRACT — take a number from text

Shown as Take a number from text in the portal. Reads the number written right before a unit word in a scraped text column (for example a product name), multiplies it by that unit’s factor, and writes a number. Use it when the pack size is only in the text, e.g. "Hygge Wool 100m/50g" → 50.

• token — the unit word as written in the text. 1–20 characters, at least one of them not a digit. Tokens must be unique, ignoring upper/lower case. Matching ignores case ("300 G" matches g).
• factor — optional number the value is multiplied by, default 1. Greater than 0 and at most 1,000,000. Use it to bring different units to one base unit (for example kg with factor 1000 gives grams).
• 1–20 units per rule.
• sourceField must be a scraped column, outputType must be NUMBER, and overwritesRawField: true is not allowed (400).

Which number is read: the number directly before the unit, with up to 3 spaces and an optional hyphen between them ("50g", "50 g", "9-Pack"). The number may have . or , as the decimal separator. The unit must end the word, so g does not match inside "grams" or "50g85m" — list grams as its own token if you need it. When several units are found, the unit listed first in units wins (if the same unit is found twice, the first one in the text wins).

Rounding: the result is rounded half-up to 4 decimals.

It never guesses — the output column is left out of the row when:

As with QUANTITY_NORMALIZE, a result that rounds to 0 is skipped, and a skip removes any stale value under outputField.

Limits:

• The number must come before the unit — "Pack of 9" or "$5.99" are not read.
• Only the first 2,000 characters of the text are read.
• A short token can match an unrelated word: with g as a token, "5G" reads as 5 g; with st, "1st" reads as 1.

POST /api/scraper/preview-rule
Authorization: Bearer <key>
Content-Type: application/json
 
{
  "rule": {
    "kind": "NUMBER_EXTRACT",
    "sourceField": "name",
    "outputField": "pack_g_name",
    "outputType": "NUMBER",
    "params": { "units": "WEIGHT_G" }
  },
  "sampleValue": "Hygge Wool 100m/50g"
}

Response: { "output": 50.0 }. For "sampleValue": "Hygge Wool 1 500 g" the response is { "skipped": "'name': grouped number" } instead. Other reasons read, for example, "'name': multipack", "'name': range", "'name': ambiguous number '1.000'", "'name': zero", "no number followed by g/gr/… in 'name'", "source 'name' is not text" or "result rounds to 0 at 4 decimals".

Unit presets

A preset name can be used instead of a list: "units": "WEIGHT_G". When the scraper is saved, the name is replaced by the explicit list below, so a saved rule does not change if a preset changes later. An unknown preset name returns 400. The current presets are also returned by GET /api/scraper/config/parameters as postProcessUnitPresets (each factor is a JSON number: a whole number when it has no decimals, e.g. 1000, otherwise a decimal, e.g. 28.349523125). Tokens are listed in order.

Cross-rule null-source posture

ENUM_MAP is the only LOOKUP rule of the five (the others are transforms), which is why it’s the only one with null-sentinel semantics. If you need True/False from null/non-null, ENUM_MAP is the rule.

Rule order and chaining

Rules in postProcessRules run top to bottom, once per row. A rule can use the result of an earlier rule in the list, but only the result of a QUANTITY_NORMALIZE or NUMBER_EXTRACT rule. The outputs of CURRENCY_CONVERT, ENUM_MAP and REGEX_CLEAN rules cannot be read by another rule.

Saving the scraper returns 400 (the message names the rule number) when a rule:

• reads the output of a later rule, or its own output;
• reads the output of a CURRENCY_CONVERT, ENUM_MAP or REGEX_CLEAN rule, including the shared <to>_rate / <to>_rate_date columns (and the old _rate / _rateDate columns) of a currency conversion;
• is a NUMBER_EXTRACT, ENUM_MAP or REGEX_CLEAN rule that reads another rule’s output;
• has an outputField equal to a column that an earlier rule reads (its sourceField or quantityField).

When a rule is skipped, the rules that read its output are skipped too: no value flows down the chain, and no column is written as 0 or null.

API-type scrapers: saving an API-type scraper does not check its rules; a rule with bad settings then fails when the scraper runs and writes no column, so test each rule with POST /api/scraper/preview-rule first. Preset names are still expanded, and unknown preset names still rejected with 400, when any scraper is saved.

Example 1 — price per 50 g in EUR. The scraper extracts price, sale_price (DKK) and pack_grams:

"postProcessRules": [
  {
    "kind": "QUANTITY_NORMALIZE",
    "sourceField": "price",
    "outputField": "price_per_50g",
    "outputType": "NUMBER",
    "params": { "quantityField": "pack_grams", "targetQuantity": 50 }
  },
  {
    "kind": "QUANTITY_NORMALIZE",
    "sourceField": "sale_price",
    "outputField": "sale_price_per_50g",
    "outputType": "NUMBER",
    "params": { "quantityField": "pack_grams", "targetQuantity": 50 }
  },
  {
    "kind": "CURRENCY_CONVERT",
    "sourceField": "price_per_50g",
    "outputField": "price_per_50g_eur",
    "outputType": "NUMBER",
    "params": { "from": "DKK", "to": "EUR" }
  },
  {
    "kind": "CURRENCY_CONVERT",
    "sourceField": "sale_price_per_50g",
    "outputField": "sale_price_per_50g_eur",
    "outputType": "NUMBER",
    "params": { "from": "DKK", "to": "EUR" }
  }
]

For a row with price 7.5, pack_grams 100 and no sale_price, the result is price_per_50g = 3.75, price_per_50g_eur = 3.75 DKK converted to EUR (plus one eur_rate and one eur_rate_date). The row has no sale_price_per_50g and no sale_price_per_50g_eur column — nothing is written as 0 or null. If the row also had a sale_price, its sale_price_per_50g_eur would share the same eur_rate and eur_rate_date.

Example 2 — pack size from the product name. The scraper extracts name and price, but no pack-size column:

"postProcessRules": [
  {
    "kind": "NUMBER_EXTRACT",
    "sourceField": "name",
    "outputField": "pack_g_name",
    "outputType": "NUMBER",
    "params": { "units": "WEIGHT_G" }
  },
  {
    "kind": "QUANTITY_NORMALIZE",
    "sourceField": "price",
    "outputField": "price_per_50g_name",
    "outputType": "NUMBER",
    "params": { "quantityField": "pack_g_name", "targetQuantity": 50 }
  }
]

For name "Hygge Wool 100 g" and price 7.5, the row gets pack_g_name = 100 and price_per_50g_name = 3.75. If the name has no weight, neither column is written.

Precision: QUANTITY_NORMALIZE and NUMBER_EXTRACT outputs are rounded half-up to 4 decimals; CURRENCY_CONVERT outputs are not rounded (see CURRENCY_CONVERT).

POST /api/scraper/preview-rule tests one rule at a time. To preview a chained rule, pass the earlier rule’s output as sampleValue (or as sampleQuantity when the rule reads it as quantityField).

Errors

On invalid rule params, the endpoint returns 200 with an error string instead of throwing — intentional non-throw shape so agents can iterate on rule params without exception handling.

400 (request body validation) / 401 / 403 (N/A) / 404 (N/A) / 429 / 500. Rule-execution failures surface as 200 { "error": "..." } in the body, NOT as an HTTP error.

SEO fields

Get SEO fields for a scraper — GET /api/scraper/{id}/seo-fields

Returns SEO metadata (title, description, og:* tags) extracted from the scraper’s configured source URL. Cached at the SEO field service layer.

curl -H "Authorization: Bearer $KEY" \
  https://portal.scrapewise.ai/api/scraper-api/api/scraper/$SCRAPER_ID/seo-fields

Response (200) — List<SeoField> (name + content per discovered meta tag). Empty list if no extractable meta tags.

Errors — 400 (scraper_not_found) / 401 / 403 (N/A) / 404 (N/A — 400 used) / 429 / 500.

Get SEO fields for an arbitrary URL — POST /api/scraper/seo-fields

Same extraction logic but takes a URL in the body — no pre-existing scraper required. Subject to the SSRF deny-list.

POST /api/scraper/seo-fields
Authorization: Bearer <key>
Content-Type: application/json
 
{ "url": "https://example.com" }

Response (200) — List<SeoField>.

Errors — 400 (SSRF_BLOCKED for internal / link-local / cloud-metadata hosts) / 401 / 403 (N/A) / 404 (N/A) / 429 / 500. Also 502 if upstream unreachable.

Group-level run + stop

Run every scraper in a group — GET /api/scraper/group/{id}/run

Triggers a manual run of EVERY scraper that belongs to the group. Each run is queued asynchronously. Requires the RUN_WITH_GROUP plan feature.

curl -H "Authorization: Bearer $KEY" \
  -H "Idempotency-Key: $(uuidgen)" \
  "https://portal.scrapewise.ai/api/scraper-api/api/scraper/group/grp_xyz/run?mergeData=true"

Response — 204 No Content. Poll GET /api/scraper/load-history per scraper to check status.

Errors — 400 (group not found) / 401 / 402 (plan lacks RUN_WITH_GROUP) / 403 (N/A) / 404 (N/A — 400 used) / 429 / 500.

Stop every running scraper in a group — GET /api/scraper/group/{id}/stop

Stops every scraper in the group that’s currently RUNNING. Scrapers not in RUNNING state are silently skipped. Partial scraped data already written is kept — clear it via DELETE /api/scraper/data per scraperJobStatusId if needed.

curl -H "Authorization: Bearer $KEY" \
  -H "Idempotency-Key: $(uuidgen)" \
  "https://portal.scrapewise.ai/api/scraper-api/api/scraper/group/grp_xyz/stop"

Response — 200 OK (empty body). State transitions complete synchronously.

Errors — 400 (group not found) / 401 / 403 (N/A) / 404 (N/A — 400 used) / 429 / 500.

Fallback scraper — PUT /api/scraper/fallback

Attach a fallback scraper to a SINGLE_PRODUCT primary scraper. The fallback runs automatically when the primary extracts no data for a link (e.g. a JSON-LD parser as a backup to CSS-selector extraction). Requires the FALLBACK_SCRAPER plan feature.

PUT /api/scraper/fallback
Authorization: Bearer <key>
Idempotency-Key: <uuid>
Content-Type: application/json
 
{
  "scraperId": "scr_abc123",
  "fallback": { /* full ScraperDTO of the fallback scraper */ }
}

Response (200) — the persisted fallback ScraperDTO with its assigned id, or null on silent failure (rare; usually surfaces as a 4xx instead).

Errors — 400 (primary scraper isn’t SINGLE_PRODUCT type / scraperId doesn’t exist) / 401 / 402 (plan lacks FALLBACK_SCRAPER) / 403 (N/A) / 404 (N/A — 400 used) / 429 / 500.

Debug: load raw site HTML — GET /api/scraper/load-site + /load-site-content

Two thin debug fetchers that go through the proxy infrastructure + SSRF deny-list + customer rate limit. Use to inspect a page’s structure when designing a scraper config or debugging selectors.

# Raw text/html
curl -H "Authorization: Bearer $KEY" \
  "https://portal.scrapewise.ai/api/scraper-api/api/scraper/load-site?url=https://example.com&useCache=true"
 
# JSON envelope { "content": "<html>" } — for MCP-style tooling
curl -H "Authorization: Bearer $KEY" \
  "https://portal.scrapewise.ai/api/scraper-api/api/scraper/load-site-content?url=https://example.com"

Errors (both) — 400 (SSRF_BLOCKED) / 401 / 403 (N/A) / 404 (N/A) / 429 (rate limit) / 500. Also 502 if upstream unreachable.

Config parameter catalogue — GET /api/scraper/config/parameters

Returns the canonical enum lists used by scraper config builders: scraper types, pagination types, parameter types, post-process kinds, and the common-parameters catalogue. Used by the scraper-builder UI; usually not interesting to API consumers unless you’re building your own builder.

curl -H "Authorization: Bearer $KEY" \
  https://portal.scrapewise.ai/api/scraper-api/api/scraper/config/parameters

Response (200) — ScraperConfigParametersDTO (scraperType, paginationType, parameterType, commonParameters, postProcessKind, postProcessUnitPresets).

postProcessUnitPresets maps each NUMBER_EXTRACT unit preset name to its list of { token, factor } entries.

Errors — 400 (N/A) / 401 / 403 (N/A) / 404 (N/A) / 429 / 500.

Common errors

See Errors for the full envelope format.

What’s next

• Read what the scraper produced → Data
• One-shot preview without saving a scraper → Preview
• Use these tools from Claude → MCP quickstart