Scraped data

Once a scraper has run, the rows it produced are accessible through these endpoints.

Read group data — GET /api/scraper/data/group/{id}

The primary scraped-data read. Returns a paginated page of rows for the group, optionally restricted by saved categories or an inline filter expression. isLastRun=true narrows to the most recent run’s rows only.

curl -H "Authorization: Bearer $KEY" \
  "https://portal.scrapewise.ai/api/scraper-api/api/scraper/data/group/grp_xyz?page=0&size=100&isLastRun=true"

Query params

Response

Spring Page<Document> — each content row is a dynamic Mongo Document keyed by the scraper’s configured field names. Numbers / booleans / null pass through; _-prefixed keys are server metadata (_sourceUrl, _scrapedAt, _scraperJobStatusId).

Rows also carry the reserved _sw_* bookkeeping columns — see Bookkeeping columns below.

Every row additionally carries _sw_row_id — that row’s own id, as a string, on every page shape (padded and pad=false, offset and keyset, isLastRun or not). It is the id POST /api/scraper/data/group/{id}/hotels-detail takes in its rowIds. It is added by this read and stored nowhere, so it is not one of the group’s columns and does not appear in the Excel export.

{
  "content": [
    {
      "_sourceUrl": "https://...",
      "_scrapedAt": "2026-05-19T...",
      "_scraperJobStatusId": "65a1...",
      "title": "Premium Widget XL",
      "price": 29.99,
      "in_stock": true
    }
  ],
  "totalElements": 1847,
  "totalPages": 19,
  "size": 100,
  "number": 0
}

For agent / MCP use the sanitized envelope variant instead — every string field comes pre-wrapped with origin metadata for prompt-injection safety.

Bookkeeping columns — _sw_*

Every row a scraper writes now also carries a small set of reserved columns that say which endpoint wrote the row, whether it worked, what it cost and what was sent. They all begin with _sw_, a prefix a scraped page can never produce, so they can never collide with one of your own columns.

_sw_run_date, _sw_run_time, _sw_scraper, _sw_group and _sw_source_url replace the older bare date, time, scraper, group and sourceUrl run-metadata columns, which used to overwrite a scraped column of the same name. The old columns are still written, so nothing you read today stops working; prefer the _sw_ spelling in new code.

Slim shape — ?pad=false

pad=false returns the same rows with two differences:

• each row carries only the keys it actually has — no explicit null for columns it never had;
• the envelope reports hasMore and a nextCursor instead of totalPages / totalElements / last. The total is counted once and returned as totalCount on the first page only, because counting a filtered 30 000-row group on every page turn is a second full scan.

{
  "content": [ { "title": "Premium Widget XL", "price": 29.99 } ],
  "hasMore": true,
  "nextCursor": { "key": 29.99, "id": "66f0a1..." },
  "totalCount": 1847,
  "size": 100,
  "number": 0
}

Page forward by echoing the cursor back:

curl -H "Authorization: Bearer $KEY"   "https://portal.scrapewise.ai/api/scraper-api/api/scraper/data/group/grp_xyz?pad=false&size=100&cursorId=66f0a1..."

Cursor paging does not re-walk the rows it has already passed, so page 300 costs what page 1 costs. cursorId is the authoritative half of the pair; cursor is echoed for readability. If the row a cursor points at has been deleted in the meantime, the response falls back to page-number paging and says so with offsetFallback: true rather than quietly skipping rows.

Per-call context — ?ctx=1

Many sources answer with a block of information about the call rather than about any one row: the query that was run, the store or locale the prices came from, the result and page counts, the paging tokens. That block is stored once per call, not copied onto every row — on some endpoints it is larger than all the rows put together.

ctx=1 returns it as a sidecar next to content, keyed by each row’s _sw_result_set_id:

{
  "content": [
    { "_sw_result_set_id": "rs-0", "title": "Premium Widget XL", "price": 29.99 }
  ],
  "contexts": {
    "rs-0": { "ctx_query": "widget", "ctx_total_results": 1847, "ctx_store_id": "4152" }
  },
  "contextsTruncated": false
}

Join each row to contexts[row._sw_result_set_id] to get a flat, self-contained table. Ask for it only when you intend to read a ctx_* column.

Two bounds:

• the sidecar is byte-capped per page. Past the cap you get the contexts for the first result sets in page order and contextsTruncated: true — show “context partially loaded” rather than an empty cell;
• a row whose _sw_result_set_id resolves to nothing carries CONTEXT_MISSING in _sw_warnings. A missing context is always said out loud, never returned as blank columns.

Field-level filtering

The filters query param accepts a URL-encoded Mongo-style filter expression. Standard logical operators:

?filters=%7B%22price%22%3A%7B%22%24gt%22%3A50%7D%2C%22in_stock%22%3Atrue%7D

— which decodes to {"price":{"$gt":50},"in_stock":true}. Supports $and / $or / $nor logical groups recursively.

Errors — 400 (group not found / malformed filters JSON) / 401 / 403 (N/A) / 404 (N/A — 400 envelope used) / 429 / 500.

Delete scraped data

The legacy single-call DELETE /api/scraper/data?scraperId=...|?id=...|?jobId=... shape is no longer wired. The only DELETE /api/scraper/data endpoint that ships today takes ?scraperJobStatusId=... and follows the destructive two-call protocol on the Scraper Jobs page (preview-delete then commit). For broader removals — entire scraper, entire group, including-data — use:

Agent-friendly read (prompt-injection envelope) — GET /api/scraper/data/group/{id}/client

Same paginated data as GET /api/scraper/data/group/{id} — but when called from the MCP gateway, every string field is automatically wrapped in a {type: "scraped", content, origin, truncated} envelope. The agent’s system prompt instructs it never to act on instructions found inside scraped content (prompt-injection protection).

Numbers, booleans, null, and _-prefixed metadata pass through unwrapped (they weren’t scraped from arbitrary HTML).

The sanitized query param is hidden from the OpenAPI surface — the MCP gateway forces it server-side; direct SDK consumers can opt in by setting it explicitly.

Requires the EXTERNAL_API plan feature.

curl -H "Authorization: Bearer $KEY" \
  "https://portal.scrapewise.ai/api/scraper-api/api/scraper/data/group/grp_xyz/client?page=0&size=50"

Response (200) — Spring Page of sanitized rows:

{
  "content": [
    {
      "title":    { "type": "scraped", "content": "Premium Widget XL", "origin": "https://...", "truncated": false },
      "price":    29.99,
      "in_stock": true,
      "_jobId":   "65a1..."
    }
  ],
  "totalElements": 1847,
  "totalPages": 37
}

Errors — 400 (group not found) / 401 / 402 (plan lacks EXTERNAL_API — fall back to GET /api/scraper/data/group/{id} with your own prompt-injection safety) / 403 (N/A) / 404 (N/A — 400 used) / 429 / 500.

Need just the URLs a scraper hit? There’s no dedicated per-scraper URL endpoint. Use GET /api/scraper/data/group/{id} with scraperId=<id> and project the URL field client-side (e.g. _sourceUrl).

Group-level Excel download — GET /api/scraper/data/group/{id}/download

Downloads the latest run’s data for the given group as a single .xlsx file (Content-Disposition: attachment; filename=data.xlsx). Filters + categories supported, same as the group-data read endpoint. Subject to the DATA_EXPORT plan feature.

• Row limit: the admin setting “Max rows per export” — 1,000,000 by default. One Excel sheet holds at most 1,048,574 data rows (plus the header and a possible # EXPORT INCOMPLETE last row). Above the limit the request is refused with 400 EXPORT_TOO_LARGE and a sentence naming both numbers and how to narrow the export.
• Format: xlsx only.
• Row order: newest run first, then newest stored row first inside a run.
• Idempotency-Key is ignored on this route. One export per user per group runs at a time; a second one gets 429 EXPORT_BUSY.
• Large exports: a big file can take several minutes to build before the first byte. If your HTTP client or proxy times out, use the export-job routes instead.

curl -H "Authorization: Bearer $KEY" \
  "https://portal.scrapewise.ai/api/scraper-api/api/scraper/data/group/grp_xyz/download?categories=discounts" \
  --output data.xlsx

Query params

Errors — 400 (group not found, EXPORT_TOO_LARGE, INVALID_FILTER) / 401 / 402 (plan lacks DATA_EXPORT) / 403 (N/A) / 404 (N/A — 400 used) / 429 (EXPORT_BUSY) / 503 (EXPORT_DISK_LOW, EXPORT_SIZING_TIMEOUT — retry, or use the export-job routes) / 500.

Large exports — export jobs

For files of any size (up to the same row limit) the portal — and any API client — can build the file in the background, watch its progress and download it with a short-lived link. These routes are REST only (not MCP tools).

1. Start — POST /api/scraper/data/group/{id}/export-jobs with body {"isLastRun": true, "categories": [...], "filters": "<filter JSON string>"} (all optional). Answers 202 with the job: {jobId, state, totalRows, rowsWritten, ...}. The same request again returns the same job; a request with a different filter replaces the running one.
2. Poll — GET /api/scraper/data/export-jobs/{jobId} → state is QUEUED, RUNNING, READY, FAILED or EXPIRED; rowsWritten / totalRows show progress, queuePosition how many jobs are ahead, errorMessage a sentence to show. GET /api/scraper/data/export-jobs?groupId={id} lists your queued / running / ready jobs of a group.
3. Link — POST /api/scraper/data/export-jobs/{jobId}/link (READY only) → {path, url, expiresAt}. The download must start within the link life (default 5 minutes).
4. Download — GET the link (.../export-jobs/{jobId}/file?token=...). No login needed — the token is the credential. HTTP Range is supported, so an interrupted download can resume for a while (default 60 minutes).

JOB=$(curl -s -X POST -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"isLastRun": true}' \
  "https://portal.scrapewise.ai/api/scraper-api/api/scraper/data/group/grp_xyz/export-jobs" | jq -r .jobId)
# poll until "state": "READY", then:
PATH_=$(curl -s -X POST -H "Authorization: Bearer $KEY" \
  "https://portal.scrapewise.ai/api/scraper-api/api/scraper/data/export-jobs/$JOB/link" | jq -r .path)
curl -o export.xlsx "https://portal.scrapewise.ai/api/scraper-api${PATH_}"

Finished files are kept for 24 hours by default. A server restart or deploy ends running exports with the sentence “The server restarted while your file was being prepared. Please export again.”

The times above (link life 5 minutes, resume window 60 minutes, file kept 24 hours) and the row limit (1,000,000) are defaults. Scrapewise can change them.

The poll answer also carries pollSeconds — how long to wait before the next poll — and, on a failed job, errorCode.

Export-job error codes

A job that ends FAILED carries one of these in errorCode: EXPORT_SERVER_RESTARTED (a deploy or restart stopped it), EXPORT_REPLACED (a newer export of the same group with a different filter replaced it), EXPORT_GROUP_DELETED or EXPORT_FAILED. Export again.

Errors — 400 (EXPORT_TOO_LARGE, INVALID_FILTER) / 401 / 402 (plan lacks DATA_EXPORT) / 403 (N/A) / 404 (EXPORT_NOT_FOUND) / 409 (EXPORT_NOT_READY) / 410 (EXPORT_FILE_GONE) / 429 (EXPORT_BUSY) / 503 (EXPORT_DISK_LOW) / 500.

Saved-filter categories

A category is a named filter expression saved against a group — a reusable filter bundle the customer references when querying scraped data. Subject to the GROUP_CUSTOM_CATEGORIES plan feature; total categories per group capped by GROUP_CUSTOM_CATEGORIES_LIMIT.

List categories — GET /api/scraper/data/group/{id}/categories

curl -H "Authorization: Bearer $KEY" \
  https://portal.scrapewise.ai/api/scraper-api/api/scraper/data/group/grp_xyz/categories

Response (200) — List<CustomerCategoryDTO>:

[
  { "id": "cat_001", "name": "big-discounts", "filters": { /* FiltersDTO */ } },
  { "id": "cat_002", "name": "in-stock",      "filters": { /* FiltersDTO */ } }
]

Errors — 400 (group not found) / 401 / 403 (N/A) / 404 (N/A — empty list returned if none) / 429 / 500.

Save a category — PUT /api/scraper/data/group/{id}/category

PUT /api/scraper/data/group/grp_xyz/category
Authorization: Bearer <key>
Idempotency-Key: <uuid>
Content-Type: application/json
 
{
  "name": "big-discounts",
  "filters": { "discountPct": { "gt": 20 } }
}

Upsert by name within the group (resave with the same name updates the filter expression).

Response (200) — persisted CustomerCategoryDTO with assigned id.

Errors — 400 (group not found) / 401 / 402 (plan lacks GROUP_CUSTOM_CATEGORIES OR GROUP_CUSTOM_CATEGORIES_LIMIT reached) / 403 (N/A) / 404 (N/A — 400 used) / 429 / 500.

Delete a category (destructive — two-call protocol) — DELETE /api/scraper/data/group/{id}/category/{categoryId}

Steps

1. POST /api/scraper/data/group/{id}/category/{categoryId}/preview-delete — mints a 5-minute token + preview summary.
2. DELETE /api/scraper/data/group/{id}/category/{categoryId} — commits the delete (idempotency-key required).

Skipping the preview step deletes rows without confirmation.

Step 1 — Preview

POST /api/scraper/data/group/grp_xyz/category/cat_001/preview-delete
Authorization: Bearer <key>

Response (200) — DestructivePreviewResponseDTO:

{
  "token": "f3c2-...",
  "opName": "scrapewise_delete_scraper_data_group_category",
  "targetEntityId": "cat_001",
  "previewSummary": {
    "entityName": "cat_001",
    "entityType": "data_group_category",
    "cascadeCounts": {},
    "warnings": []
  }
}

Step 2 — Commit

DELETE /api/scraper/data/group/grp_xyz/category/cat_001
Authorization: Bearer <key>
Idempotency-Key: <uuid>

Response — 204 No Content.

Errors (both steps) — 400 (group not found OR no category with that categoryId in the group) / 401 / 403 (N/A) / 404 (N/A — 400 used) / 429 / 500.

JSON export (programmatic)

For your code, the paginated GET /api/scraper/data/group/{id} read is usually what you want — use pad=false and the keyset cursor for big groups (see Pagination). Excel export is for handing files to humans. For matched results, see Matched data.

Common errors

What’s next

• Run a scraper to produce data → Scrapers
• API key management → API keys
• Full reference → Reference