REST API overview

The REST surface that everything else (portal, MCP gateway, SDKs) is built on. Standard JSON over HTTPS.

Base URL

https://portal.scrapewise.ai/api/scraper-api

All endpoints below use this prefix. Throughout the docs, paths like /api/key/whoami mean https://portal.scrapewise.ai/api/scraper-api/api/key/whoami.

Headers

Every request:

Optional but useful:

Pagination

Read endpoints that can return many rows use page numbers. There are two styles, depending on the endpoint.

Matched results — page and size

GET /api/scraper/group/{id}/match/data (see Matched data) and most list endpoints take Spring-style paging:

GET /api/scraper/group/grp_xyz/match/data?page=0&size=1000&sort=matchScore,desc

The response is a page envelope: content, totalElements, totalPages, number, size. Stop when number + 1 equals totalPages.

Keep a multi-page read consistent. The matched results always come from the group’s newest finished matching run. If a new run finishes while you are paging, later pages come from the new run. To avoid mixing two runs, read the active run id once from GET /api/scraper/group/{id}/match/buckets (activeMatcherRunId) and put it in filters on every page, for example filters={"matcherRunId":"<run id>"} (URL-encoded). If the run changes mid-read, the next pages come back empty instead of mixed. Start again from page 0 when that happens.

Scraped data — page and size, or a keyset cursor

GET /api/scraper/data/group/{id} (see Scraped data) takes the same page / size / sort params (default size 20, maximum 10,000) and returns the same page envelope.

For big groups, ask for the slim shape with pad=false. The response then carries hasMore and a nextCursor object instead of totalPages. Send the cursor back on the next call:

GET /api/scraper/data/group/grp_xyz?pad=false&size=1000
GET /api/scraper/data/group/grp_xyz?pad=false&size=1000&cursor=<nextCursor.key>&cursorId=<nextCursor.id>&cursorSort=<nextCursor.sort>

1. First call: no cursor params.
2. Next call: copy nextCursor.key, nextCursor.id and nextCursor.sort into cursor, cursorId and cursorSort. Keep the same sort, filters and size.
3. Stop when hasMore is false.

Cursor paging does not re-read the rows it has already passed, so page 300 is as fast as page 1, and rows added during the read do not shift your pages. The total is counted once, on the first page only (totalCount). A cursor sent with a different sort is refused with 400.

Load everything in one call

GET /api/scraper/group/{id}/match/data?loadAll=true returns one bucket of matched results in a single response, up to 5,000 rows. The response carries cap (5,000) and the true totalElements. When totalElements is larger than the number of rows you got, page through the bucket instead.

Filtering + sorting

Most list endpoints accept query params:

For full filter syntax on a specific endpoint, see that endpoint’s page in the sidebar.

Errors

All errors return JSON in the problem-details envelope:

{
  "status": 422,
  "code": "validation_failed",
  "title": "Validation failed",
  "detail": "Field 'startUrls' must be a non-empty array.",
  "correlationId": "req_..."
}

Switch on code (machine-readable, stable). Include correlationId in support tickets.

Rate limits

Limits are counted per account (all API keys of one account share them), per second and per day. These are the default values. Scrapewise can change them per plan, so your account may have different limits.

Every allowed response carries X-RateLimit-Remaining — the requests left in the current window. When you go over a limit, you get 429 Too Many Requests with:

• Retry-After: <seconds> — wait this long, then retry;
• X-RateLimit-Remaining: 0;
• a JSON body with the code RATE_LIMITED and a sentence.

Spread bulk reads over time, use big pages (size up to 10,000) instead of many small ones, and back off on 429.

MCP gateway

The MCP gateway has its own limits on top of the account limits above. Defaults: 30 requests per second per API key (bursts up to 60) and 50 requests per second per IP address (bursts up to 100). Over the limit, the gateway answers 429 with Retry-After.

OpenAPI spec

Live spec served at:

https://portal.scrapewise.ai/api/scraper-api/v3/api-docs

Plus a Redoc-rendered browser view (when shipped — pending PR-Docs3). The spec is the source of truth — if this doc and the spec disagree, the spec wins.

For MCP-tag-only filter (the subset exposed to MCP agents):

https://portal.scrapewise.ai/api/scraper-api/v3/api-docs/mcp

Top-5 endpoints

The fastest path to “useful” — covered in detail below:

What’s next

• Fastest “hello world” → Preview
• Create + run a real scraper → Scrapers
• Read scraped data → Data
• Read matched results → Matched data