Delivery status and webhook

Your system needs to know when today’s data of a group is complete: every feed scraped, the matching done, and the daily matched file built. There are two ways:

• Ask: read the delivery status of a group for one day.
• Get told: set a webhook. Scrapewise calls your address when the day is complete, partial or stale, and when the daily file is ready.

All four are free. The same calls are MCP tools: scrapewise_get_group_delivery_status, scrapewise_update_group_delivery_webhook, scrapewise_delete_group_delivery_webhook and scrapewise_run_group_delivery_webhook_test. The portal shows the same status in the group’s delivery dialog.

Read the delivery status — GET /api/scraper/group/{id}/delivery-status

curl -H "Authorization: Bearer $KEY" \
  "https://portal.scrapewise.ai/api/scraper-api/api/scraper/group/grp_xyz/delivery-status?day=2026-10-23"

Response (200) — shortened:

{
  "groupId": "grp_xyz",
  "day": "2026-10-23",
  "timezone": "Europe/Tallinn",
  "status": "COMPLETE",
  "complete": true,
  "reasons": [],
  "scrapes": {
    "expected": 3, "finished": 3, "partial": 0, "failed": 0, "running": 0, "notStarted": 0,
    "feeds": [
      { "scraperId": "65c1...", "name": "shop-a", "scheduled": true, "runId": "65d0...",
        "state": "COMPLETED", "rows": 812, "previousRows": 805, "verdict": "OK" }
    ]
  },
  "matching": {
    "configured": true, "status": "COMPLETED", "lastMatchedAt": "2026-10-23T05:12:03Z",
    "hasNewData": false, "coversDay": true
  },
  "file": { "ready": true, "latestPath": "/api/scraper/group/grp_xyz/match/export/latest?format=jsonl&bucket=matched" },
  "webhook": { "url": "https://example.com/hooks/scrapewise", "enabled": true, "autoDisabledAt": null },
  "cutoffAt": "2026-10-23T11:00:00Z",
  "computedAt": "2026-10-23T08:04:39Z"
}

How the day is counted

• Time zone. Days and the cut-off use the platform time zone, Europe/Tallinn (timezone in the answer).
• A run counts for the day it finished. A scrape that starts at 23:50 and ends at 01:00 belongs to the second day.
• Before the cut-off (14:00 Europe/Tallinn by default), a feed that has not run yet is NOT_STARTED and the day is PENDING.
• At the cut-off, a feed that has not run becomes FAILED (reason NOT_RUN) and a feed still running becomes PARTIAL (reason OVERDUE). The day turns PARTIAL the same day.
• Row drop. A feed with far fewer rows than its previous run is PARTIAL (reason ROW_DROP).
• Stale. When nothing happened for 36 hours by default, the day is STALE.

Errors — 400 (INVALID_DAY: not YYYY-MM-DD or out of range; or no group with this id) / 401 / 429 / 500.

Webhook — PUT /api/scraper/group/{id}/delivery-webhook

Scrapewise sends an HTTPS POST with a signed JSON body to your address when something changes for the day.

curl -X PUT -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  "https://portal.scrapewise.ai/api/scraper-api/api/scraper/group/grp_xyz/delivery-webhook" \
  -d '{"url":"https://example.com/hooks/scrapewise","events":["FILE_READY","DAY_COMPLETE","DAY_PARTIAL","DAY_STALE"]}'

Response (200) — {url, events, enabled, createdAt, lastDeliveryAt, lastStatus, lastHttpStatus, autoDisabledAt, secret}.

Only the group’s owner can set, test or delete its webhook: with a Full key or in the portal. A Read key and a user the group is shared with are refused.

Errors — 400 (WEBHOOK_URL_REFUSED: not https, a private or local address, an unknown host, or a user name in the address; WEBHOOK_INVALID: an unknown event) / 401 / 403 (WEBHOOK_OWNER_ONLY, or a Read key) / 429 / 503 (WEBHOOK_UNAVAILABLE: webhooks are off on the server right now; the status still works) / 500.

Events

Each event is sent once per state, day, matching run and price refresh. So the same event can come again for the same day after a matching run or a price refresh. The body of a DAY_* event is the delivery-status document of that day.

Headers

Check the signature

The signed string is exactly <X-Scrapewise-Timestamp>.<raw body>: the timestamp header, a dot, and the body bytes as you received them, before any JSON parsing (UTF-8). The signature is HMAC-SHA256 of that string with your secret, written as lower-case hex.

• Accept the call when any one v1= value matches. Compare in constant time.
• You may refuse old timestamps, for example older than 5 minutes. Scrapewise does not enforce a window: your receiver chooses.

import hashlib
import hmac
 
from flask import Flask, abort, request
 
app = Flask(__name__)
SECRET = b"<the secret from the create call>"
 
@app.post("/hooks/scrapewise")
def scrapewise_webhook():
    raw = request.get_data()  # the raw bytes, before JSON parsing
    timestamp = request.headers.get("X-Scrapewise-Timestamp", "")
    signed = timestamp.encode() + b"." + raw
    expected = hmac.new(SECRET, signed, hashlib.sha256).hexdigest()
    header = request.headers.get("X-Scrapewise-Signature", "")
    values = [part.strip()[3:] for part in header.split(",") if part.strip().startswith("v1=")]
    if not any(hmac.compare_digest(expected, value) for value in values):
        abort(401)
    # Answer fast. Queue the work and process it later.
    return "", 204

Answers, retries and switching off

• Success is any 2xx answer. A 3xx is a failure: redirects are not followed.
• Timeout is 10 seconds by default. Answer fast and process the call later.
• Retries. By default a delivery has 5 attempts: the first, then retries after 1 minute, 5 minutes, 30 minutes and 2 hours. After that the delivery is dead.
• Switching off. After 20 deliveries in a row that used all their attempts (by default), the webhook switches itself off. The status webhook block then shows autoDisabledAt. Send PUT with "enabled": true to switch it on again; that also resets the count.

De-duplicate and keep the newest

• Delivery is at least once. De-duplicate on X-Scrapewise-Delivery: a retry has the same id.
• The delivery id removes retries only. A new send after a matching run or a price refresh has a new id.
• Events can arrive out of order. Per group and day, keep the one with the newest computedAt.

Change the secret

Delete the webhook, then create it again. The new secret is in the answer to that create call. Between the two calls there is a short gap with no webhook: an event in that gap is not sent, so read the delivery status after the new create.

Send a test call — POST /api/scraper/group/{id}/delivery-webhook/test

Sends one signed PING to your address now and tells you what your receiver answered. Use it right after you create the webhook, to check your address and your signature check.

curl -X POST -H "Authorization: Bearer $KEY" \
  "https://portal.scrapewise.ai/api/scraper-api/api/scraper/group/grp_xyz/delivery-webhook/test"

The body is {"event":"PING","groupId":…,"deliveryId":…,"computedAt":…}, with the same headers as real events (X-Scrapewise-Event: PING). A test is not queued and not retried. At most one test per minute per group.

Response (200) — {delivered, deliveryId, httpStatus, error}. delivered: true when your receiver answered 2xx.

Errors — 401 / 403 (WEBHOOK_OWNER_ONLY, or a Read key) / 404 (WEBHOOK_NOT_FOUND: the group has no webhook) / 429 (WEBHOOK_TEST_LIMIT: one test per minute) / 503 (WEBHOOK_UNAVAILABLE) / 500.

Remove the webhook — DELETE /api/scraper/group/{id}/delivery-webhook

Removes the webhook and its secret. Deliveries still waiting are not sent. Removing a webhook that does not exist is not an error.

Response (200) — {deleted, cancelledDeliveries}. deleted: false when the group had no webhook.

What’s next

• Download the daily file → Daily matched file
• Read matched rows → Matched data