Scrapers

Sitemaps

Many shops publish a sitemap, a file that lists all their pages. Scrapewise can read it and keep the addresses for one scraper. You then pick the product pages out of it and turn them into the scraper’s link list.

The address of the sitemap is looked up in the shop’s robots.txt. Index files and zipped files are followed. Every robots.txt or sitemap file that is fetched costs one plain page.

Sitemap entries are kept for a short time only. They are wiped every night at 02:00 UTC. Read them again when you need them. A link list is the lasting list, see Link lists.

Open the 3-dots menu on the scraper box and pick Manage Links.
Open the Sitemaps view. Scrapewise reads the shop’s sitemap and lists the addresses.
Type a word in Search to keep only addresses that contain it, for example /product/.
Pick a choice in Restrict URLs to narrow the list by the shape of the address: None keeps every address, Same base path keeps addresses under the scraper’s base path, Same slash count keeps addresses as deep as the scraper’s own address. Use Rows per page to see more rows.
Press Add Links to put the addresses you kept on the scraper’s link list.
Run the scraper, see Runs and errors.

Base address https://portal.scrapewise.ai/api/scraper-api, header Authorization: Bearer <key>. Which key: a Read key looks, a Full key also changes things, see Read or Full.
With a Full key, read the sitemap: POST /api/scraper/{scraperId}/sitemaps/harvest with a new Idempotency-Key. The address is taken from the scraper’s config.sourceConfig.url. The query cap limits how many addresses are kept (default 500,000). The answer is the number of addresses stored. 0 means no sitemap was found or it was empty.
With a Read key, look at what was found: GET /api/scraper/{scraperId}/sitemaps with page, size, sort, search and restriction (NONE, BASE_PATH or SLASH_COUNT).
With a Full key, save the addresses you want as a link list, see Link lists.
There is no schedule for sitemap reads. If you need fresh addresses, call the harvest again yourself, for example once a day from your own script after 02:00 UTC. While entries for the shop are still stored, a new harvest returns the stored count and fetches (and charges) nothing.

bash

curl -X POST "https://portal.scrapewise.ai/api/scraper-api/api/scraper/[SCRAPER ID]/sitemaps/harvest?cap=1000" \
  -H "Authorization: Bearer $SCRAPEWISE_FULL_KEY" \
  -H "Idempotency-Key: [NEW UUID]"

bash

curl "https://portal.scrapewise.ai/api/scraper-api/api/scraper/[SCRAPER ID]/sitemaps?page=0&size=100&search=shoes&restriction=BASE_PATH" \
  -H "Authorization: Bearer $SCRAPEWISE_READ_KEY"

restriction BASE_PATH keeps entries under the scraper’s base path (portal: Same base path). SLASH_COUNT keeps entries with the same depth as the scraper’s address (portal: Same slash count). NONE keeps all (portal: None). A harvest stopped early by a limit or by the wallet sends the header X-Harvest-Limit-Reached (MAX_FILES, MAX_DEPTH, MAX_URLS, MAX_BYTES or WALLET_BUDGET). Each entry has url, image, imageTitle and lastSeen.

Connect your assistant with a Full key. See Connect Claude.
Ask it to read the sitemap: scrapewise_run_scraper_sitemaps_harvest (Full) with scraperId and an optional cap.
Ask it to look at the result: scrapewise_get_scraper_sitemaps (Read) with scraperId, search and restriction.
Ask it to save the addresses you want with scrapewise_create_scraper_site (Full), see Link lists.

json

scrapewise_get_scraper_sitemaps
{ "scraperId": "[SCRAPER ID]", "search": "/product/", "restriction": "BASE_PATH", "page": 0, "size": 100 }

Errors you may see

Harvest answers 0

The shop has no sitemap, it is empty, or it could not be reached. Check https://shop.com/robots.txt by hand.

400 scraper_not_found

No scraper with this id in your account.

402 WALLET_INSUFFICIENT_BALANCE

Reading the sitemap costs plain pages. Top up, see Wallet and billing.

429

A harvest is already running for your account. One runs at a time. Wait for it to end, then call again.

502

The sitemap address is unreachable or not valid. Open https://shop.com/robots.txt and the sitemap address in a browser. If the shop has no working sitemap, use Link lists instead: paste the product links or take them from a category scraper.

Questions

Why is my sitemap list empty the next day?

Sitemap entries are wiped every night at 02:00 UTC. Save what you need as a link list, or read the sitemap again.

Does reading a sitemap cost money?

Yes, one plain page for each file it fetches. See Wallet and billing for the price.

Should I put the whole sitemap on the link list?

No. A sitemap holds category and info pages too. Use Search or search to keep product pages only.

Everything on this screen

Sitemaps window:

None: A choice of Restrict URLs. It keeps every address.
Same base path: A choice of Restrict URLs. It keeps addresses under the scraper’s base path.
Same slash count: A choice of Restrict URLs. It keeps addresses as deep as the scraper’s address.

Last updated 4 November 2026