Scrapers
Link lists
A link list is the set of web addresses a scraper reads on each run. A scraper that reads a fixed set of product pages needs one. Keep one scraper for each shop and put all the product links of that shop on its list. Never make one scraper for each link: each scraper is billed, scheduled and listed on its own.
There are three ways to fill a list. Pick one.
| You have | Use |
|---|---|
| A short list you can paste | Paste it in Add New Links |
| A long list | Upload it as a file, then feed the scraper from the file |
| Links that another scraper collects (for example a category scraper) | Scraper-based links |
- The list sits on a scraper, so make the scraper first: see Read a website. Then open the 3-dots menu on the scraper box and pick Manage Links.
- To paste links, press Add New Links. Type each address in a row, add a title if you like, and press Add Row for more rows. You can also upload an Excel file with the columns URL and Title, or paste from the clipboard. Press Add Links. For API scrapers the button is Add APIs and each row has a cURL command.
- To fill the list from another scraper, open Scraper-based links. Pick the Group and the Source scraper, then press Fetch links and add them. Switch on Automatically update links when the source scraper updates to refill the list after every run of the source.
- For an idealo scraper, pick a Market, type product codes or names, and press Make links. Load a file takes a list from a file.
- Remove single links with Remove. Clear All empties the list. Confirm Delete asks once more.
- Switch on Calculate Curls Automatically when an API scraper should build its cURL commands itself. Reuse same Conf on selected links copies the setup of one link to the ticked links.
- Run the scraper, see Runs and errors.
Saving a list replaces the old list. To add links, keep the old ones in the list and add the new ones.
- Base address
https://portal.scrapewise.ai/api/scraper-api, headerAuthorization: Bearer <key>. Which key: a Read key looks, a Full key also changes things, see Read or Full. - With a Full key, save a list on an existing scraper (no scraper yet? make one with
PUT /api/scraper, see Read a website, or use step 6):PUT /api/scraper/sitewithscraperIdandlinks(urlandtitlefor each), and a newIdempotency-Key. The list you send replaces the stored list. - Send the Site
idon every later save. A save withoutidmakes a new Site and runs keep reading the first one. Read it withGET /api/scraper/{id}/site(Read). - With a Full key, fill the list from another scraper:
PUT /api/scraper/sitewithscraperId,linksSourceScraperId,urlFieldNameandupdateLinksBySourceScraperAutomatically: true. The server copies the addresses. For a big list,PUT /api/scraper/site/build-from-source(Full) builds it on the server: sendscraperId,linksSourceScraperId, a new UUID asbuildRequestIdand a newIdempotency-Key.urlFieldNameandtitleFieldNameare optional. The answer has the saved Site andfeedCounts(rows read, rows with no address, repeats dropped, links built). - With a Read key, check the stored list:
GET /api/scraper/site/{siteId}/linkswithpage,size(at most 1000) andfilters. - With no scraper yet, use the bulk route with a Full key:
POST /api/scraper/url-list-run. It makes one scraper, stores the list and starts one run. - To remove a list, use a Full key:
DELETE /api/scraper/site/{siteId}. The scraper stays.
To see what a source scraper would give, before you fill a list from it, use a Read key. GET /api/scraper/data/group/{groupId}/scraper/{scraperId}/url-fields lists the columns of its latest run that hold a web address. GET /api/scraper/data/group/{groupId}/scraper/{scraperId}/urls returns the links built from that run. Choose the columns with titleFieldName and urlFieldName. A very big source can answer 429 FEED_BUSY: wait the seconds in Retry-After and ask again.
curl -X PUT "https://portal.scrapewise.ai/api/scraper-api/api/scraper/site" \
-H "Authorization: Bearer $SCRAPEWISE_FULL_KEY" \
-H "Idempotency-Key: [NEW UUID]" \
-H "Content-Type: application/json" \
-d '{
"scraperId": "[SCRAPER ID]",
"links": [
{ "url": "https://shop.com/product/1", "title": "DE | SKU-1 | shop.com" },
{ "url": "https://shop.com/product/2", "title": "DE | SKU-2 | shop.com" }
]
}'
curl -X PUT "https://portal.scrapewise.ai/api/scraper-api/api/scraper/site" \
-H "Authorization: Bearer $SCRAPEWISE_FULL_KEY" \
-H "Idempotency-Key: [NEW UUID]" \
-H "Content-Type: application/json" \
-d '{
"scraperId": "[SCRAPER ID]",
"linksSourceScraperId": "[SOURCE SCRAPER ID]",
"urlFieldName": "url",
"titleFieldName": "title",
"updateLinksBySourceScraperAutomatically": true
}'
curl -X PUT "https://portal.scrapewise.ai/api/scraper-api/api/scraper/site/build-from-source" -H "Authorization: Bearer $SCRAPEWISE_FULL_KEY" -H "Idempotency-Key: [NEW UUID]" -H "Content-Type: application/json" -d '{
"scraperId": "[SCRAPER ID]",
"linksSourceScraperId": "[SOURCE SCRAPER ID]",
"urlFieldName": "url",
"titleFieldName": "title",
"updateLinksBySourceScraperAutomatically": true,
"buildRequestId": "[NEW UUID]"
}'
To retry the same build, send the same buildRequestId again. After a build that worked, you get the stored result and nothing is built again.
curl -X POST "https://portal.scrapewise.ai/api/scraper-api/api/scraper/url-list-run" \
-H "Authorization: Bearer $SCRAPEWISE_FULL_KEY" \
-H "Content-Type: application/json" \
-d '{ "groupId": "[GROUP ID]", "name": "shop.com", "urls": ["https://shop.com/product/1", "https://shop.com/product/2"] }'
url-list-run takes up to 5,000 addresses as JSON. For up to 100,000 addresses send JSON Lines (REST only): the first line is a header with groupId, name and a clientRequestId, then one address per line. Duplicates are collapsed and never charged twice. Retry with the exact same name and the same addresses. A repeat of a finished call answers 409 PREVIOUS_RUN_ALREADY_CHARGED and does not scrape again. Send rerun: true to fetch only the addresses never tried, or force: true to scrape everything again and pay again.
A list that comes from a source scraper holds up to about 700,000 links for a product-page scraper.
- Connect your assistant with a Full key. See Connect Claude.
- Ask it to save a list:
scrapewise_create_scraper_site(Full) with a newidempotency_keyand abodywithscraperIdandlinks. It sends the Siteidwhen one exists. - Ask it to read the stored list:
scrapewise_get_scraper_site(Read) for the Site,scrapewise_get_scraper_site_links(Read) for the links. - For thousands of links, ask it (Full key) to upload a CSV with the columns
urlandtitleas an upload scraper, see Upload a file. Then it feeds the scraper’s list from that file withscrapewise_create_scraper_siteandlinksSourceScraperId,urlFieldName: "url",titleFieldName: "title". - Or build the list in the background from a source scraper:
scrapewise_update_scraper_site_from_source(Full). - To make a scraper from a plain list in one step:
scrapewise_run_scraper_url_list(Full), up to 5,000 addresses. - To remove a list:
scrapewise_delete_scraper_site(Full).
scrapewise_create_scraper_site
{
"idempotency_key": "[NEW UUID]",
"body": {
"scraperId": "[SCRAPER ID]",
"links": [ { "url": "https://shop.com/product/1", "title": "DE | SKU-1 | shop.com" } ]
}
}
Note
A scraper keeps a URL only once. If you compare one URL with two of your products, add a harmless parameter to the second link, for example &sku=SKU-456. Do not make a second scraper.
Errors you may see
rerun or force if you want a new run.scraperId to replace its list.buildRequestId was used for a different build. Make a new UUID.Retry-After and send it again.Questions
Why does my second save make a new list?
A save without the Site id makes a new Site. Read the scraper’s Site and send its id every time.
How do I add links to an existing list?
There is no add-one call. Send the old links plus the new ones. The list you send replaces the stored list.
What goes in the title?
Anything that helps you later, for example market, your own SKU and the shop: DE | SKU-123 | shop.com. Avoid numbers with 8 or more digits, they look like barcodes.
Do deleted links delete my data?
No. Deleting a list keeps the scraper and the rows it already saved.
Everything on this screen
Add New Links window:
- Optional title: A name for the row. For an API scraper it is called API Title. For a search link it is a keyword.
- The cURL box of a row, for API scrapers.
- The address box of a row.
- The upload icon. It reads an Excel or CSV file with the columns URL and Title (and CURL for API rows).
- Paste from clipboard: Fills the rows from what you copied.
- The bin at the end of a row. It removes that row.
- Add links: Puts the rows on the link list.
Scraper-based links and idealo:
- Group: Picks the group that holds the source scraper.
- A round button on a column. It marks the column that holds the picture address.
- Adds the links fetched from the source scraper to the list. The number is how many were fetched.
- The file box behind Load a file in the idealo forms.
- The example lines in the idealo boxes. Type one EAN or one search per line.
Last updated 4 November 2026