Scrapers
Read a website
A scraper reads a web page, finds the values you want on it (name, price, picture, stock) and saves them as rows. Make one scraper for each shop. Give that one scraper all the product links of the shop. Do not make one scraper for each link.
- Open your group on the home page. If you have no group yet, make one first, see Groups.
- Press + on the group box. The scraper form opens. The portal calls a scraper an automation.
- Type an Automation Name. Use the name of the shop, for example
shop.com. - Paste the page address into Website URL or cURL command. Use a product page, not a category page. Paste a cURL command instead when the shop has a JSON address behind its listing.
- Pick how to read the page. AI Automation finds the fields for you. Advanced lets you pick them yourself: open Content area selector and press Select from page to click the part of the page you want.
- Press Start training bot. Scrapewise checks the page for ready data first, then builds the fields. Press Run Automation if the form shows that button instead.
- Look at the sample rows. Check the price, name and picture against the live page. Press Back to Fields / Preview Data to change the fields, or Go Back to change the address.
- Press Save Selected Fields to keep the fields you ticked.
- Pick what comes next: Run scraper, Schedule scraper, Manage Links to add more product links, Go to Data to see the rows, or View All.
To change a scraper later, open the 3-dots menu on its box and pick Edit Automation. View Automation shows it without changing it. The same screen has Edit Config and View Data, and Back to Options returns to the choices from step 9. Enable / Disable switches the scraper on or off.
Optional settings on the form:
- Add page URLs saves the address of every row.
- Add SEO Fields also saves the page title and description.
- Mark as optional on a field lets a row stay valid when that field is empty.
- Next Link and Infinite Scroll tell the scraper how the shop shows page 2 and later. Page 2 URL, Page 3 URL and Pages describe numbered pages. Page link is the link to follow.
- Add a rule adds an after-scrape rule to this scraper only. Type a sample value and press Try it to see the result before you save. Rules that all scrapers of a group share belong on the group, see Rules for all scrapers.
- To keep a backup, open History on the group’s data page and press Create fallback scrapper. The backup runs when the main scraper finds nothing on a link.
AI scrapers read the fields of a custom schema. In the form pick Custom Schema Define your own fields, then Use existing or Create new. In the builder type a Schema name, then for each field a Field name and a Type, and switch on Mark as required where needed. Add field adds a row and Look sample shows an example. Save the schema.
To delete a scraper, open its 3-dots menu and pick Delete. The Delete the Scrapper dialog has the switch Delete associated data?. Leave it off to keep the rows. To delete several, press Select multiple, tick the boxes and confirm in Confirm Bulk Delete with Delete All. Press Exit selection mode when you are done.
- Base address
https://portal.scrapewise.ai/api/scraper-api, headerAuthorization: Bearer <key>. Which key: a Read key looks, a Full key also changes things, see Read or Full. - With a Full key, preview a page:
POST /api/scraper-simple/url-basedwithname,groupIdandurl. For a cURL command usePOST /api/scraper-simple/curl-basedwithcurl. Nothing is saved. Each preview fetches a page and costs one page. - Read the answer. It is a list of candidates, each with a
scraperDTOandsampleData. An empty list means no detector could read the page. - With a Full key, save the candidate you picked:
PUT /api/scraperwith thescraperDTOas the body and a newIdempotency-Keyheader. Leave outidto create. Sendidto update. Send the whole scraper, not a part. - Read
ruleWarningsin the answer. A warning is not an error, the save worked. - With a Full key, take a small sample before a full run:
GET /api/scraper/{id}/get-sample-data. It reads the first page only and saves nothing. - Run it with a Full key, see Runs and errors.
curl -X POST "https://portal.scrapewise.ai/api/scraper-api/api/scraper-simple/url-based" \
-H "Authorization: Bearer $SCRAPEWISE_FULL_KEY" \
-H "Content-Type: application/json" \
-d '{ "name": "shop.com", "groupId": "[GROUP ID]", "url": "https://shop.com/product/1" }'
curl -X PUT "https://portal.scrapewise.ai/api/scraper-api/api/scraper" \
-H "Authorization: Bearer $SCRAPEWISE_FULL_KEY" \
-H "Idempotency-Key: [NEW UUID]" \
-H "Content-Type: application/json" \
-d @candidate.json
Other calls on this page:
| Call | Key | What it does |
|---|---|---|
GET /api/scraper/list | Read | Lists your scrapers. Filter with groupId and q |
GET /api/scraper/{id} | Read | One scraper with its full setup |
PUT /api/scraper/v2 and GET /api/scraper/v2/{id} | Full and Read | Same as above, with the group, the schema and run totals in the answer |
GET /api/scraper/config/parameters | Read | The allowed values for types, pagination and rule kinds |
GET /api/scraper/load-site and GET /api/scraper/load-site-content | Full | Fetch the raw HTML of a page, to design selectors. Costs one page |
GET /api/scraper/{id}/seo-fields and POST /api/scraper/seo-fields | Full | Page title, description and social tags of a scraper’s address or of any address |
POST /api/scraper/preview-rule | Full | Test one after-scrape rule on a sample value. Nothing is saved, see Rules for all scrapers |
PUT /api/scraper/fallback | Full | Attach a backup scraper that runs when the main one finds nothing on a link |
PATCH /api/scraper/{id}/schema | Full | Change the schema of an AI scraper |
GET /api/schema/customer, PUT /api/schema/customer | Read, Full | List and make your own field lists for AI scrapers |
GET /api/schema and GET /api/schema/get/{id} | Read | The ready-made field lists (templates), and one field list by its id |
GET /api/schema/items/existing | Read | The common field names you can pick from, for example price |
GET /api/schema/custom | Read | One row for each custom schema you made, with its versions |
PATCH /api/schema/custom/{groupId}/archive | Full | Hides a custom schema from your list. This is what the bin icon on a schema card does |
Delete a scraper in two calls. Call 1 shows what will go and gives a token that is valid for 5 minutes. Call 2 deletes.
curl -X POST "https://portal.scrapewise.ai/api/scraper-api/api/scraper/[SCRAPER ID]/preview-delete?withData=false" \
-H "Authorization: Bearer $SCRAPEWISE_FULL_KEY"
curl -X DELETE "https://portal.scrapewise.ai/api/scraper-api/api/scraper/[SCRAPER ID]?withData=false" \
-H "Authorization: Bearer $SCRAPEWISE_FULL_KEY" \
-H "Idempotency-Key: [NEW UUID]"
withData=false keeps the rows the scraper already saved. withData=true deletes them too and cannot be undone. A scraper with a run in progress answers 409 SCRAPER_RUN_ACTIVE: stop the run first. A custom schema is deleted the same way with POST /api/schema/customer/{id}/preview-delete and then DELETE /api/schema/customer/{id}.
- Connect your assistant with a Full key. See Connect Claude.
- Ask it to read the page and show candidates:
scrapewise_preview_scraper_from_url(Read) withname,groupIdandurl, orscrapewise_preview_scraper_from_curl(Read) withcurl. Each preview costs one page. - Before it makes a scraper, ask it to look at what the group already has:
scrapewise_get_scraper_list(Read) withgroupId, thenscrapewise_get_scraper(Read) to see the address of a candidate. One scraper for each shop and data source. Add links to the one that exists. - Ask it to save the candidate:
scrapewise_create_scraper(Full) with a newidempotency_key. It sends the whole scraper. Usescrapewise_create_scraper_v2(Full) to get the group and run totals back. - Ask it to read the sample before a full run:
scrapewise_get_scraper_sample_data(Read). It saves nothing. - Ask it to test each after-scrape rule:
scrapewise_preview_scraper_preview_rule(Read). - To delete, ask it for the preview first:
scrapewise_delete_scraper_preview(Full), show you the result, and only after your yesscrapewise_delete_scraper(Full).
scrapewise_preview_scraper_from_url
{ "name": "shop.com", "groupId": "[GROUP ID]", "url": "https://shop.com/product/1" }
More tools on this page: scrapewise_get_scraper_v2 (Read), scrapewise_get_scraper_config_parameters (Read), scrapewise_get_scraper_load_site and scrapewise_get_scraper_load_site_content (Read, one page each), scrapewise_get_scraper_seo_fields and scrapewise_search_scraper_seo_fields (Read), scrapewise_update_scraper_fallback (Full), scrapewise_update_scraper_schema (Full), scrapewise_create_customer_schema (Full), scrapewise_list_customer_schema (Read), scrapewise_delete_customer_schema_preview and scrapewise_delete_customer_schema (Full).
When you ask an assistant to build scrapers for a whole shop list, point it to the agent skill. It follows the best practice below.
Best practice
- One scraper for each shop and data source. Put all links of that shop on its link list, see Link lists.
- Use the same column names on every scraper of a group, for example
name,price,sale_price,currency,url. The after-scrape rules of the group then work on all of them. - Scrape only raw values. Do not scrape a price per unit or a price in EUR. The group rules calculate those.
- Pick the most stable source first: a JSON address behind the shop, then a JSON-LD block, then page selectors, then an AI scraper.
- Run the first scrape of each scraper alone and check the rows before you run many.
Errors you may see
Questions
Does the preview cost money?
Yes, one page each, the same as a normal page. The price is on Wallet and billing. Saving a scraper is free.
Why does a save need the whole scraper?
A save replaces the scraper. Read it first with GET /api/scraper/{id}, change what you need and send all of it back.
Can I make one scraper for many links?
Yes, that is the normal way. See Link lists for pasting a list, uploading it or feeding it from another scraper.
The site changes its web address every week. What now?
Switch on AI control for the group, see AI control. It checks the scrapers and fixes broken ones. For sites that rotate parts of their address, open the scraper and add a changing part (Which page to look at, Add a changing part, Where to find it, How to grab it).
Everything on this screen
Add automation dialog on the home page:
- Cancel: closes the dialog without saving.
- Retry: tries the read again after an error.
- Try Another URL: goes back to type a different web address.
- Looks Good - do not save data: keeps the automation and does not save the test rows.
- Save automation + save data: keeps the automation and also saves the rows of this test read.
- Not happy with the result?: opens ways to improve the read.
- Show templates for a specific industry: opens ready-made starting points. Hide closes them. A tile that says it is not available on your plan cannot be picked.
First-run wizard (the first scraper of a new account):
- Method cards. The card marked “This is what we used” shows how Scrapewise read your page. A card marked “Not available on your plan” cannot be picked.
- Cancel: Closes the open window. Nothing is changed.
- Try Another URL: Goes back so you can type a different page address.
- Retry: Reads the page again after an error.
- Looks Good - do not save data: Keeps the scraper but does not run it now.
- Save automation + save data: Saves the scraper and starts its first run. The run is charged like any other run.
- Not happy with the result?: Opens other ways to read the same page.
- Show templates for a specific industry: Shows or hides the ready-made templates for one kind of business.
Templates on the scraper form (AI Automation):
- + Industry-specific automations: Opens the industry templates.
- − Hide: Closes the industry templates.
- An industry card (ai-conf-industry): press it to see the templates for that industry.
- A template card (ai-conf-sub-schema): press it to read the page with that ready-made field list.
- A greyed template card marked Not available for this account. Your account cannot use it.
Page window and form controls:
- The depth box. It says how many levels of links the scraper follows when it looks through a site. The default is 4.
- Select Elements: Turns on point-and-click mode in the page window. Click a part of the page to pick it.
- Enter Selector Mode: Turns the click-to-pick mode on or off.
- Enter Area Selection: Lets you draw a box around the part of the page you want, and leave that mode again.
- Close browser: Closes the page window.
- BACK: Goes back one step in the form.
- Preview Data: Switches between the field list and a preview of the data.
- Value: The value of one page setting (parameter) of the scraper, next to its name.
- close: The cross at the top of a window. It closes the window.
Rules on one scraper (Add a rule):
- The rules box. It says that rules make new columns from your scraped data, that your original columns stay the same, and that they run at scrape time. Click a chip to edit a rule.
- The icon on a rule chip: removes that rule from the scraper.
- Remove this row: Takes one row out of a rule list or out of a custom schema.
- A rule condition, first box: the column the condition looks at.
- A rule condition, second box: how it compares, for example equals or is not blank.
- A rule condition, third box: the value it compares with.
- The bin on a condition: takes that condition out of the rule.
- Unit word box of a Price per unit rule: a word that stands for a unit in the text, for example kg.
- The number box next to a unit word: how many base units that word is worth.
- The bin on a unit word: takes it out of the rule.
- The tick box under a rule: when it is on, the rule writes over the value the page gave in that column.
A page address that changes (changing part):
- Show me how it works: Opens a worked example of a changing part in a web address.
- The bin on a changing part: takes that changing part out of the address.
- The cross on a message or small window: closes it.
Custom schema (your own field list):
- The bin icon on a field row: takes the field out of the schema.
- Save as new version: Saves the schema. Save as new version keeps the old version and adds a new one.
- The window “A schema with this name already exists” appears when the name is taken. Pick what to do next.
- The button that saves your change as a new version of the schema with that name.
- Create separate schema: Makes a new schema with its own name instead.
- The buttons that list schemas with the same name. Pick the one to add a version to.
- The window “This schema changed” appears when someone saved the schema while you were editing it.
- Save anyway: Saves your version on top of the newer one.
- The window “Example: Product Availability” shows a worked example of a custom schema.
- Got it: Closes the example window.
- The bin icon on a schema card. It hides the schema from your list (the API tab calls this archive).
Group box and its menu (more on each page):
- The three dots on a group box or a scraper box. They open its menu.
- The plus on a group box. It opens the scraper form.
- The Add button of a group box. It opens the scraper form.
- One scraper row in a group box.
- The group box title and its description line.
- The group menu: Schedule, Data retention, Rules for all scrapers, AI control, Data, Delivery status, Edit info, Move to desktop, Upload, Find emails, Merge and Delete. See Groups.
- The scraper menu: Schedule, Data retention, Edit Automation, View Automation, Manage Links, Find emails, Enable or Disable and Delete.
- The window that asks you to confirm before a scraper is switched on or off.
- Delete: Deletes what the open window asks about.
- Update: Saves the group or desktop name you typed.
- The box above the group table. Type a part of a group name to find it.
- Edit: Opens the group in the table for editing.
- View: Opens the data of the group in the table.
Windows from the group menu (each has its own page):
- The AI control window. See AI control.
- The delivery status window. See Exports and delivery.
- The window for the rules of the whole group. See Rules for all scrapers.
- Save: Saves the window you are in.
- Retry: Tries the last step again.
- A note on a scraper: the group sets how long data is kept.
- The data retention window of a group. See Groups.
- A note on a scraper: the group sets when it runs.
- The schedule window of a group. See Schedules.
- Time: The time of day the scheduled run starts.
- Timezone: The time zone that time belongs to.
- Details: Opens the details of a row in the Scrapers list of Rules for all scrapers.
- The list that filters the Scrapers list by label.
- The pencil on a rule or a row. It opens that item for editing.
- Prev: Shows the previous page of the list.
- A page number of the list. Press it to open that page.
- The box for a new label name in Rules for all scrapers, Labels.
- Saves the new label name.
Windows from the group menu: Find emails (see Find emails):
- Find emails: The Find emails window.
- The button that starts the Find emails window.
- From a scraper: Takes the websites from a scraper of the group.
- Paste or upload: Takes the websites you paste or upload.
- Scraper: Picks the source scraper.
- Website column: Picks the column that holds the web addresses.
- A switch. Each later run then adds its new websites to the list.
- Upload file: Picks a file of websites.
- The file box behind Upload file.
- Name: The name of the email list.
- Next: Goes to the cost step.
- Back: Goes back to change the list.
- Add funds: Opens the top-up window when the balance is too low.
Last updated 4 November 2026