Connect

Skill: build competitor price scrapers

A skill is a short instruction file for your AI assistant. This one teaches the assistant how to set up competitor price scrapers on Scrapewise from start to finish, through the MCP gateway.

It is the same method we use for our own customer setups. It covers two cases:

You have product links (for example 12,000 product pages on 35 competitor shops). The assistant scrapes only those links and gives you comparable prices, for example price per 50 g in EUR.
You have shop names and want the whole catalogue of each shop (often tens of thousands of products), matched to your own product list.

Built for large setups

The skill is written for setups with dozens of competitor shops and thousands to tens of thousands of links. There is no limit on shops per group. These are the platform limits it plans with:

What	Limit
Upload a file of links or products	CSV or JSON Lines up to 200 MB, Excel up to 36 MB. See file limits
Link list fed from a file or another scraper	Up to about 700,000 links in one product-page scraper, filled by the server in one step
Links in one MCP call	Up to 5,000 (`scrapewise_run_scraper_url_list`)
Links in one REST JSON Lines upload	Up to 100,000. See Many URLs

For big lists the assistant does not paste links into a chat. It uploads your file once and lets the server fill the link lists from it.

Download

SKILL.md - the skill as one Markdown file.
build-scrapers.zip - the same file in a folder, ready to upload to apps that take a skill as a ZIP.

What the skill does

Step	What the assistant does	Tools it uses
1. Intake	Asks for shops, markets, links or shop names, the unit to compare (per item, per 50 g, per piece) and how often	none
2. Mode	Picks “given links” (one group, market and SKU in the link title) or “catalogue” (one group per market)	`scrapewise_get_scraper_group_list`, `scrapewise_create_scraper_group`
3. Source per shop	Looks for the most reliable source first: a JSON API, the shop platform’s product data, page data, and an AI scraper only as the last choice	`scrapewise_get_scraper_load_site_content`, `scrapewise_preview_scraper_from_url`, `scrapewise_preview_scraper_from_curl`
4. Build	Creates one scraper per shop with the same column names. Small link lists are saved directly; big ones are uploaded as a file and fed into the scrapers	`scrapewise_create_scraper`, `scrapewise_create_scraper_site`, `scrapewise_create_customer_schema`, `scrapewise_create_file_scraper`
5. Rules	Turns raw prices into comparable prices: price per unit, EUR, stock status. Scraper rules read only the shop’s own fields; EUR and per-unit prices are saved once on the group, with one choice per unit (per 50 g, per 100 g, per piece) picked by a label on each scraper	`scrapewise_preview_scraper_preview_rule`, `scrapewise_update_scraper_group_post_process_rules`
6. Test	One small sample per scraper, checked against the live page	`scrapewise_get_scraper_sample_data`
7. First runs	One scraper at a time, then the whole group	`scrapewise_run_scraper`, `scrapewise_run_scraper_group`, `scrapewise_get_scraper_load_history`, `scrapewise_get_scraper_job_errors`
8. Check	Reads the rows and counts complete links. Names each missing link and why	`scrapewise_get_scraper_data_group_client`
9. Cost and schedule	Adds up the cost of a run. Sets a schedule only when you ask	`scrapewise_get_scraper_load_history`, `scrapewise_update_scraper_group_schedule`

Key ideas in the skill:

Reliable first, then scale, then cost. A shop’s own product data survives page redesigns. AI reading of the page is the last choice.
The AI copies, rules calculate. The AI extractor copies values as printed. Price per unit and currency conversion are done by after-scrape rules (QUANTITY_NORMALIZE, NUMBER_EXTRACT, CURRENCY_CONVERT, ENUM_MAP, REGEX_CLEAN).
Fewest scrapers, fewest columns, no duplicates. One scraper per shop and data source, reused before a new one is made. The same column names on every scraper, and no column that a rule can make. No duplicate rows, sources or rules.
One column set for every setup. The same snake_case names on every scraper (price, sale_price, unit, unit_price, unit_price_eur …). One set of unit columns for every product kind, not one per unit.
Labels, not extra scrapers. Product kinds (for example yarn per 50 g, accessories per piece) are labels on the scrapers and one choice group on the group. One scraper per shop stays one scraper.
Shared rules live on the group. Rules every scraper needs (EUR, per unit, stock, text cleaning) are saved once for the whole group. See Rules for all scrapers in a group.
Ask before spending. The assistant tests with single pages and asks you before full runs and schedules.

Set it up

Open the portal, press your name at the bottom of the portal menu, then API Keys. Type a Key name, pick the Full scope (the portal shows it as LLM_FULL) and press Generate API Key. The skill creates and runs scrapers, so it needs a Full key, not a Read key. See Read or Full.
Connect your assistant to https://mcp.scrapewise.ai/mcp with that key. See Connect your assistant (MCP).
Make sure your wallet holds some balance. See Wallet and billing.
Install the skill. Claude Code: put the file in your skills folder with mkdir -p ~/.claude/skills/build-scrapers and curl -o ~/.claude/skills/build-scrapers/SKILL.md https://docs.scrapewise.ai/skills/build-scrapers/SKILL.md. For one project only use .claude/skills/build-scrapers/SKILL.md. Start a new session.
Claude Desktop and claude.ai: if your plan supports skills, open Settings, find Skills under Capabilities and upload build-scrapers.zip. Turn the Scrapewise connector on for the chat.
Any other assistant: paste the content of SKILL.md into its system prompt or project instructions.
Ask for something like “Build scrapers for these competitor links”. The assistant shows you its plan and asks before it spends.

The skill drives the MCP tools, not raw REST calls. If your own code should do the same work, follow the same order with the REST routes on the feature pages. Base address https://portal.scrapewise.ai/api/scraper-api, header Authorization: Bearer <key>.
Use a Full key. A Read key cannot create or run scrapers. See Read or Full.
With a Full key, make the group and the scrapers: Groups and Read a website. Fill the link lists: Link lists or Upload a file.
With a Full key, save the shared rules once on the group: Rules for all scrapers.
With a Full key, take a small sample, then run the scrapers one at a time. Read the history with a Read key: Runs and errors.
Read the rows and count the complete links: Exports and delivery.

Use a Full key. A Read key cannot create or run scrapers.
The skill calls the tools listed in the table above, in order, from scrapewise_get_scraper_group_list (Read) to scrapewise_update_scraper_group_schedule (Full). Most steps that change something use a Full key.
With a Full key, the assistant tests with single pages and file uploads first. They cost little or nothing. It asks you before full runs and before it sets a schedule.

Example session

This is an example setup, not a customer result.

You: Build scrapers for the competitor links in this file. Compare price per 50 g in EUR. Markets DK, DE and UK. Weekly is enough.

Assistant: The file has 12,000 links on 35 shops. 22 shops run on a shop platform with product data I can read directly, 9 have product data on the page, and 4 need AI reading. I uploaded the file once and made one group, “Competitor prices”, with one scraper per shop. Each scraper’s link list is filled from the file, with titles like DK | SKU-1003 | shop.dk. A sample of each scraper matches the live page. Shall I run the scrapers one at a time now? If most pages are plain pages, a full run of 12,000 pages costs about €2 to €4.

You: Yes.

Assistant: Done. 11,640 of 12,000 links have a complete row: price, sale price, pack size, price per 50 g and the EUR values. 360 links show “product not found” on the shop. I put them in a list for you. Shall I set a weekly run on Monday?

Good to know

Tests of one page and file uploads cost little or nothing. Full runs are charged per page delivered. See what a tool call costs.
The skill never deletes your scrapers, groups or data unless you ask.
You can edit the skill. For example, add your own column names or your own unit.

Last updated 4 November 2026