Skill: build competitor price scrapers

A skill is a short instruction file for your AI assistant. This one teaches the assistant how to set up competitor price scrapers on Scrapewise from start to finish, through the MCP gateway.

It is the same method we use for our own customer setups. It covers two cases:

You have product links (for example 12,000 product pages on 35 competitor shops). The assistant scrapes only those links and gives you comparable prices, for example price per 50 g in EUR.
You have shop names and want the whole catalogue of each shop (often tens of thousands of products), matched to your own product list.

Built for large setups

The skill is written for setups with dozens of competitor shops and thousands to tens of thousands of links. There is no limit on shops per group. These are the platform limits it plans with:

What	Limit
Upload a file of links or products	CSV or JSON Lines up to 200 MB, Excel up to 36 MB. See file limits
Link list fed from a file or another scraper	Up to about 700,000 links in one product-page scraper, filled by the server in one step
Links in one MCP call	Up to 5,000 (`scrapewise_run_scraper_url_list`)
Links in one REST JSON Lines upload	Up to 100,000. See Many URLs

For big lists the assistant does not paste links into a chat. It uploads your file once and lets the server fill the link lists from it.

Download

SKILL.md - the skill as one Markdown file.
build-scrapers.zip - the same file in a folder, ready to upload to apps that take a skill as a ZIP.

What the skill does

Step	What the assistant does	Tools it uses
1. Intake	Asks for shops, markets, links or shop names, the unit to compare (per item, per 50 g, per piece) and how often	none
2. Mode	Picks “given links” (one group, market and SKU in the link title) or “catalogue” (one group per market)	`scrapewise_get_scraper_group_list`, `scrapewise_create_scraper_group`
3. Source per shop	Looks for the most reliable source first: a JSON API, the shop platform’s product data, page data, and an AI scraper only as the last choice	`scrapewise_get_scraper_load_site_content`, `scrapewise_preview_scraper_from_url`, `scrapewise_preview_scraper_from_curl`
4. Build	Creates one scraper per shop with the same column names. Small link lists are saved directly; big ones are uploaded as a file and fed into the scrapers	`scrapewise_create_scraper`, `scrapewise_create_scraper_site`, `scrapewise_create_customer_schema`, `scrapewise_create_file_scraper`
5. Rules	Turns raw prices into comparable prices: price per unit, EUR, stock status	`scrapewise_preview_scraper_preview_rule`, `scrapewise_create_scraper`
6. Test	One small sample per scraper, checked against the live page	`scrapewise_get_scraper_sample_data`
7. First runs	One scraper at a time, then the whole group	`scrapewise_run_scraper`, `scrapewise_run_scraper_group`, `scrapewise_get_scraper_load_history`, `scrapewise_get_scraper_job_errors`
8. Check	Reads the rows and counts complete links. Names each missing link and why	`scrapewise_get_scraper_data_group_client`
9. Cost and schedule	Adds up the cost of a run. Sets a schedule only when you ask	`scrapewise_get_scraper_load_history`, `scrapewise_update_scraper_group_schedule`

Key ideas in the skill:

Reliable first, then scale, then cost. A shop’s own product data survives page redesigns. AI reading of the page is the last choice.
The AI copies, rules calculate. The AI extractor copies values as printed. Price per unit and currency conversion are done by after-scrape rules (QUANTITY_NORMALIZE, NUMBER_EXTRACT, CURRENCY_CONVERT, ENUM_MAP, REGEX_CLEAN).
Ask before spending. The assistant tests with single pages and asks you before full runs and schedules.

Before you start

A Scrapewise account and some wallet balance. See Sign up.
An API key with scope LLM_FULL (the skill creates and runs scrapers). See Scopes.
Your assistant connected to https://mcp.scrapewise.ai/mcp. See Claude Code, Claude Desktop or claude.ai.

Install

Claude Code

Put the file in your skills folder. For all projects:


mkdir -p ~/.claude/skills/build-scrapers
curl -o ~/.claude/skills/build-scrapers/SKILL.md https://docs.scrapewise.ai/skills/build-scrapers/SKILL.md

For one project only, use .claude/skills/build-scrapers/SKILL.md inside the project. Start a new session. Claude Code uses the skill when you ask for something like “build scrapers for these competitor links”.

Claude Desktop and claude.ai

If your plan supports skills, open Settings, find Skills (under Capabilities), and upload build-scrapers.zip. Make sure the Scrapewise connector is on for the chat.

Other MCP clients

Any assistant that can call MCP tools can use the skill as a plain prompt. Paste the content of SKILL.md into the system prompt, custom instructions or project instructions, and connect the Scrapewise MCP server.

Example session

This is an example setup, not a customer result.

You: Build scrapers for the competitor links in this file. Compare price per 50 g in EUR. Markets DK, DE and UK. Weekly is enough.

Assistant: The file has 12,000 links on 35 shops. 22 shops run on a shop platform with product data I can read directly, 9 have product data on the page, and 4 need AI reading. I uploaded the file once and made one group, “Competitor prices”, with one scraper per shop. Each scraper’s link list is filled from the file, with titles like DK | SKU-1003 | shop.dk. A sample of each scraper matches the live page. Shall I run the scrapers one at a time now? If most pages are plain pages, a full run of 12,000 pages costs about €2 to €4.

You: Yes.

Assistant: Done. 11,640 of 12,000 links have a complete row: price, sale price, pack size, price per 50 g and the EUR values. 360 links show “product not found” on the shop. I put them in a list for you. Shall I set a weekly run on Monday?

Good to know

Tests of one page and file uploads cost little or nothing. Full runs are charged per page delivered. See what a tool call costs.
The skill never deletes your scrapers, groups or data unless you ask.
You can edit the skill. For example, add your own column names or your own unit.