Skill: web-scraping

Fork

Clean LLM-ready web scraping via Firecrawl (scrape/crawl/map/agent/search). Trigger when the user wants to extract content from a page, crawl a site, collect structured data, bypass anti-bot/JS-rendering, or perform a web search with integrated extraction. Fallback to Playwright/curl if Firecrawl is unavailable.

Configuration

Property	Value
Context	fork
Pre-approved tools	None declared: the project's permission settings apply
Trigger	keywords: `web`, `scraping`, `extract data from ...`, `crawl site x`, `fetch all articles from ...`, `parse this dynamic page`, `web research`

Detailed description

Web Scraping (Firecrawl-first)

Goal

Extract LLM-ready web content without hacking around: clean markdown, structured JSON, anti-bot and JS-rendering handled. Firecrawl is the reference wrapper; fallback to Playwright or curl + html2text if unavailable.

When to trigger this skill

"scrape this page / this site"
"extract data from ..."
"crawl site X"
"fetch all articles from ..."
"search the web and extract the content"
"parse this dynamic page" (site with JS-rendering)
"bypass the paywall / anti-bot" (legitimate use only)

When NOT to use this skill

Quick web search without structured extraction -> WebSearch is enough
A single static URL, simple page -> WebFetch is enough
Visual test / browser interaction -> agent-browser, or ask the user to run /qa:qa-chrome (the qa-chrome skill is manual-only)
Form / login automation -> agent-browser or Playwright directly

Prerequisites

Option 1: Firecrawl cloud (recommended)

export FIRECRAWL_API_KEY="fc-xxx"      # https://firecrawl.dev
npm install -g firecrawl-cli           # the CLI (`firecrawl`); the SDKs are firecrawl / firecrawl-py

Option 2: Firecrawl self-hosted

Docker compose available on github.com/mendableai/firecrawl. Useful if data is sensitive or budget is limited.

Option 3: Fallback without Firecrawl

If Firecrawl is missing, degrade gracefully:

Need	Fallback	Limitation
Simple static page	`curl -sL URL \| pandoc -f html -t markdown`	No JS rendering
JS-heavy page	`npx playwright` + `page.content()` + markdownify	Heavy, 300MB+ of deps
Whole site	recursive filtered wget	No deduplication, no LLM-ready output

IMPORTANT: always announce when degrading. The user must know if the content is partial (JS not rendered).

The 5 Firecrawl operations (CLI `firecrawl-cli` v1.25)

1. Scrape (one URL)

firecrawl scrape https://example.com/article \
  --format markdown,links \
  --only-main-content

Output: clean markdown (navigation / footers stripped), list of links, OG metadata.

2. Crawl (whole site)

firecrawl crawl https://docs.example.com \
  --limit 100 \
  --include-paths /docs \
  --exclude-paths /docs/legacy \
  --scrape-options '{"formats":["markdown"]}' \
  --wait -o crawl.json

Output: one JSON file holding every page (markdown in each entry). Ask for confirmation before crawl > 50 pages (API costs + time).

3. Map (URL discovery)

firecrawl map https://example.com --search "pricing"

Output: list of relevant URLs. Useful BEFORE a crawl to target the right sections.

4. Agent (structured data via LLM)

firecrawl agent "Extract the plans with name, price and features" \
  --urls https://example.com/pricing \
  --schema-file plans.schema.json \
  --max-credits 50 --wait

Output: JSON conforming to the schema. Saves hours of fragile CSS selectors. An agent job takes minutes and spends credits: always set --max-credits. (The CLI's former extract command is gone; agent replaced it.)

5. Search (search + extract in one pass)

firecrawl search "best pve proxmox backup strategies" \
  --limit 10 \
  --scrape --scrape-formats markdown

Output: top N results with extracted content. Replaces WebSearch + N WebFetch.

Recommended workflow

1. IDENTIFY the need
   - 1 page               -> scrape
   - N known pages        -> scrape in a loop with `xargs -P 4`
   - Whole site           -> map (recon) -> targeted crawl
   - Structured data      -> agent with a schema
   - Search + extract     -> search

2. ESTIMATE costs
   - Firecrawl cloud: credits per page scraped
   - Ask for confirmation if > 50 pages or > 10 MB expected

3. RUN with limits on the first attempt
   - --limit 5 to test
   - Inspect the output
   - Re-run at full volume if OK

4. SAVE the result
   - `./scraped/<date>/<domain>.md` by convention
   - Commit if data is reusable (mind copyright)

5. CHECK legality / ethics
   - Respect robots.txt unless explicitly authorized
   - No personal data without consent (GDPR)
   - No commercial paywall bypass

Concrete examples

Extract a lib's docs for RAG

firecrawl crawl https://docs.terraform.io/language \
  --limit 200 --scrape-options '{"formats":["markdown"]}' \
  --wait -o rag-corpus/terraform.json

Compare pricings of 5 competitors

firecrawl agent "Extract the pricing plans of each site" \
  --urls url1,url2,url3,url4,url5 \
  --schema-file pricing.schema.json \
  --max-credits 100 --wait -o pricing-compared.json

Monitor a changelog

firecrawl scrape https://example.com/changelog --format markdown -o new-changelog.md
if ! diff -q new-changelog.md last-changelog.md >/dev/null 2>&1; then
  diff last-changelog.md new-changelog.md   # what changed
  mv new-changelog.md last-changelog.md
fi

Red Flags — STOP immediately

Signal	Reaction
Missing `FIRECRAWL_API_KEY` AND firecrawl self-hosted not detected	Propose explicit fallback, ask the user for their choice
`robots.txt` forbids scraping the target path	STOP — ask for explicit authorization before continuing
More than 100 pages without confirmation	STOP — announce estimated costs and wait for validation
Personal data detected (email, phone, ID) in the output	STOP — do not save without GDPR legal basis
Site with login / commercial paywall	STOP — scraping illegal except with explicit contract
Repeated 429 rate limit	STOP — exponential backoff, do not hammer

Integration with the rest of the foundation

Combo	Usage
`web-scraping` -> `dev:dev-rag`	Build a corpus for RAG ingestion
`web-scraping` -> `biz:biz-competitor`	Factual competitive & market analysis on real data
`web-scraping` + `writing-skills`	Import third-party lib docs into a local skill
`qa-chrome` instead of `web-scraping`	Visual tests, DOM interaction, screenshots

Anti-patterns

NEVER scrape without checking robots.txt AND Terms of Service
NEVER commit scraped data without checking the rights
NEVER launch a crawl > 50 pages without user confirmation
NEVER use Firecrawl to replace WebSearch for a simple factual question (needlessly expensive)
NEVER bruteforce a site in massive parallel (max 4 workers by default)

Absolute rules

IMPORTANT: Always announce when degrading to a fallback (Playwright / curl) — the content may be partial.

IMPORTANT: Ask for confirmation before any crawl exceeding 50 pages or a site outside the user's control.

YOU MUST respect robots.txt and the target site's ToS.

YOU MUST save outputs in ./scraped/<date>/ with timestamp for traceability.

NEVER bypass an anti-bot system without documented legitimate justification.

Automatic triggering

This skill is automatically activated when:

The matching keywords are detected in the conversation
The task context matches the skill's domain

Triggering examples

"I want to web..."
"I want to scraping..."
"I want to extract data from ......"

Context fork

Fork means the skill runs as a sub-agent (context: fork):

Sees none of the conversation, only its own instructions
Returns its result to the conversation when done
Kept for self-contained jobs, reviewed case by case

Skill: web-scraping

Configuration

Detailed description

Web Scraping (Firecrawl-first)

Goal

When to trigger this skill

When NOT to use this skill

Prerequisites

Option 1: Firecrawl cloud (recommended)

Option 2: Firecrawl self-hosted

Option 3: Fallback without Firecrawl

The 5 Firecrawl operations (CLI `firecrawl-cli` v1.25)

1. Scrape (one URL)

2. Crawl (whole site)

3. Map (URL discovery)

4. Agent (structured data via LLM)

5. Search (search + extract in one pass)

Recommended workflow

Concrete examples

Extract a lib's docs for RAG

Compare pricings of 5 competitors

Monitor a changelog

Red Flags — STOP immediately

Integration with the rest of the foundation

Anti-patterns

Absolute rules

See also

Automatic triggering

Triggering examples

Context fork

See also

Configuration​

Detailed description​

Web Scraping (Firecrawl-first)

Goal​

When to trigger this skill​

When NOT to use this skill​

Prerequisites​

Option 1: Firecrawl cloud (recommended)​

Option 2: Firecrawl self-hosted​

Option 3: Fallback without Firecrawl​

The 5 Firecrawl operations (CLI firecrawl-cli v1.25)​

1. Scrape (one URL)​

2. Crawl (whole site)​

3. Map (URL discovery)​

4. Agent (structured data via LLM)​

5. Search (search + extract in one pass)​

Recommended workflow​

Concrete examples​

Extract a lib's docs for RAG​

Compare pricings of 5 competitors​

Monitor a changelog​

Red Flags — STOP immediately​

Integration with the rest of the foundation​

Anti-patterns​

Absolute rules​

See also​

Automatic triggering​

Triggering examples​

Context fork​

See also​

Configuration

Detailed description

Goal

When to trigger this skill

When NOT to use this skill

Prerequisites

Option 1: Firecrawl cloud (recommended)

Option 2: Firecrawl self-hosted

Option 3: Fallback without Firecrawl

The 5 Firecrawl operations (CLI `firecrawl-cli` v1.25)

1. Scrape (one URL)

2. Crawl (whole site)

3. Map (URL discovery)

4. Agent (structured data via LLM)

5. Search (search + extract in one pass)

Recommended workflow

Concrete examples

Extract a lib's docs for RAG

Compare pricings of 5 competitors

Monitor a changelog

Red Flags — STOP immediately

Integration with the rest of the foundation

Anti-patterns

Absolute rules

See also

Automatic triggering

Triggering examples

Context fork

See also