← Back to all projects

Scrapling

Scraping that survives redesigns: adaptive selectors, three grades of fetcher, and a Scrapy-style spider

BSD-3-Clause
Stars
90.5k
Forks
9.4k
Open issues
22
Last commit
29 Oct 2026

What is Scrapling?

You pick a fetcher per job — plain HTTP with TLS fingerprint impersonation when speed matters, a Playwright-driven browser for JavaScript pages, or a hardened stealth browser that patches headless tells and solves Cloudflare Turnstile — then select elements six ways, from CSS and XPath to text and structural similarity. Adaptive mode fingerprints matched elements so the same call finds them again after a site redesign, and the spider added in 0.4 crawls with deduplication, per-domain limits, pause-and-resume and JSON or CSV export. An MCP server lets a coding agent scrape conversationally, trimming pages by selector before they reach the model. It is a structured-extraction library for one machine: whole-site-to-markdown corpus building is Crawl4AI's territory, and the Turnstile-solving stealth sits in open tension with the README's own education-and-research disclaimer.

What can you do with Scrapling?

  • Three fetchers, one decision: speed or stealth — The plain fetcher does browser-like headers by default, TLS fingerprint impersonation and HTTP/3 at full speed; the dynamic one drives Chromium through Playwright and can attach to a real browser; the stealthy one adds headless-detection patches, canvas-fingerprint noise and automatic Turnstile solving — at the price of a browser's memory and pace. Sessions keep state and tab pools across requests.
  • Elements found again after the redesign — With adaptive and auto-save enabled, a matched element's tag, text, attributes, siblings and ancestor path are stored per domain; when the layout changes, the same call re-finds the element by similarity. Only the first match per selector is remembered — a stated limit worth knowing.
  • Six ways to say which element — CSS with text and attribute pseudo-elements, XPath, find-by-text, regular expressions, BeautifulSoup-style filters and find-similar for structurally alike elements — and it can generate a selector from any element it found, so a one-off discovery becomes a stable script.
  • A spider for when one page stops being enough — Scrapy-like async spiders bring start URLs, a priority queue with URL deduplication, per-domain concurrency limits, a robots.txt option, blocked-request detection with retries, checkpoint-based pause and resume, and JSON, JSONL, CSV or XML export — plus a proxy rotator and crawl templates.
  • Scraping as a conversation — The bundled MCP server exposes fetch and stealth-fetch tools with bulk variants, narrows pages by CSS selector before content reaches the model, and sanitises hidden DOM content against prompt injection; an interactive shell and terminal extract commands cover page-to-markdown without writing Python.

Before you choose Scrapling

  • Anti-bot bypass is the product — automatic Turnstile solving included — while the README limits use to education and research and defers to sites' terms; squaring those is your call to make before depending on it.
  • The API is still settling — 0.3 was a rewrite and 0.4 broke interfaces again — and the commit history is effectively one person's.

Star history

19 Aug to 30 Oct · +15.3k

75.2k90.5k

Frequently asked questions

Is Scrapling free for commercial use?

Scrapling is released under the BSD-3-Clause licence — OSI-approved open source, which permits commercial use.

How can Scrapling be deployed?

Scrapling is available as Runs locally.