
Scrapling
Scraping that survives redesigns: adaptive selectors, three grades of fetcher, and a Scrapy-style spider
What is Scrapling?
You pick a fetcher per job — plain HTTP with TLS fingerprint impersonation when speed matters, a Playwright-driven browser for JavaScript pages, or a hardened stealth browser that patches headless tells and solves Cloudflare Turnstile — then select elements six ways, from CSS and XPath to text and structural similarity. Adaptive mode fingerprints matched elements so the same call finds them again after a site redesign, and the spider added in 0.4 crawls with deduplication, per-domain limits, pause-and-resume and JSON or CSV export. An MCP server lets a coding agent scrape conversationally, trimming pages by selector before they reach the model. It is a structured-extraction library for one machine: whole-site-to-markdown corpus building is Crawl4AI's territory, and the Turnstile-solving stealth sits in open tension with the README's own education-and-research disclaimer.
What can you do with Scrapling?
- Three fetchers, one decision: speed or stealth — The plain fetcher does browser-like headers by default, TLS fingerprint impersonation and HTTP/3 at full speed; the dynamic one drives Chromium through Playwright and can attach to a real browser; the stealthy one adds headless-detection patches, canvas-fingerprint noise and automatic Turnstile solving — at the price of a browser's memory and pace. Sessions keep state and tab pools across requests.
- Elements found again after the redesign — With adaptive and auto-save enabled, a matched element's tag, text, attributes, siblings and ancestor path are stored per domain; when the layout changes, the same call re-finds the element by similarity. Only the first match per selector is remembered — a stated limit worth knowing.
- Six ways to say which element — CSS with text and attribute pseudo-elements, XPath, find-by-text, regular expressions, BeautifulSoup-style filters and find-similar for structurally alike elements — and it can generate a selector from any element it found, so a one-off discovery becomes a stable script.
- A spider for when one page stops being enough — Scrapy-like async spiders bring start URLs, a priority queue with URL deduplication, per-domain concurrency limits, a robots.txt option, blocked-request detection with retries, checkpoint-based pause and resume, and JSON, JSONL, CSV or XML export — plus a proxy rotator and crawl templates.
- Scraping as a conversation — The bundled MCP server exposes fetch and stealth-fetch tools with bulk variants, narrows pages by CSS selector before content reaches the model, and sanitises hidden DOM content against prompt injection; an interactive shell and terminal extract commands cover page-to-markdown without writing Python.
Before you choose Scrapling
- Anti-bot bypass is the product — automatic Turnstile solving included — while the README limits use to education and research and defers to sites' terms; squaring those is your call to make before depending on it.
- The API is still settling — 0.3 was a rewrite and 0.4 broke interfaces again — and the commit history is effectively one person's.
Star history
19 Aug to 30 Oct · +15.3k
Frequently asked questions
Is Scrapling free for commercial use?
Scrapling is released under the BSD-3-Clause licence — OSI-approved open source, which permits commercial use.
How can Scrapling be deployed?
Scrapling is available as Runs locally.