← Back to all projects

Crawl4AI
Async Python crawler that emits model-ready output
OfficialApache-2.0
What is Crawl4AI?
A permissively licensed, self-hosted alternative for the same job as the hosted crawlers, with extraction strategies you can define per site. Being a library rather than a service means you own the proxy handling and rate limiting yourself.
What can you do with Crawl4AI?
- Turn a page into Markdown — AsyncWebCrawler.arun() returns result.markdown, and passing PruningContentFilter or BM25ContentFilter to DefaultMarkdownGenerator also produces a fit_markdown with boilerplate stripped.
- Crawl from the command line — The crwl command takes -o markdown for one page, --deep-crawl bfs --max-pages 10 to walk a site, and -q "Extract all product prices" for question-directed LLM extraction, with no Python written.
- Extract JSON without an LLM — JsonCssExtractionStrategy takes a schema of baseSelector plus CSS/XPath fields and returns structured rows, with LLMExtractionStrategy and a Pydantic schema as the fallback when selectors are not enough.
- Serve crawling over HTTP — The unclecode/crawl4ai Docker image exposes a FastAPI /crawl endpoint on port 11235 alongside a playground, monitoring dashboard and MCP integration, with auth on by default since v0.9.0.
- Reuse a logged-in browser — BrowserConfig with user_data_dir and use_persistent_context=True keeps cookies and auth state across runs, while js_code, hooks and proxy settings handle pages that need interaction first.
Before you choose Crawl4AI
- The HTTP server is not the library with a web interface in front. Twelve areas of the library are listed as unreachable through it, so check the approach you need works over HTTP before choosing the server.Reported in#1452
Star history
15 Aug to 30 Oct · +8.4k
78.2k86.6k
Frequently asked questions
Is Crawl4AI free for commercial use?
Crawl4AI is released under the Apache-2.0 licence — OSI-approved open source, which permits commercial use.
How can Crawl4AI be deployed?
Crawl4AI is available as Self-hosted / Runs locally.