← Back to all projects

Crawl4AI

Async Python crawler that emits model-ready output

OfficialApache-2.0
Stars
86.6k
Forks
9k
Open issues
300
Last commit
30 Oct 2026

What is Crawl4AI?

A permissively licensed, self-hosted alternative for the same job as the hosted crawlers, with extraction strategies you can define per site. Being a library rather than a service means you own the proxy handling and rate limiting yourself.

What can you do with Crawl4AI?

  • Turn a page into Markdown — AsyncWebCrawler.arun() returns result.markdown, and passing PruningContentFilter or BM25ContentFilter to DefaultMarkdownGenerator also produces a fit_markdown with boilerplate stripped.
  • Crawl from the command line — The crwl command takes -o markdown for one page, --deep-crawl bfs --max-pages 10 to walk a site, and -q "Extract all product prices" for question-directed LLM extraction, with no Python written.
  • Extract JSON without an LLM — JsonCssExtractionStrategy takes a schema of baseSelector plus CSS/XPath fields and returns structured rows, with LLMExtractionStrategy and a Pydantic schema as the fallback when selectors are not enough.
  • Serve crawling over HTTP — The unclecode/crawl4ai Docker image exposes a FastAPI /crawl endpoint on port 11235 alongside a playground, monitoring dashboard and MCP integration, with auth on by default since v0.9.0.
  • Reuse a logged-in browser — BrowserConfig with user_data_dir and use_persistent_context=True keeps cookies and auth state across runs, while js_code, hooks and proxy settings handle pages that need interaction first.

Before you choose Crawl4AI

  • The HTTP server is not the library with a web interface in front. Twelve areas of the library are listed as unreachable through it, so check the approach you need works over HTTP before choosing the server.Reported in#1452

Star history

15 Aug to 30 Oct · +8.4k

78.2k86.6k

Frequently asked questions

Is Crawl4AI free for commercial use?

Crawl4AI is released under the Apache-2.0 licence — OSI-approved open source, which permits commercial use.

How can Crawl4AI be deployed?

Crawl4AI is available as Self-hosted / Runs locally.