crawl4ai
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown.
free(Free tier)84.8k github stars
About crawl4ai
Latest: v0.9.4 (23 Sep 2026) · all releases →
Crawl4AI turns any website into clean, LLM-ready Markdown for RAG, AI agents and data pipelines. Run the open-source web crawler and scraper yourself, free forever, or use it hosted with one key: scrape, search and extract through one API, with MCP for your agent.
Key Features
- 🧹 Clean Markdown: headings, lists, tables and code blocks, in a structure an LLM reads well.
- 🎯 Fit Markdown: filters remove menus, footers and boilerplate: PruningContentFilterLXML, BM25ContentFilter (for a query) and LLMContentFilter.
- Citations: page links become a numbered reference list.
- ️ Your own strategy: plug in a custom Markdown generator.