anakin
Open-source web scraping API that converts websites into LLM-ready markdown or structured JSON.
anakin 소개
Anakin is a self-hosted, open-source web scraping API built in Go. It enables developers to extract clean, structured data from websites with minimal setup — supporting both synchronous and asynchronous scraping, batch operations, and AI-powered content extraction.
It uses Camoufox (an anti-detect Firefox fork) for realistic browser fingerprints and includes proxy auto-selection powered by Thompson Sampling to improve success rates across domains. The tool is designed for integration into RAG pipelines, AI agents, and automated data workflows.
The system can run zero-config with just Go or via Docker for full-stack operation, including a built-in React dashboard for job monitoring, domain configuration, and proxy management.
주요 기능
- Domain-specific scraping strategies with configurable handlers, timeouts, retries, custom headers, domain blocking, and content validation via pattern matching.
- Anti-detect browser using Camoufox for realistic fingerprinting, avoiding detection better than headless Chrome.
- Proxy auto-selection using Thompson Sampling to dynamically choose the best proxy per domain based on real-time success/failure feedback.
- Structured JSON extraction powered by Gemini AI (bring your own API key) for semantic data parsing from any webpage.
- Flexible API modes: sync (/v1/scrape), async with polling (/v1/url-scraper), and batch scraping up to 10 URLs.
- LLM-ready markdown output with automatic boilerplate removal and clean content extraction, suitable for direct ingestion into RAG or LLMs.
- Built-in web dashboard (React-based) for scraping jobs, domain config management, and proxy health monitoring.
- Zero-config mode — runs with only Go installed; optional Docker deployment for full stack including database and UI.
장점 & 단점
장점
- • Fully self-hosted and open-source with no cloud dependencies or vendor lock-in.
- • Designed specifically for AI/LLM use cases like RAG and agent tooling.
- • Real anti-detect capabilities via Camoufox, not just headless browser masking.
- • Intelligent proxy selection adapts over time without manual tuning.
단점
- • Requires bringing your own Gemini API key for structured JSON extraction — no built-in AI model or free tier.
- • No official documentation links or hosted demo provided in README — users must rely on code and minimal inline docs.
- • Dashboard and full-stack features require Docker; zero-config mode lacks persistence and UI.