Crawler.sh
Turn any website into clean, RAG-ready Markdown. Runs locally, renders JavaScript, respects robots.txt. No headless Chrome, no per-page fees.
Who it is for: Public pages not specified
Core capabilities
- Clean Markdown from any website. Without the cloud bill.: crawler.sh: Local Markdown Extractor for AI Training and RAG Crawler.sh Open main menu Product
- Markdown for AI: SEO analysis with CSV/TXT export
- RAG-ready Markdown from any page.: JSON and Sitemap XML export
- JavaScript Rendering: Crawl up to 10,000 pages per session
- SPAs rendered without headless Chrome.: NDJSON, JSON, Sitemap XML output formats
Pricing
Free · $0 · $99 / year · free
Updated 2026-08-17