Stack hub
Crawlers optimized for clean markdown and structured extracts - built for AI and RAG pipelines, not classic unblock APIs alone.
12 tools · AI crawlers
Scraping API
Managed scraping API that returns page HTML or structured data while handling proxies, JavaScript rendering, and CAPTCHA challenges for you. Credits scale with render mode and hard domains such as Amazon, Google, or LinkedIn, so you can model unit cost before you buy volume.
Scraping API
Headless-browser scraping API for JS-heavy public pages, billed in credits across classic, premium, and stealth modes. Free credits let you smoke-test rendering and unblock behavior before you commit to a monthly bucket.
Scraping API
Universal Scraper API with transparent credit multipliers for JavaScript rendering and premium proxies. Includes a short trial so you can validate unblock modes before buying Developer, Startup, or Business buckets.
AI crawler
AI-oriented crawl and scrape API that returns LLM-ready markdown, HTML, or structured extracts from sites and sitemaps. Strong fit when your pipeline feeds agents or RAG rather than classic HTML parsing alone.
Reviewed catalog tools for this scrape category: pricing, capabilities, and vendor links.
Scraping API
Managed scraping API that returns page HTML or structured data while handling proxies, JavaScript rendering, and CAPTCHA challenges for you. Credits scale with render mode and hard domains such as Amazon, Google, or LinkedIn, so you can model unit cost before you buy volume.
Scraping API
Headless-browser scraping API for JS-heavy public pages, billed in credits across classic, premium, and stealth modes. Free credits let you smoke-test rendering and unblock behavior before you commit to a monthly bucket.
Scraping API
Universal Scraper API with transparent credit multipliers for JavaScript rendering and premium proxies. Includes a short trial so you can validate unblock modes before buying Developer, Startup, or Business buckets.
AI crawler
AI-oriented crawl and scrape API that returns LLM-ready markdown, HTML, or structured extracts from sites and sitemaps. Strong fit when your pipeline feeds agents or RAG rather than classic HTML parsing alone.
Scraping API
Crawling API billed per successful request with rates by domain complexity and regular vs JavaScript paths, plus Smart AI Proxy monthly credit plans for proxy-style integration.
Scraping API
Scraping API with monthly successful-credit plans where core features (proxies, JS rendering, geo-targeting, browser play) are unlocked on every tier. Free plan includes 1,000 credits and 5 concurrent requests with no card.
Scraping API
Web Scraper API sold in credit packs with a free trial, plus adjacent SERP, Unlocker, Scraping Browser, and proxy products on the same platform. Structured JSON, CSV, and XLSX outputs are part of the public positioning.
Scraping API
Scraper API with unlimited monthly plans priced by concurrent threads, or PAYG packs from roughly $0.13 to $0.35 per 1,000 requests. Free tier includes 1,500 requests per month plus a short unlimited trial.
Scraping API
Web Scraper API with Basic (datacenter) and Pro (residential + JavaScript) monthly request packages. Data Platform no-code access is included on Pro from Starter Project upward.
Scraping API
Modular Web Scraping API: pick Standard or Premium proxies and toggle JavaScript rendering per request, then pay per 1,000 successful requests on Free through Business monthly plans. Templates and MCP support are part of the product story.
Scraping API
Swiss-hosted Scraper API with PAYG or monthly credit plans, billed on successful results. Supports request vs browser modes, residential geotargeting, and AI enhancement options for structured output.
Scraping API
Flat monthly Scraping API priced by concurrent threads with unlimited requests inside the tier. JavaScript rendering, rotating proxies, and anti-bot handling are included without credit multipliers.
Choosing an AI crawler means optimizing for LLM-ready content, not only unblock success. These tools emphasize crawl + extract into markdown or JSON for retrieval systems and agents. Firecrawl is a common shortlist name; verify credit costs for scrape vs crawl vs JSON extract - those modes are not priced the same.
Pick an AI crawler when your output target is clean content for RAG or agents. Pick a scraping API when you need raw HTML and classic unblock mechanics for custom parsers - compare ScraperAPI or ScrapingBee. Overlap exists, but optimizing for the wrong output format wastes budget and engineering time.
For RAG, evaluate crawl depth controls, sitemap/URL list ingestion, noise removal quality, and structured extract options. Cheap crawls that leave boilerplate and nav chrome can hurt retrieval more than they save money. Use free tiers to test a representative site set before you scale corpus jobs.
Discovery and page fetch are different jobs. If rankings are the entry point, start on the Google SERP hub with tools like SerpAPI, then crawl the URLs you are authorized to ingest. For hard public pages that still need unblock APIs, keep a scraping-API failover from the scraping APIs hub.
ScraperDB lists AI crawlers beside scraping APIs so you can compare categories honestly. We do not invent crawl success-rate benchmarks - re-check vendor pricing before you scale.
Only collect public data you are authorized to access. Follow each platform’s terms, robots rules, and applicable law. ScraperDB does not publish bypass or evasion guides.
A practical shortlist process before you commit budget.
Markdown for chunks, JSON for fields, or both. Your RAG schema should drive the tool choice before you compare credit menus.
Single pages, site sections, or full domains - depth and allowlists change credit burn dramatically.
JSON extract often costs more than basic scrape/crawl. Budget the mode you will use in production, not the demo mode.
Inspect chunk cleanliness in your vector store. Boilerplate-heavy markdown is a false economy.
Match the job to the category before comparing brand names.
You want markdown/structured JSON for LLM, RAG, and agent pipelines.
You need HTML for custom parsers or classic unblock workflows.
Agents need interactive browser sessions, not only crawled text.
Discovery starts from search rankings rather than known site URLs.
Scraping APIs optimize for page fetch/unblock. AI crawlers optimize for clean content into LLM pipelines. Overlap exists - pick by output format.
Firecrawl is commonly shortlisted for markdown and structured extracts aimed at LLM/RAG pipelines. Verify current credit costs for scrape vs JSON extract on the vendor site.
Often one credit per scraped/crawled page, with extra credits for JSON extract modes. Always confirm on the provider pricing page - and pilot via free tiers when available.
No. SERP APIs such as SerpAPI return structured search results. Crawlers ingest page content from URLs you already have or discover separately via the Google hub.
A representative site set, your real extract schema, and a retrieval quality check in your vector store - plus cost after extract multipliers.
