# Web search, crawling and scraping

How an agent gets information from the web: search APIs that return results made for models, crawlers and scrapers that turn pages into clean text or structured data, and readers that fetch one page as Markdown.

57 tools, in alphabetical order. Nothing is ranked by popularity.

- [AgentQL](https://modelmoot.com/tools/agentql.md): A query language and SDK for locating elements and pulling structured data from web pages with a model. (TinyFish; not open source, hosted service)
- [Apify](https://modelmoot.com/tools/apify.md): A platform for running web scrapers and automation programs called Actors, with a store of ready-made ones. (Apify; not open source, hosted service)
- [Apify MCP server](https://modelmoot.com/tools/apify-mcp.md): Apify's MCP server that exposes Apify Actors, scrapers and automations as tools. (Apify; open source, hosted service)
- [Brave Search API](https://modelmoot.com/tools/brave-search-api.md): A search API backed by the independent Brave web index, with endpoints for web, news, image and video results. (Brave; not open source, hosted service)
- [Brave Search MCP Server](https://modelmoot.com/tools/brave-search.md): Brave's MCP server for web, local, image, video and news search and summaries via the Brave Search API. (Brave; open source)
- [Bright Data](https://modelmoot.com/tools/bright-data.md): A web data platform with proxies, scraping APIs, a search API and a scraping browser, plus an MCP server. (Bright Data; not open source, hosted service)
- [Bright Data MCP](https://modelmoot.com/tools/bright-data-mcp.md): Bright Data's MCP server for web search, scraping and browser access through its unblocking network. (Bright Data; open source, hosted service)
- [Claude web search tool](https://modelmoot.com/tools/claude-web-search.md): A server-side tool in the Claude API that searches the web and cites results in the reply. (Anthropic; not open source, hosted service)
- [Context7](https://modelmoot.com/tools/context7.md): MCP server that gives coding agents current documentation and code examples for libraries. (Upstash; open source, hosted service)
- [Crawl4AI](https://modelmoot.com/tools/crawl4ai.md): An open-source Python crawler that turns web pages into Markdown for models, running locally or in Docker. (Crawl4AI; open source)
- [Crawlee](https://modelmoot.com/tools/crawlee.md): A Node.js and TypeScript library for building scrapers and crawlers with Playwright, Puppeteer or plain HTTP. (Apify; open source)
- [Diffbot](https://modelmoot.com/tools/diffbot.md): A service that extracts structured data from web pages and offers a large knowledge graph built from the web. (Diffbot; not open source, hosted service)
- [Exa](https://modelmoot.com/tools/exa.md): A search API over its own web index that supports meaning-based queries and returns page contents for models. (Exa; not open source, hosted service)
- [Exa MCP server](https://modelmoot.com/tools/exa-mcp.md): Exa's MCP server for web search, page fetching and code search through the Exa API. (Exa; open source, hosted service)
- [Fetch MCP server](https://modelmoot.com/tools/fetch.md): Reference MCP server that fetches a web page and returns its content as Markdown for a model to read. (Model Context Protocol project; open source)
- [Firecrawl](https://modelmoot.com/tools/firecrawl.md): An API and open-source project that scrapes, crawls and searches the web and returns pages as Markdown or structured data. (Firecrawl; open source, hosted service)
- [Firecrawl MCP server](https://modelmoot.com/tools/firecrawl-mcp.md): Firecrawl's MCP server for scraping, crawling, searching and extracting data from web pages. (Firecrawl; open source, hosted service)
- [GPT Researcher](https://modelmoot.com/tools/gpt-researcher.md): An open-source research agent that searches many web sources and writes a cited report. (Assaf Elovic; open source)
- [Grounding with Google Search](https://modelmoot.com/tools/google-search-grounding.md): A tool in the Gemini API that lets the model run Google searches and cite the pages it used. (Google; not open source, hosted service)
- [Hyperbrowser](https://modelmoot.com/tools/hyperbrowser.md): A hosted service that provides cloud browser sessions and scraping endpoints for agents, with Python and Node SDKs. (Hyperbrowser; not open source, hosted service)
- [Jina AI](https://modelmoot.com/tools/jina-ai.md): Hosted embedding, reranking and reader APIs, plus small open models for search over text, images and long documents. (Jina AI; not open source, hosted service)
- [Jina AI MCP server](https://modelmoot.com/tools/jina.md): Jina AI's remote MCP server for reading and searching the web, with embeddings and reranking tools. (Jina AI; open source, hosted service)
- [Jina Reader](https://modelmoot.com/tools/jina-reader.md): A service that turns a web page address into clean text for models, and also offers a search endpoint. (Jina AI; open source, hosted service)
- [Kagi API](https://modelmoot.com/tools/kagi-api.md): APIs from the Kagi search company, including search, summarising and enrichment endpoints. (Kagi; not open source, hosted service)
- [Kagi MCP server](https://modelmoot.com/tools/kagi.md): Kagi's MCP server for web, news and image search and page extraction with a Kagi API key. (Kagi; open source, hosted service)
- [Linkup](https://modelmoot.com/tools/linkup.md): A search API for agents that returns sourced answers or raw results from the web and partner content. (Linkup; not open source, hosted service)
- [Maxun](https://modelmoot.com/tools/maxun.md): A no-code platform for training robots that scrape data from sites by recording actions, with an API and self-hosting. (Maxun; open source, hosted service)
- [Microsoft Learn MCP Server](https://modelmoot.com/tools/microsoft-learn.md): Microsoft's remote MCP server for searching and fetching official Microsoft Learn documentation. (Microsoft; not open source, hosted service)
- [Olostep](https://modelmoot.com/tools/olostep.md): A web data API for scraping, crawling, searching and mapping sites, returning Markdown or structured data. (Olostep; not open source, hosted service)
- [OpenAI web search tool](https://modelmoot.com/tools/openai-web-search.md): A built-in tool in the OpenAI Responses API that searches the web and returns answers with citations. (OpenAI; not open source, hosted service)
- [Oxylabs Web Scraper API](https://modelmoot.com/tools/oxylabs.md): A scraping API and proxy network, with an MCP server that exposes page fetching and search to models. (Oxylabs; not open source, hosted service)
- [Parallel](https://modelmoot.com/tools/parallel.md): A set of web APIs for agents covering search, page extraction and longer research tasks. (Parallel Web Systems; not open source, hosted service)
- [Perplexity API](https://modelmoot.com/tools/perplexity-api.md): APIs from Perplexity that return web search results and answers with citations, plus an MCP server. (Perplexity; not open source, hosted service)
- [Perplexity MCP Server](https://modelmoot.com/tools/perplexity.md): Perplexity's MCP server for web search and answers through the Perplexity API. (Perplexity; open source, hosted service)
- [Readability.js](https://modelmoot.com/tools/readability.md): The JavaScript library behind Firefox Reader View, used by many tools to pull article text out of a page. (Mozilla; open source)
- [Repomix](https://modelmoot.com/tools/repomix.md): A command line tool that packs a code repository, local or remote, into one file formatted for a model. (Yamadashy; open source)
- [ScrapeGraphAI](https://modelmoot.com/tools/scrapegraphai.md): A Python library and hosted API that uses a language model to extract structured data from web pages from a prompt. (ScrapeGraphAI; open source, hosted service)
- [Scrapfly](https://modelmoot.com/tools/scrapfly.md): A web scraping API with anti-bot handling, browser rendering and an extraction API that can use models. (Scrapfly; not open source, hosted service)
- [ScrapingBee](https://modelmoot.com/tools/scrapingbee.md): A scraping API that fetches pages with headless browsers and rotating proxies and offers extraction rules. (ScrapingBee; not open source, hosted service)
- [Scrapingdog](https://modelmoot.com/tools/scrapingdog.md): A scraping API with endpoints for general pages, Google results and several large sites. (Scrapingdog; not open source, hosted service)
- [Scrapling](https://modelmoot.com/tools/scrapling.md): A Python scraping library with adaptive element finding, stealth fetchers and a spider framework, including an MCP server. (Scrapling; open source)
- [Scrapy](https://modelmoot.com/tools/scrapy.md): A Python framework for writing crawlers that fetch pages and extract data into structured items. (Zyte and contributors; open source)
- [SearchApi](https://modelmoot.com/tools/searchapi.md): An API that returns structured results from Google, Bing and other search engines and platforms. (SearchApi; not open source, hosted service)
- [SearXNG](https://modelmoot.com/tools/searxng.md): A self-hosted metasearch engine that queries many search engines and offers a JSON output used by agent tools. (SearXNG contributors; open source)
- [SerpApi](https://modelmoot.com/tools/serpapi.md): An API that returns structured results from Google and many other search engines and marketplaces. (SerpApi; not open source, hosted service)
- [Serper](https://modelmoot.com/tools/serper.md): A low-latency API that returns Google search results as JSON, used as a search tool by agent frameworks. (Serper; not open source, hosted service)
- [Spider](https://modelmoot.com/tools/spider.md): A crawler written in Rust with a hosted API that returns pages as Markdown or text for models. (Spider; open source, hosted service)
- [Tavily](https://modelmoot.com/tools/tavily.md): A web search and content extraction API built for language models and agents. (Tavily; not open source, hosted service)
- [Tavily MCP server](https://modelmoot.com/tools/tavily-mcp.md): Tavily's MCP server for web search, page extraction, site mapping and crawling. (Tavily; open source, hosted service)
- [TinyFish](https://modelmoot.com/tools/tinyfish.md): A web agent platform that runs browser-based agents and returns data from sites that require interaction. (TinyFish; not open source, hosted service)
- [Trafilatura](https://modelmoot.com/tools/trafilatura.md): A Python library and command line tool that extracts the main text and metadata from web pages and can crawl sites. (Adrien Barbaresi; open source)
- [Valyu](https://modelmoot.com/tools/valyu.md): A search API that covers the web and licensed sources such as research papers and financial data, for models. (Valyu; not open source, hosted service)
- [Web search tool in Azure AI Foundry](https://modelmoot.com/tools/azure-web-search.md): A tool for Azure AI Foundry agents that grounds answers in live web results with citations. (Microsoft; not open source, hosted service)
- [xAI search tools](https://modelmoot.com/tools/grok-search-tools.md): Server-side tools in the xAI API that let Grok search the web and posts on X during a request. (xAI; not open source, hosted service)
- [You.com APIs](https://modelmoot.com/tools/you-com-api.md): Web search, content and research APIs from You.com for use by agents and applications. (You.com; not open source, hosted service)
- [ZenRows](https://modelmoot.com/tools/zenrows.md): A scraping API and scraping browser service that handle blocking and return page content. (ZenRows; not open source, hosted service)
- [Zyte API](https://modelmoot.com/tools/zyte.md): A web data service with a fetch API that handles blocking and a Scrapy integration, plus automatic data extraction. (Zyte; not open source, hosted service)

HTML: https://modelmoot.com/tools/c/web-data. All categories: https://modelmoot.com/tools.md
