A web crawlers list is a reference of the bots that visit your site, with the user agent each one sends and whether to allow or block it. The bots that matter most in 2026 are Googlebot and Bingbot for search, plus GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot and PerplexityBot for AI.
Bots are now the majority of traffic. Imperva’s 2025 Bad Bot Report found that automated traffic reached 51% of all web traffic, the first time in a decade it passed human activity. Search engines, AI platforms and social networks all send crawlers to read your pages, and most WordPress owners have never looked at them in their server logs.
This list covers 27 crawlers, grouped by what they do, with allow or block advice and copy-paste robots.txt rules. We rechecked each user agent against the vendor’s own documentation page on 7 October 2026 and link the source in each entry. Where a vendor publishes no page we could load, we say so instead of guessing.
Last updated 7 October 2026
Web Crawlers List At A Glance: Training, Search And Live Fetch
Most AI vendors now run separate bots for separate jobs, and you can control each one on its own in robots.txt. This table shows which bot does what, so you can block training without disappearing from AI search.
| Company | Training or opt-out | Search index | Live fetch for a user or answer |
|---|---|---|---|
Google-Extended (control token) | Googlebot | User-triggered fetchers (various) | |
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | None listed (not used for foundation models) | PerplexityBot | Perplexity-User |
| Meta | Meta-ExternalAgent | Meta-WebIndexer | Meta-ExternalFetcher |
| Apple | Applebot-Extended (control token) | Applebot | None listed |
| DuckDuckGo | None (not used for training) | DuckDuckBot | DuckAssistBot |
| Microsoft | None listed | bingbot | None listed |
| Common Crawl | CCBot (open dataset) | None | None |
For a deeper look at the OpenAI trio, see our guides on GPTBot, OAI-SearchBot and ChatGPT-User. This page stays focused on the full list.
What Are Web Crawlers?
A web crawler (also called a web spider, bot, or robot) is an automated program that systematically browses the internet to discover, read, and index web pages. Crawlers follow links from one page to another, building a map of the web that search engines and other services use to organize information.
Googlebot is the most well-known web crawler, responsible for indexing billions of pages for Google Search. But dozens of other crawlers operate across the web, each serving a different purpose.
Also Read: wondering whether to allow ClaudeBot on your site? We break down what Anthropic’s crawler does.
How Do Web Crawlers Work?
A web crawler starts with a seed list of URLs. It visits each URL, reads the page content and HTML structure, extracts all the links it finds, and adds those new URLs to its queue. The crawler then repeats this process continuously.
As it works, the crawler stores the page data in an index, a structured database that allows the parent service (such as Google Search) to quickly retrieve relevant results. Crawlers follow rules set in your site’s robots.txt file and respect meta directives like noindex and nofollow to determine which pages to skip.
Most crawlers also limit their request rate to avoid overloading your server. If you want the full picture of how that budget works, read our guide to crawl budget in WordPress.
Types Of Web Crawlers
Understanding the different types of web crawlers helps you decide which ones to allow or block on your site.
General-purpose crawlers index the entire web for search engines. Googlebot, Bingbot and YandexBot fall into this category. They aim to discover and catalog as much content as possible.
AI training crawlers collect data used to train large language models. GPTBot, ClaudeBot and Meta-ExternalAgent are examples. They are newer and more controversial because they use content for model training rather than search indexing.
AI search and live-fetch bots index pages for AI answer engines or fetch a page when a person asks. OAI-SearchBot, PerplexityBot, ChatGPT-User and Claude-User are examples. Blocking them can remove you from AI answers.
Social media crawlers fetch page metadata (title, description, image) when someone shares a link on platforms like Facebook or LinkedIn. They generate the link preview cards you see in social feeds.
SEO and analytics crawlers scan websites to collect data for marketing and SEO tools. SemrushBot and AhrefsBot gather backlink data, keyword rankings, and site health metrics.
Looking for ways to protect your website from hackers? Check out the 5 Best WordPress Security Plugins to Protect Your Site
27 Most Common Web Crawlers In 2026
Here is the complete bot crawler list for 2026, grouped by purpose. Each entry gives the user agent, the owner, what the bot is for, our allow or block advice, and the vendor page we checked.
Search Engine Crawlers
1. Googlebot
Googlebot is Google’s main crawler for Search. Google’s documentation lists Googlebot Smartphone and Desktop variants, plus the Googlebot-Image, Googlebot-Video and Googlebot-News tokens for specific products.
- User-agent:
Googlebot/2.1(for exampleMozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)),Googlebot-Image/1.0 - Owner: Google
- Purpose: Crawling for Google Search, Discover, Images, Video and News
- Allow or block? Allow. Blocking Googlebot removes your pages from Google Search.
- Key detail: For Search, Googlebot crawls the first 2MB of a supported file type and the first 64MB of a PDF. Verify a visit with reverse DNS, covered below.
- Source: Google common crawlers and Googlebot documentation
2. Bingbot
Bingbot is Microsoft’s crawler for Bing Search. Bing results also feed Microsoft Copilot, which is why Bing visibility now affects AI citations too. See our explainer on getting cited in Copilot.
- User-agent:
bingbot - Owner: Microsoft
- Purpose: Crawling for Bing Search
- Allow or block? Allow.
- Not re-verified: Bing’s crawler page is a script-rendered help site that did not load for us on 7 October 2026, so this entry carries the long-standing token only.
3. YandexBot
YandexBot is the main indexing robot of Yandex. Yandex runs many other robots as well, such as YandexImages/3.0 and YandexVideo/3.0, plus service bots for its Metrika and Webmaster tools.
- User-agent:
YandexBot/3.0 - Owner: Yandex
- Purpose: Indexing pages for Yandex Search
- Allow or block? Allow if you serve audiences where Yandex is used. Otherwise it is optional.
- Key detail: Yandex says its genuine robots come from hosts ending in
yandex.ru,yandex.netoryandex.com. - Source: Yandex robot documentation
4. DuckDuckBot
DuckDuckBot is the web crawler for DuckDuckGo, the privacy-focused search engine.
- User-agent:
DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html) - Owner: DuckDuckGo
- Purpose: Improving DuckDuckGo search results
- Allow or block? Allow.
- Source: DuckDuckGo help: About DuckDuckBot
5. Applebot
Applebot is Apple’s web crawler. Its data powers search features in Spotlight, Siri and Safari, and Apple says it may also help train Apple foundation models. Publishers opt out of that training use with the separate Applebot-Extended robots.txt token.
- User-agent:
Applebot/0.1(for exampleMozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)) - Owner: Apple
- Purpose: Search features across Apple products, plus optional AI training
- Allow or block? Allow Applebot for Apple search visibility. Block
Applebot-Extendedif you do not want training use. - Key detail: If robots.txt does not mention Applebot but mentions Googlebot, Applebot follows the Googlebot rules. It does not follow crawl-delay.
- Source: About Applebot
robots.txt rule:
User-agent: Applebot-Extended Disallow: /
6. Baiduspider
Baiduspider is the crawler for Baidu, the main search engine in mainland China. It matters if you target Chinese-speaking audiences.
- User-agent:
Baiduspider - Owner: Baidu
- Purpose: Indexing pages for Baidu Search
- Allow or block? Allow if you target mainland China. Otherwise it is optional.
- Not re-verified: Baidu’s English robot page did not load for us on 7 October 2026, so we list the token only and make no version claims.
7. Slurp (Yahoo)
Slurp is the Yahoo Search robot. Yahoo says some of its results come from partners, but sites should allow Slurp to appear in Yahoo Mobile Search results. Older guides call Slurp retired, and Yahoo’s own page says otherwise.
- User-agent:
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) - Owner: Yahoo
- Purpose: Crawling for Yahoo Search and content for Yahoo News, Finance and Sports
- Allow or block? Allow. It obeys robots.txt, including crawl-delay.
- Source: Yahoo help: Why is Slurp crawling my page?
AI Training And Opt-Out Crawlers
These bots collect content for model training, or act as the token you use to opt out of it. Blocking them does not change your search rankings.
8. GPTBot (OpenAI)
GPTBot crawls content that may be used to train OpenAI’s generative AI foundation models. OpenAI says disallowing it signals that your content should not be used for training. This entry is deliberately short because we cover it in full in our GPTBot guide for WordPress.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
- Owner: OpenAI
- Purpose: Training data for generative AI models
- Allow or block? Block if you do not want your content used for training. This does not block OAI-SearchBot or ChatGPT-User.
- Published IPs: openai.com/gptbot.json
- Source: OpenAI crawler overview
robots.txt rule:
User-agent: GPTBot Disallow: /
9. ClaudeBot (Anthropic)
ClaudeBot collects web content that could contribute to the training of Anthropic’s Claude models. When a site restricts it, Anthropic says the site’s future material is excluded from training datasets. Our ClaudeBot guide covers the allow-or-block decision.
- Owner: Anthropic
- Purpose: Training data for Claude models
- Allow or block? Your call. Block it to opt out of training. Anthropic says its bots honor robots.txt and support the non-standard
Crawl-delaydirective. - Key detail: Anthropic’s page lists the bot name but no full user-agent string, so match on
ClaudeBot. Anthropic warns that blocking by IP address may not work reliably and recommends robots.txt. - Published IPs: claude.com/crawling/bots.json
- Source: Anthropic: does Anthropic crawl data from the web? (updated 8 April 2026)
robots.txt rule:
User-agent: ClaudeBot Disallow: /
10. Google-Extended
Google-Extended is a robots.txt control token, not a separate crawler. Google says it lets publishers manage whether crawled content may be used to train future Gemini models and to ground answers in Gemini Apps and Vertex AI.
- robots.txt token:
Google-Extended(no separate user-agent string) - Owner: Google
- Purpose: Opt-out control for Gemini training and grounding
- Allow or block? Block if you do not want that use. Google states it does not affect inclusion in Google Search or act as a ranking signal.
- Key detail: Real fetches still arrive with Google’s normal user agents, so you cannot find Google-Extended in your logs. For how Google’s AI features treat your content, see our AI Overviews and AI Mode guide.
- Source: Google common crawlers
robots.txt rule:
User-agent: Google-Extended Disallow: /
11. Meta-ExternalAgent (Meta)
Meta-ExternalAgent crawls the web for uses such as training foundation AI models and indexing content directly for Meta products.
- User-agent:
meta-externalagent/1.1 - Owner: Meta
- Purpose: AI model training and direct content indexing
- Allow or block? Block if you do not want Meta training on your content. It does not affect link previews.
- Source: Meta web crawlers documentation
robots.txt rule:
User-agent: meta-externalagent Disallow: /
12. Bytespider (ByteDance)
Bytespider is the crawler associated with ByteDance, the TikTok parent company. Most Western sites block it, and it has a reputation for aggressive crawling.
- User-agent:
Bytespider - Owner: ByteDance
- Purpose: Data collection for ByteDance AI products
- Allow or block? Most Western sites block it. Allow it only if you want exposure in Chinese-language AI products.
- Not re-verified: We found no official ByteDance documentation page to load on 7 October 2026, so this entry has no version or user-agent string claim.
robots.txt rule:
User-agent: Bytespider Disallow: /
13. CCBot (Common Crawl)
CCBot is the crawler of Common Crawl, a non-profit that publishes an open dataset of web crawl data. Many AI training sets are built from that dataset, so blocking CCBot is a common opt-out step.
- User-agent:
CCBot/2.0 (https://commoncrawl.org/faq/) - Owner: Common Crawl
- Purpose: Building an open web crawl archive
- Allow or block? Block if you do not want your pages in open datasets that others train on.
- Key detail: Common Crawl warns that other crawlers falsely identify as CCBot, and says the real CCBot runs on dedicated IP ranges with reverse DNS.
- Source: Common Crawl CCBot page
robots.txt rule:
User-agent: CCBot Disallow: /
AI Search And Answer-Engine Crawlers
These bots index pages so AI products can cite and link to you. Blocking them can remove you from AI answers, so most sites allow them. Our guides on ChatGPT Search and Perplexity citations show what that visibility is worth.
14. OAI-SearchBot (OpenAI)
OAI-SearchBot surfaces websites in ChatGPT’s search features. OpenAI says sites that opt out will not be shown in ChatGPT search answers, though they can still appear as navigational links. OpenAI lets you allow it while disallowing GPTBot. Full detail is in our OAI-SearchBot guide.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
- Owner: OpenAI
- Purpose: Search results inside ChatGPT, not model training
- Allow or block? Allow if you want ChatGPT search visibility. OpenAI says robots.txt changes can take about 24 hours to apply.
- Published IPs: openai.com/searchbot.json
- Source: OpenAI crawler overview
robots.txt rule to allow:
User-agent: OAI-SearchBot Allow: /
15. Claude-SearchBot (Anthropic)
Claude-SearchBot navigates the web to improve the quality of search results for Claude users. Anthropic says disabling it may reduce your visibility and accuracy in those results.
- Owner: Anthropic
- Purpose: Search result quality for Claude
- Allow or block? Allow if you want to appear in Claude search answers.
- Source: Anthropic crawler page
robots.txt rule to allow:
User-agent: Claude-SearchBot Allow: /
16. PerplexityBot (Perplexity)
PerplexityBot is designed to surface and link websites in Perplexity search results. Perplexity states it is not used to crawl content for AI foundation models.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
- Owner: Perplexity AI
- Purpose: Search results in Perplexity
- Allow or block? Allow if you want Perplexity citations. Perplexity says changes can take up to 24 hours to apply.
- Published IPs: perplexity.com/perplexitybot.json
- Source: Perplexity bots documentation
robots.txt rule to allow:
User-agent: PerplexityBot Allow: /
17. Meta-WebIndexer (Meta)
Meta-WebIndexer navigates the web to improve Meta AI search results. Meta says allowing it helps Meta AI cite and link to your content.
- User-agent:
meta-webindexer/1.1 - Owner: Meta
- Purpose: Meta AI search quality
- Allow or block? Allow if you want links from Meta AI answers.
- Source: Meta web crawlers documentation
robots.txt rule to allow:
User-agent: meta-webindexer Allow: /
18. DuckAssistBot (DuckDuckGo)
DuckAssistBot crawls pages in real time for DuckDuckGo’s AI-assisted answers, which cite their sources. DuckDuckGo says the data is not used to train AI models.
- User-agent:
DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html) - Owner: DuckDuckGo
- Purpose: Source pages for AI-assisted answers
- Allow or block? Allow if you want source citations. A block takes about 72 hours to take effect, and DuckDuckGo says opting out does not affect organic search.
- Published IPs: duckduckgo.com/duckassistbot.json
- Source: DuckDuckGo help: About DuckAssistBot
robots.txt rule to allow:
User-agent: DuckAssistBot Allow: /
User-Triggered AI Fetchers
These fetchers run when a person asks an AI product to read a page. Because a human started the request, vendors say robots.txt may not apply, so treat them as visits from real readers.
19. ChatGPT-User (OpenAI)
ChatGPT-User visits a page when a person asks ChatGPT or a Custom GPT a question, or when a GPT Action calls out to a site. OpenAI says it is not used to crawl automatically, and that robots.txt rules may not apply. It is also not used to decide whether content appears in Search. The allow-or-block detail lives in our ChatGPT-User guide.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
- Owner: OpenAI
- Purpose: User-initiated page visits inside ChatGPT
- Published IPs: openai.com/chatgpt-user.json
- Also from OpenAI:
OAI-AdsBot/1.0checks landing pages submitted as ChatGPT ads and is not used for training. - Source: OpenAI crawler overview
20. Claude-User (Anthropic)
Claude-User supports Claude users. When someone asks Claude a question, it may fetch a site with this agent. Anthropic says disabling it prevents retrieval of your content in response to user queries and may reduce your visibility for user-directed search.
- Owner: Anthropic
- Purpose: User-initiated page visits inside Claude
- Allow or block? Allow on public content you want Claude to read for its users.
- Good to know: Older guides, including earlier versions of this one, listed
Claude-Webandanthropic-ai. Anthropic’s current page lists only ClaudeBot, Claude-User and Claude-SearchBot. - Source: Anthropic crawler page
robots.txt rule to allow:
User-agent: Claude-User Allow: /
21. Perplexity-User (Perplexity)
Perplexity-User runs when a Perplexity user asks a question that needs a specific page. Perplexity says that because a user requested the fetch, this fetcher generally ignores robots.txt rules.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
- Owner: Perplexity AI
- Purpose: User-initiated page visits
- Allow or block? Treat it as a reader and a citation chance. A robots.txt block does little.
- Published IPs: perplexity.com/perplexity-user.json
- Source: Perplexity bots documentation
22. Meta-ExternalFetcher (Meta)
Meta-ExternalFetcher fetches individual links at a user’s request and supports product functions such as helping AI navigate websites to complete tasks. Meta says it may bypass robots.txt rules.
- User-agent:
meta-externalfetcher/1.1 - Owner: Meta
- Purpose: User-requested fetches for agentic AI features
- Allow or block? A robots.txt block may not stop it. Use a firewall rule if you must.
- Source: Meta web crawlers documentation
Social Media Crawlers
23. FacebookExternalHit (Meta)
FacebookExternalHit crawls an app or website when its link is shared on Facebook, Instagram or Messenger. It gathers and caches the title, description and thumbnail image for the link preview.
- User-agent:
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) - Owner: Meta
- Purpose: Link preview cards on Meta apps
- Allow or block? Allow. Meta notes it might bypass robots.txt for security or integrity checks such as malware scans.
- Key detail: It reads Open Graph tags such as
og:title,og:descriptionandog:image. Missing tags give an incomplete preview. - Source: Meta web crawlers documentation
24. Twitterbot (X)
Twitterbot fetches page metadata when a link is shared on X to build the card preview.
- User-agent:
Twitterbot - Owner: X Corp
- Purpose: Link card previews
- Allow or block? Allow.
- Not re-verified: X’s developer page we tried returned an error on 7 October 2026, so this entry carries the token only.
25. LinkedInBot (LinkedIn)
LinkedInBot fetches page metadata when a link is shared on LinkedIn. LinkedIn’s sharing guidance asks for Open Graph tags og:title, og:image, og:description and og:url, with images of at least 1200 by 627 pixels.
- User-agent:
LinkedInBot - Owner: LinkedIn (Microsoft)
- Purpose: Link previews on LinkedIn
- Allow or block? Allow, especially for B2B sites.
- Source: LinkedIn sharing requirements. The page does not publish the bot’s user-agent string, so we list the token only.
SEO And Analytics Crawlers
26. SemrushBot (Semrush)
SemrushBot is the crawler Semrush uses to collect web data for its backlink and marketing tools. Semrush also runs several purpose-specific bots, such as SiteAuditBot, SemrushBot-BA and SplitSignalBot, each with its own robots.txt token.
- User-agent:
SemrushBot - Owner: Semrush
- Purpose: Backlink graph and SEO data
- Allow or block? Allow, or block to save bandwidth. It does not affect your search rankings. It supports
Crawl-delaywith intervals up to 10 seconds. - Source: Semrush bot documentation
27. AhrefsBot (Ahrefs)
Ahrefs runs two main crawlers: AhrefsBot for its link index and AhrefsSiteAudit for its Site Audit tool. Both respect robots.txt allow and disallow rules and Crawl-Delay, and Cloudflare lists both as verified good bots.
- User-agent:
Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)andMozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit) - Owner: Ahrefs
- Purpose: Link index and site audits
- Allow or block? Allow, or rate-limit with
Crawl-Delayif it loads a small server. - Source: Ahrefs robot documentation
Other Google Crawlers Worth Knowing
Google documents more crawlers than Googlebot. These appear in server logs and are controlled with their own robots.txt tokens, according to Google’s common crawlers page.
| Crawler | robots.txt token | What it affects |
|---|---|---|
GoogleOther | GoogleOther | No specific product. Generic fetching, for example one-off research and development crawls. |
Google-InspectionTool | Google-InspectionTool | Search Console URL Inspection and the Rich Results Test only. No effect on Search. |
Storebot-Google | Storebot-Google | Google Shopping surfaces, including the Shopping tab in Search. |
Google-CloudVertexBot | Google-CloudVertexBot | Crawls site owners request for building Vertex AI Agents. No effect on Search. |
How To Control Web Crawler Access On WordPress
WordPress gives you several ways to manage which crawlers can access your site and which pages they can index.
robots.txt: Your site’s robots.txt file (at yourdomain.com/robots.txt) is the primary control. A common 2026 setup blocks training bots and keeps search and live-fetch bots allowed:
# Block AI training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
# Keep search and answer-engine crawlers allowed
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Googlebot
Allow: /
User-agent: bingbot
Allow: /Remember the limits: ChatGPT-User, Perplexity-User and Meta-ExternalFetcher can ignore robots.txt because a person started the request, and FacebookExternalHit may bypass it for security checks.
Meta robots tags: Add <meta name="robots" content="noindex, nofollow"> to specific pages you want to exclude from indexing entirely.
WordPress SEO plugins: Plugins like Rank Math and Yoast SEO let you set noindex and nofollow rules per page, per post type, or per taxonomy from the dashboard without editing code.
Nexter Extension includes security features such as Limit Login Attempts, so you can manage unwanted bot traffic alongside the rest of your site tools from one dashboard.
Further Read: How about highly securing your website content from malicious attacks and cyber threats? Here’s the blog about the 5 Best WordPress Content Protection Plugins.
Beyond robots.txt, two newer controls matter in 2026. The first is an llms.txt file for WordPress, a separate file that gives language models a curated view of your most important pages. It complements robots.txt rather than replacing it. The second is firewall-level blocking. If you need a hard block against bots that ignore robots.txt, write a Cloudflare firewall rule that matches the user agent and returns a 403, which stops the request at the edge before WordPress spends CPU on it. Our Cloudflare Turnstile guide covers the wider edge setup.
Markup matters too. If you want crawlers to actually understand and cite your content, a clean DOM helps, and block themes versus classic themes explains why a clean block theme is the easier path. For the bigger picture on search agents, read our agentic SEO guide.
How To Check That A Crawler Is Real
A user-agent string is just text, and anyone can copy it. Common Crawl warns that other crawlers falsely identify as CCBot, and the same trick works for any bot on this list. Before you trust or block a visitor, confirm it with one of these checks:
- Reverse DNS: Run a reverse lookup on the visiting IP and check the host. Google documents this in Verifying Google requests, Apple uses
*.applebot.apple.com, and Yandex uses hosts ending inyandex.ru,yandex.netoryandex.com. - Published IP lists: OpenAI publishes GPTBot, OAI-SearchBot and ChatGPT-User ranges. Anthropic publishes its crawler IPs, Perplexity publishes PerplexityBot and Perplexity-User ranges, DuckDuckGo publishes DuckAssistBot ranges, and Apple publishes Applebot ranges.
- Match both: Perplexity recommends combining user-agent matching with IP verification in your firewall rules, and that is a sound default for every bot.
Suggested Reading
- What Is GPTBot? Should You Allow Or Block OpenAI’s Crawler On WordPress: the full allow-or-block guide for OpenAI’s training bot.
- OAI-SearchBot: the crawler that decides whether ChatGPT search can find you.
- ChatGPT-User: what the user-triggered fetcher does and how to handle it.
- What Is ClaudeBot? The allow-or-block decision for Anthropic’s crawler.
- How Perplexity AI Cites WordPress Sites: what gets you cited and what gets ignored.
- What Is An llms.txt Generator? How to create and auto-update one on WordPress.
- Crawl Budget In WordPress: when it matters and the new AI-crawler problem.
- Google AI Mode And AI Overviews: how Google-Extended and AI features treat your content.
Complete Web Crawlers List: Summary Table
| # | Crawler | Owner | Category | User-agent token | Recommendation |
|---|---|---|---|---|---|
| 1 | Googlebot | Search engine | Googlebot/2.1 | Allow | |
| 2 | Bingbot | Microsoft | Search engine | bingbot | Allow |
| 3 | YandexBot | Yandex | Search engine | YandexBot/3.0 | Allow if relevant |
| 4 | DuckDuckBot | DuckDuckGo | Search engine | DuckDuckBot/1.1 | Allow |
| 5 | Applebot | Apple | Search engine | Applebot/0.1 | Allow |
| 6 | Baiduspider | Baidu | Search engine | Baiduspider | Allow if China audience |
| 7 | Slurp | Yahoo | Search engine | Yahoo! Slurp | Allow |
| 8 | GPTBot | OpenAI | AI training | GPTBot/1.4 | Block to opt out of training |
| 9 | ClaudeBot | Anthropic | AI training | ClaudeBot | Your call |
| 10 | Google-Extended | AI training token | Google-Extended | Block to opt out | |
| 11 | Meta-ExternalAgent | Meta | AI training | meta-externalagent/1.1 | Block to opt out |
| 12 | Bytespider | ByteDance | AI training | Bytespider | Block (most Western sites) |
| 13 | CCBot | Common Crawl | Open dataset | CCBot/2.0 | Block to opt out |
| 14 | OAI-SearchBot | OpenAI | AI search | OAI-SearchBot/1.4 | Allow |
| 15 | Claude-SearchBot | Anthropic | AI search | Claude-SearchBot | Allow |
| 16 | PerplexityBot | Perplexity | AI search | PerplexityBot/1.0 | Allow |
| 17 | Meta-WebIndexer | Meta | AI search | meta-webindexer/1.1 | Allow |
| 18 | DuckAssistBot | DuckDuckGo | AI answers | DuckAssistBot/1.2 | Allow |
| 19 | ChatGPT-User | OpenAI | User-triggered | ChatGPT-User/1.0 | Allow (robots.txt may not apply) |
| 20 | Claude-User | Anthropic | User-triggered | Claude-User | Allow |
| 21 | Perplexity-User | Perplexity | User-triggered | Perplexity-User/1.0 | Allow (generally ignores robots.txt) |
| 22 | Meta-ExternalFetcher | Meta | User-triggered | meta-externalfetcher/1.1 | May bypass robots.txt |
| 23 | FacebookExternalHit | Meta | Social media | facebookexternalhit/1.1 | Allow |
| 24 | Twitterbot | X Corp | Social media | Twitterbot | Allow |
| 25 | LinkedInBot | Social media | LinkedInBot | Allow | |
| 26 | SemrushBot | Semrush | SEO tool | SemrushBot | Allow or block |
| 27 | AhrefsBot | Ahrefs | SEO tool | AhrefsBot/7.0 | Allow or rate-limit |
Stay updated with Helpful WordPress Tips, Insider Insights, and Exclusive Updates – Subscribe now to keep up with Everything Happening on WordPress!
Wrapping Up
Web crawlers are the backbone of how search engines, AI platforms, and social networks discover and organize content. The 27 bots above are the ones you are most likely to meet in your WordPress access logs in 2026, from classic Googlebot to the newer AI search and live-fetch bots.
The practical takeaway is to keep the crawlers that bring you search and AI visibility (Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot), and decide separately about the training bots (GPTBot, ClaudeBot, Google-Extended, Meta-ExternalAgent, CCBot). Set clear rules in robots.txt, and verify suspicious traffic with reverse DNS or published IP lists.
Build On A Crawler-Friendly WordPress Stack
If you want a WordPress theme that outputs clean, semantic HTML for crawlers to read, Nexter Theme is built with zero jQuery. Combined with 90+ Gutenberg blocks and 50+ site extensions, you get a crawler-friendly stack without plugin bloat. See Nexter pricing to get started.
Get Exclusive WordPress Tips, Tricks And Resources Delivered Straight To Your Inbox!
Subscribe to stay updated with everything happening on WordPress.
FAQs On Web Crawlers
What Are The Most Common Web Crawlers?
The best-known web crawlers are Googlebot (Google Search), Bingbot (Bing Search), GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), FacebookExternalHit (Meta), AhrefsBot (Ahrefs) and SemrushBot (Semrush). Googlebot is the most important for most sites because it feeds Google Search. See the full list above for all 27.
Is ChatGPT A Web Crawler?
ChatGPT itself is not a crawler, but OpenAI runs three bots you can control separately. GPTBot collects training data, OAI-SearchBot indexes pages for ChatGPT search, and ChatGPT-User visits a page when a person asks about it. Blocking GPTBot does not block the other two. The ChatGPT-User guide explains the live-fetch side.
How Do I Block AI Crawlers From My WordPress Site?
Add a disallow rule for each AI bot to your robots.txt file. The training bots most owners block are GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot and Bytespider. Rank Math and Yoast let you edit robots.txt from the dashboard. Bots that ignore robots.txt need a firewall rule instead.
How Do I Check Which Crawlers Are Visiting My Site?
Check your server access logs for bot user-agent strings. Most hosting dashboards (cPanel, Plesk, RunCloud) offer a log viewer. Google Search Console and Bing Webmaster Tools show Googlebot and Bingbot activity, and Cloudflare analytics can show bot traffic. Confirm suspicious entries with reverse DNS or published IP lists, since user agents can be faked.
How Much Of A Page Does Googlebot Read?
For Search, Googlebot crawls the first 2MB of a supported file type and the first 64MB of a PDF, according to Google’s Googlebot documentation. Keep important content and links high in the HTML so they sit well inside that limit.
Do Web Crawlers Affect Website Performance?
Yes, aggressive crawling can add server load. Several bots support a Crawl-delay directive, including ClaudeBot, SemrushBot, AhrefsBot and Slurp, while Applebot does not follow it. For a heavy hitter, add a delay or a disallow rule for that one bot. Our crawl budget guide shows how to tell whether crawling is actually a problem.
What Is The Difference Between A Web Crawler And A Web Scraper?
A web crawler systematically discovers and indexes pages by following links. A web scraper extracts specific data from pages for a targeted purpose, such as collecting prices or contact details. Crawlers from major vendors follow robots.txt, while scrapers are often custom-built and may not respect access controls.
