Top Web Crawlers & Bot Traffic Stats for 2025
WAF360
14 min read
Total Bot Traffic
51.37%
Is Dominated by Just 3 Bots
Search Engine Share
~45%
of top bot traffic is from major search engines.
The Rise of AI
>12%
of bot activity is now driven by AI training and queries.
The Digital Giants: Traffic Share
The internet's bot traffic is highly concentrated. A handful of major players from search and AI account for the vast majority of all automated hits, leaving a small fraction for everyone else.
Top 15 Bots by Traffic
While Google and Bing lead, the rise of AI crawlers like ChatGPT-User and GPTBot is undeniable. This chart compares the traffic percentage of the most active bots on the web.
Who's Crawling? A Field Guide to Bots
Search Engines
The foundation of the web, these bots index content for search results.
- Googlebot (Smartphone & Desktop)
- Bingbot
- Applebot
- YandexBot
- Yahoo! Slurp
AI & Data Crawlers
The new generation of bots gathering data to train large language models.
- ChatGPT-User
- GPTBot
- OAI-SearchBot
- ClaudeBot
- PerplexityBot
Social & Link Preview
These bots fetch data to create the link previews you see in apps.
- Meta-ExternalAgent
- FacebookExternalHit
- LinkedInBot
- Twitterbot
- DiscordBot
SEO & Analytics
Tools used by marketers to analyze websites and track rankings.
- Barkrowler
- BLEXBot
- DataForSeoBot
- AhrefsBot
- SemrushBot
The following list is detected by the WAF360 intelligent bot management system.
Top Bot List 2025
| Rank | Bot Name | % of Total | Description |
|---|---|---|---|
| 1 | Googlebot Mobile | 19.08% | Google's mobile-first crawler simulating a smartphone user |
| 2 | Bingbot | 18.91% | Microsoft's main web crawler for Bing Search |
| 3 | Meta-ExternalAgent | 13.38% | Meta's bot for fetching external content, likely for link previews |
| 4 | ChatGPT-User | 7.54% | Identifier for ChatGPT traffic fetching site data |
| 5 | Googlebot Desktop | 5.80% | Google's crawler simulating a desktop browser |
| 6 | FacebookExternalHit | 4.58% | Facebook's bot for generating link previews |
| 8 | GPTBot | 3.03% | OpenAI's crawler for AI training and search |
| 10 | Applebot | 2.66% | Apple's crawler for Siri and Spotlight search |
| 11 | OAI-SearchBot | 2.14% | OpenAI's bot for search-related content retrieval |
| 12 | Barkrowler | 1.52% | SEO crawling bot from Barkrowler services |
| 13 | YandexAccessibilityBot | 1.16% | Yandex bot testing accessibility and rendering |
| 14 | Meta-ExternalAds | 1.15% | Meta's bot for validating ads and previews |
| 15 | GoogleOther | 0.94% | Miscellaneous Google crawler activity |
| 16 | Yahoo! Slurp | 0.90% | Yahoo's historical search crawler |
| 17 | YandexBot | 0.86% | Main search crawler for Yandex |
| 18 | LinkedInBot | 0.84% | LinkedIn's link preview generator |
| 19 | BLEXBot | 0.74% | SEO analytics bot for backlink analysis |
| 20 | DataForSeoBot | 0.71% | DataForSEO's bot for SERP and keyword analysis |
| 21 | Twitterbot | 0.64% | Twitter's crawler for link cards and previews |
| 22 | Amazonbot | 0.61% | Amazon's web crawler for various indexing uses |
| 23 | MojeekBot | 0.38% | Independent search engine crawler |
| 24 | ClaudeBot | 0.37% | Anthropic's AI-related crawler |
| 25 | Googlebot-Image | 0.36% | Specialized Google crawler for images |
| 26 | PerplexityBot | 0.34% | Perplexity AI's content-fetching crawler |
| 27 | Cortex | 0.31% | Bot associated with Cortex AI or analytics tools |
| 28 | FlipboardRSS | 0.29% | Flipboard's bot for pulling RSS content |
| 29 | Sogou Web Spider | 0.28% | Chinese search engine Sogou's crawler |
| 30 | Feedly | 0.22% | RSS reader Feedly's content fetcher |
| 31 | BingPreview | 0.18% | Bing's page preview generator |
| 32 | Protopage | 0.18% | Protopage's bot for RSS and widget updates |
| 33 | Moreover | 0.16% | News aggregation bot from Moreover |
| 34 | CCBot | 0.16% | Common Crawl's open web indexing bot |
| 35 | TrendictionBot | 0.15% | Trendiction's content aggregation crawler |
| 36 | YaDirectFetcher | 0.15% | Yandex Direct ads fetching bot |
| 37 | AdsBot-Google | 0.13% | Google's bot for verifying ads landing pages |
| 38 | Feeder.co | 0.13% | Feeder RSS reader's crawler |
| 39 | PetalBot | 0.13% | Huawei's search engine bot |
| 40 | Turnitin | 0.12% | Plagiarism detection service crawler |
| 41 | AdsBot-Google-Mobile | 0.12% | Mobile version of Google's ads verification bot |
| 42 | Google-Site-Verification | 0.11% | Bot checking site ownership for Google Search Console |
| 43 | SeznamBot | 0.11% | Crawler for Czech search engine Seznam |
| 44 | AmazonAdBot | 0.11% | Amazon's ad verification bot |
| 45 | Site24x7 | 0.09% | Website uptime monitoring bot |
| 46 | Yeti | 0.09% | Naver's crawler (South Korea) |
| 47 | Google Favicon | 0.08% | Google's bot fetching favicons |
| 48 | YaK | 0.07% | Yandex's image or content fetching bot |
| 49 | Inoreader | 0.07% | RSS reader Inoreader's bot |
| 50 | Google Web Preview Analytics | 0.07% | Google's analytics preview bot |
| 51 | Known Attack M | 0.06% | Malicious traffic variant flagged by system |
| 52 | Omgilibot | 0.06% | Bot used for indexing online libraries |
| 53 | Google Web Preview | 0.06% | Google's preview fetching bot |
| 54 | Adreview | 0.05% | Ad content quality check bot |
| 55 | PlayStore-Google | 0.05% | Google Play Store's content verification bot |
| 56 | Peer39 Crawler | 0.05% | Peer39's semantic ad targeting bot |
| 57 | Photon | 0.05% | Image proxy and optimization bot |
| 58 | LivelapBot | 0.05% | Bot for sports/live event data updates |
| 59 | Google-Read-Aloud | 0.05% | Bot for Google's read-aloud service |
| 60 | BUbiNG | 0.05% | High-speed web crawler research project |
| 61 | DotBot 1.1 | 0.05% | Moz's SEO crawler |
| 63 | Google-Ads-Conversions | 0.04% | Bot for tracking ad conversions |
| 64 | Linkbot | 0.04% | SEO link-checking bot |
| 65 | SentiBot | 0.04% | Sentiment analysis crawler |
| 66 | StatusCake | 0.04% | Uptime monitoring bot |
| 67 | Feedspot | 0.04% | Feedspot RSS aggregator bot |
| 68 | Semantic-Visions | 0.04% | Semantic Visions data collection bot |
| 69 | OnlineOrNot Bot | 0.04% | Website uptime and performance monitoring bot |
| 70 | AhrefsBot | 0.04% | Popular SEO backlink analysis crawler |
| 71 | Google-PageRenderer | 0.04% | Google's rendering bot for JavaScript-heavy pages |
| 72 | Chrome-Lighthouse | 0.03% | Google's performance and accessibility audit bot |
| 73 | YisouSpider | 0.03% | Crawler for Yisou search engine (China) |
| 74 | FlipboardProxy | 0.03% | Flipboard's proxy fetching bot |
| 75 | GoogleWebLight | 0.03% | Google's mobile data-saving crawler |
| 76 | Brightbot | 0.03% | Bot for content marketing or SEO testing |
| 77 | Neticle Crawler | 0.03% | Neticle's online monitoring crawler |
| 78 | DotBot 1.2 | 0.03% | Moz's updated SEO crawling bot |
| 79 | Coccocbot | 0.03% | Vietnamese search engine Cốc Cốc's bot |
| 80 | DuckDuckBot | 0.03% | DuckDuckGo's web crawler |
| 81 | ClaudeBot | 0.03% | Anthropic's AI crawler (duplicate variant) |
| 82 | Adbeat Bot | 0.02% | Competitive ad intelligence crawler |
| 83 | Uptime Bot | 0.02% | Website uptime monitoring bot |
| 84 | YandexRenderResources | 0.02% | Yandex bot fetching render resources |
| 85 | DuckAssistBot | 0.02% | DuckDuckGo's AI answer bot |
| 86 | SkypeUriPreview | 0.02% | Skype's link preview bot |
| 87 | Baiduspider | 0.02% | Baidu's main search engine crawler |
| 88 | Google-Safety | 0.02% | Google's safety verification bot |
| 89 | 360Spider | 0.02% | Qihoo 360 search engine crawler |
| 90 | FeedBurner | 0.02% | Google's FeedBurner RSS fetcher |
| 91 | Outbrain | 0.02% | Outbrain's content recommendation bot |
| 92 | SemrushBot | 0.02% | SEO analysis crawler from Semrush |
| 93 | DiscordBot | 0.01% | Discord's link preview bot |
| 94 | UptimeRobot | 0.01% | Popular uptime monitoring service bot |
| 95 | SiteAuditBot | 0.01% | SEO site audit crawler |
| 96 | Storebot-Google | 0.01% | Google Play Store's indexing bot |
| 97 | MJ12bot | 0.01% | Distributed web crawler project |
| 98 | Imgproxy | 0.01% | Image proxy and transformation service |
| 99 | Perplexity-User | 0.01% | Perplexity AI user agent for fetching content |
| 100 | Slack-ImgProxy | 0.01% | Slack's image proxy bot |
| 101 | Google-Shopping-Quality | 0.01% | Google Shopping quality crawler |
| 102 | WadooImageBot/1.0 | 0.01% | Image crawling bot from Wadoo |
| 103 | Mediapartners-Google | 0.01% | Google AdSense crawler |
| 104 | MJ12bot/v2.0.2 | 0.01% | MJ12bot distributed crawler (v2.0.2) |
| 105 | theoldreader.com | 0.01% | RSS reader bot |
| 106 | agentslug | 0.01% | Bot used for content aggregation |
| 107 | DongleEmulatorBot/0.1 | 0.01% | Emulator bot for testing |
| 108 | GumGum-Bot/1.0 | 0.01% | Ad/visual recognition bot |
| 109 | GoogleDocs; documents | 0.01% | Google Docs content fetcher |
| 110 | AhrefsBot/7.0 SA | 0.01% | SEO backlink crawler |
| 111 | Embedly/0.2 | 0.01% | Link preview and embedding bot |
| 112 | archive.org_bot | 0.01% | Internet Archive crawler |
| 113 | TelegramBot | 0.01% | Telegram link preview bot |
| 114 | allOrigins/3 | 0.01% | Data fetching bot for web origins |
| 115 | IABot/2.0 | 0.01% | IAB (Interactive Advertising Bureau) crawler |
| 116 | SentiBot | 0.01% | Sentiment analysis crawler |
| 117 | Googlebot snippet | 0.01% | Specialized Google snippet crawler |
| 118 | proximic | 0.01% | Contextual ad analytics crawler |
| 119 | dlvr.it/1.0 | 0.01% | Social media content distribution bot |
| 120 | IbouBot/1.0 | 0.01% | Generic web content crawler |
| 121 | 0.01% | Miscellaneous Google crawling activity | |
| 122 | PubMatic Crawler Bot | 0.01% | Ad verification crawler |
| 123 | Chrome Privacy Preserving Prefetch Proxy | 0.01% | Chrome prefetch proxy bot |
| 124 | GoogleAdwordsExpress | 0.01% | Adwords Express crawler |
| 125 | Qwantbot/1.0 | 0.01% | Qwant search engine crawler |
| 126 | coccocbot-image/1.0 | 0.01% | Cốc Cốc image crawler |
| 127 | YandexImages/3.0 | 0.01% | Yandex image crawler |
| 128 | um-IC/1.0 | 0.01% | Unknown bot, user agent um-IC |
| 129 | kakaotalk-scrap/1.0 | 0.01% | KakaoTalk content scraping bot |
| 130 | ReverseEngineeringBot/0.1 | 0.01% | Bot for reverse engineering/testing |
| 131 | Verity/1.1 | 0.01% | Web crawling/monitoring bot |
| 132 | VirusTotalBot | 0.01% | Malware analysis bot |
| 133 | TextRazor Downloader | 0.01% | TextRazor content fetching bot |
| 134 | YisouSpider | 0.01% | Yisou search engine crawler |
| 135 | Qwantbot/1.0_3151071 | 0.01% | Variant of Qwantbot crawler |
| 136 | CommaFeed/4.4.0 | 0.01% | RSS reader crawler |
| 137 | DecompilationBot/0.1 | 0.01% | Decompilation testing bot |
| 138 | DuckDuckGo-Favicons-Bot/1.0 | 0.01% | DuckDuckGo favicon fetcher |
| 139 | Paqlebot/2.0 | 0.01% | Content aggregation bot |
| 140 | l9scan | 0.01% | Security scanning bot |
| 141 | Dataprovider | 0.01% | Content/data collection bot |
| 142 | LinkpadBot/2.4 | 0.01% | SEO link checking bot |
| 143 | imageSpider | 0.01% | Image scraping bot |
| 144 | HatenaBlog-bot/0.02 | 0.01% | Hatena Blog crawler |
| 145 | NetSystemsResearch | 0.01% | Monitoring/security bot |
| 146 | GPTBot/1.0 | 0.01% | OpenAI GPTBot variant |
| 147 | YandexMarket/2.0 | 0.01% | Yandex Market crawler |
| 148 | OrgProbe/2.0.0 | 0.01% | Web probing bot |
| 149 | VelenPublicWebCrawler/1.0 | 0.01% | Public web crawler |
| 150 | Google PP Default | 0.01% | Google PageSpeed default crawler |
| 151 | BrightEdge Crawler/1.0 | 0.01% | SEO crawler for content marketing |
| 152 | KOCMOHABT Web Explorer | 0.01% | Web exploration bot |
| 153 | um-LN/1.0 | 0.01% | Unknown bot user agent |
| 154 | Nekstbot | 0.01% | Web crawling bot |
| 155 | PocketParser/2.0 | 0.01% | RSS parser crawler |
| 156 | Bytespider | 0.01% | SEO/content crawler |
| 157 | Whibse | 0.01% | Web crawling bot |
| 158 | SemrushBot-BA | 0.01% | SEO analysis crawler |
| 159 | Expanse | 0.01% | Web asset discovery bot |
| 160 | Custom-AsyncHttpClient | 0.01% | HTTP fetching bot |
| 161 | YandexMobileBot/3.0 | 0.01% | Yandex mobile crawler |
| 162 | Keydrop.io/1.0 | 0.01% | Link and SEO crawler |
| 163 | Instapaper/4.0 | 0.01% | Instapaper content fetcher |
| 164 | linkdex.com/v2.0 | 0.01% | SEO and link analysis bot |
| 165 | Y!J-ASR/1.0 | 0.01% | Yahoo Japan search crawler |
| 166 | GoogleDocs; drawings | 0.01% | Google Docs drawing fetcher |
| 167 | line-poker/1.0 | 0.01% | Unknown content fetcher |
| 168 | OrgProbe/2.2.0 | 0.01% | Web probing bot variant |
| 169 | Qwantbot/1.0_crawler-fetcher-25 | 0.01% | Qwantbot variant for crawling |
| 170 | SemrushBot-SI/0.97 | 0.01% | SEO crawler variant |
| 171 | TestLocally/1.0 | 0.01% | Testing bot |
| 172 | SemrushBot | 0.01% | SEO crawler |
| 173 | bl.uk_ldfc_bot/3.4.0 | 0.01% | British Library content bot |
| 174 | GCore Labs Cyberthreat Research | 0.01% | Threat research crawler |
| 175 | google-xrawler | 0.01% | Google scraping/testing bot |
| 176 | gocrawl v0.4 | 0.01% | Web crawling bot |
| 177 | adidxbot/2.0 | 0.01% | Ad indexing crawler |
| 178 | Xpanse | 0.01% | Web scanning bot |
| 179 | Baiduspider/2.0 mobile | 0.01% | Baidu mobile crawler |
Unleash the true power of your digital presence with our state-of-the-art solutions. Elevate your online experience and command attention in the digital landscape. Our cutting-edge technologies are meticulously designed to propel your brand forward, ensuring not just a presence, but a commanding and impactful visibility. Explore limitless possibilities and carve your path to digital success with our innovative solutions, tailored to empower and amplify your online journey.
Contact Us