Crawl Activity
SEO Compare — seocompare.co.uk
Bing crawl ran 1h ago · 16 May, 13:26 · runs daily 06:00
Cloudflare AI Crawl Control (14d)
Edge-level bot counts, true hits — not sampled
4,609
13 bots tracked
05-09 05-16
Bingbot 1,082
Amazonbot 967
ChatGPT-User 935
Applebot 439
ClaudeBot 273
1550
Total Bing Crawls
227
Pages Indexed (Bing)
13
AI Bots Allowed
0
AI Bots Blocked
Crawler Activity
Bing (bars, left axis) vs AI bots (lines, right axis)
AI Crawler Access
GPTBot allowed
ChatGPT-User allowed
Google-Extended allowed
GoogleOther allowed
ClaudeBot allowed
Claude-Web not_mentioned
PerplexityBot allowed
Bytespider allowed
CCBot allowed
cohere-ai allowed
Amazonbot allowed
anthropic-ai not_mentioned
FacebookBot allowed
Applebot-Extended allowed
Bingbot allowed
Server-Side Bot Hits
Real crawler visits detected at the server level (last 30 days)
14526
Total Bot Hits
3696
AI Crawler Hits
2852
Search Crawler Hits
21
Unique Bots
Bot Breakdown
Other Crawler 5867
Bingbot 1686
GPTBot 1624
Other Bot 1500
ChatGPT-User 1004
ClaudeBot 759
Applebot 691
MajesticBot 515
YandexBot 238
Googlebot 217
PerplexityBot 157
Meta-Agent 107
Facebook 67
Bytespider 29
DuckDuckBot 20
Daily Hits
16 May
44 AI 80 search 6 social 12 seo 159
15 May
85 AI 108 search 2 social 12 seo 242
14 May
73 AI 77 search 1 social 6 seo 180
13 May
79 AI 70 search 3 social 12 seo 185
12 May
111 AI 124 search 4 social 17 seo 421
11 May
90 AI 122 search 3 social 22 seo 371
10 May
147 AI 149 search 3 social 21 seo 598
9 May
58 AI 143 search 2 social 6 seo 336
8 May
485 AI 168 search 1 social 15 seo 1096
7 May
752 AI 149 search 2 social 10 seo 936
6 May
89 AI 137 search 6 social 5 seo 262
5 May
92 AI 146 search 2 social 12 seo 1500
4 May
119 AI 110 search 2 social 12 seo 1125
3 May
174 AI 95 search 5 social 14 seo 765
Live Crawler Access Test (HTTP Response)
Tests if each bot actually gets a 200 response when visiting the homepage
GPTBot
ClaudeBot
PerplexityBot
Googlebot
Bingbot
Topic Clusters (229 detected)
/research/
4
pages in cluster
/ai-search-questions/
1
pages in cluster
/what-is-ai-search-optimisation/
1
pages in cluster
/ai-seo-solicitors/
1
pages in cluster
/ai-seo-private-healthcare/
1
pages in cluster
/ai-seo-law-firms-uk/
1
pages in cluster
/insights/
8
pages in cluster
/pure-ai-search-agencies/
1
pages in cluster
/top-chatgpt-seo-agencies/
1
pages in cluster
/best-ai-seo-audit-services/
1
pages in cluster
/ai-search-visibility/
1
pages in cluster
/ai-search-agencies-london/
1
pages in cluster
/ai-seo-audit-birmingham/
1
pages in cluster
/best-ai-website-builders-for-ecommerce-uk/
1
pages in cluster
/how-to-get-cited-ai-overviews/
1
pages in cluster
/affordable-ai-seo-uk/
1
pages in cluster
/ai-search-agencies-chelmsford/
1
pages in cluster
/ai-search-agencies-enterprise/
1
pages in cluster
/best-geo-agencies-b2b/
1
pages in cluster
/ai-seo-b2b-guide/
1
pages in cluster
/ai-seo-tools-saas/
1
pages in cluster
/b2b-ai-overviews/
1
pages in cluster
/chatgpt-vs-perplexity-vs-gemini/
1
pages in cluster
/contact/
1
pages in cluster
/ai-seo-law-firms/
1
pages in cluster
/tools/
4
pages in cluster
/best-llmo-agencies/
1
pages in cluster
/top-ai-seo-financial-services/
1
pages in cluster
/best-ai-seo-agencies-healthcare/
1
pages in cluster
/ai-seo-agencies-usa/
1
pages in cluster
/ai-seo-recruitment-agencies/
1
pages in cluster
/ai-seo-audit-newcastle/
1
pages in cluster
/ai-search-agencies-cardiff/
1
pages in cluster
/ai-search-agencies-edinburgh/
1
pages in cluster
/top-geo-agencies-2026/
1
pages in cluster
/best-seo-agencies-uk-2026/
1
pages in cluster
/ai-search-agencies-canada/
1
pages in cluster
/show-on-ai-search/
1
pages in cluster
/privacy/
1
pages in cluster
/ai-search-agencies-saas/
1
pages in cluster
/getcito-review/
1
pages in cluster
/ai-seo-audit-sheffield/
1
pages in cluster
/otterly-ai-review/
1
pages in cluster
/disclaimer/
1
pages in cluster
/ai-search-agencies-birmingham/
1
pages in cluster
/agency-comparison-matrix/
1
pages in cluster
/top-ai-seo-agencies-b2b/
1
pages in cluster
/how-to-rank-b2b-chatgpt/
1
pages in cluster
/ai-seo-audit-chelmsford/
1
pages in cluster
/top-5-ai-seo-agencies/
1
pages in cluster
/chatgpt-seo-healthcare/
1
pages in cluster
/best-perplexity-seo-agencies/
1
pages in cluster
/ai-seo-ecommerce-guide/
1
pages in cluster
/ai-seo-tools-uk/
1
pages in cluster
/geo-for-healthcare/
1
pages in cluster
/best-ai-seo-agencies-saas/
1
pages in cluster
/ai-seo-audit-oxford/
1
pages in cluster
/best-ai-search-website-builders-uk/
1
pages in cluster
/geo-for-law-firms/
1
pages in cluster
/authors/
1
pages in cluster
/agencies-with-case-studies/
1
pages in cluster
/ai-seo-audit-brighton/
1
pages in cluster
/ai-seo-vs-geo-vs-aeo-agencies/
1
pages in cluster
/top-ai-seo-agencies/
1
pages in cluster
/top-ai-seo-healthcare/
1
pages in cluster
/find-an-agency/
1
pages in cluster
/top-ai-seo-agencies-saas/
1
pages in cluster
/chatgpt-seo-financial/
1
pages in cluster
/chatgpt-seo-law-firms/
1
pages in cluster
/ai-seo-audit-cambridge/
1
pages in cluster
/saas-ai-overviews/
1
pages in cluster
/ai-seo-audit-cardiff/
1
pages in cluster
/ai-search-agencies-europe/
1
pages in cluster
/ai-seo-tools-ecommerce/
1
pages in cluster
/ai-seo-tools-b2b/
1
pages in cluster
/how-we-are-funded/
1
pages in cluster
/ai-seo-dentists-uk/
1
pages in cluster
/ai-search-agencies-brighton/
1
pages in cluster
/best-ai-visibility-agencies/
1
pages in cluster
/best-ai-seo-agencies-b2b/
1
pages in cluster
/methodology/
1
pages in cluster
/best-geo-agencies/
1
pages in cluster
/ai-seo-ifa-financial-advisors/
1
pages in cluster
/best-free-ai-website-builders-uk/
1
pages in cluster
/ai-search-agencies-southampton/
1
pages in cluster
/chatgpt-seo-ecommerce/
1
pages in cluster
/best-aeo-agencies-ecommerce/
1
pages in cluster
/ai-seo-agencies-saas-uk/
1
pages in cluster
/schema-markup-ai-search/
1
pages in cluster
/terms/
1
pages in cluster
/ai-seo-audit-southampton/
1
pages in cluster
/peec-ai-review/
1
pages in cluster
/best-aeo-agencies-uk/
1
pages in cluster
/ai-seo-saas-vs-b2b/
1
pages in cluster
/ai-seo-healthcare/
1
pages in cluster
/top-ai-visibility-agencies/
1
pages in cluster
/ai-search-hospitality/
1
pages in cluster
/best/
16
pages in cluster
/ai-seo-professional-services/
1
pages in cluster
/ai-seo-nhs/
1
pages in cluster
/agencies-offering-audits/
1
pages in cluster
/ai-seo-audit-leeds/
1
pages in cluster
/about/
1
pages in cluster
/get-found-on-chatgpt/
1
pages in cluster
/best-chatgpt-seo-agencies/
1
pages in cluster
/ai-search-statistics-2026/
1
pages in cluster
/ai-search-agencies-liverpool/
1
pages in cluster
/best-ai-seo-agencies-uk/
1
pages in cluster
/ai-seo-banking/
1
pages in cluster
/ai-citation-gaps/
1
pages in cluster
/top-aeo-agencies-2026/
1
pages in cluster
/ai-search-agencies-leeds/
1
pages in cluster
/blog/
37
pages in cluster
/ai-search-agencies-colchester/
1
pages in cluster
/semrush-vs-ahrefs-ai-visibility/
1
pages in cluster
/how-to-rank-ecommerce-chatgpt/
1
pages in cluster
/chatgpt-seo-agency-uk/
1
pages in cluster
/ai-seo-agency-manchester/
1
pages in cluster
/ai-seo-vs-traditional-seo/
1
pages in cluster
/ai-seo-shopify/
1
pages in cluster
/best-ai-seo-companies/
1
pages in cluster
/ai-overviews-agencies/
1
pages in cluster
/ai-seo-audit-glasgow/
1
pages in cluster
/ai-search-agencies-oxford/
1
pages in cluster
/ai-seo-audit-nottingham/
1
pages in cluster
/best-aeo-agencies/
1
pages in cluster
/uk-ai-overviews-agencies/
1
pages in cluster
/how-to-get-cited-gemini/
1
pages in cluster
/chatgpt-optimisation-agencies/
1
pages in cluster
/ai-visibility-agency-uk/
1
pages in cluster
/ai-seo-agencies-ecommerce-uk/
1
pages in cluster
/chatgpt-visibility/
1
pages in cluster
/agency-matcher/
1
pages in cluster
/best-ai-search-tools/
1
pages in cluster
/agency-directory/
1
pages in cluster
/aeo-vs-geo-vs-llmo/
1
pages in cluster
/se-ranking-ai-tracking-review/
1
pages in cluster
/ai-search-agencies-middle-east/
1
pages in cluster
/ai-seo-barristers/
1
pages in cluster
/ai-seo-audit/
1
pages in cluster
/ai-search-agencies-uk/
1
pages in cluster
/best-llmo-agencies-2026/
1
pages in cluster
/build-brand-ai-trust/
1
pages in cluster
/ai-seo-dental/
1
pages in cluster
/guides/
1
pages in cluster
/ai-seo-audit-london/
1
pages in cluster
/best-geo-agencies-2026/
1
pages in cluster
/editorial-policy/
1
pages in cluster
/ai-search-agencies-reading/
1
pages in cluster
/structure-website-ai-crawlers/
1
pages in cluster
/ai-seo-audit-colchester/
1
pages in cluster
/affordable-ai-seo-ecommerce/
1
pages in cluster
/ai-seo-manufacturing/
1
pages in cluster
/best-geo-agencies-ecommerce/
1
pages in cluster
/ai-search-agencies-norwich/
1
pages in cluster
/top-ai-seo-agencies-ecommerce/
1
pages in cluster
/ai-search-agencies-asia/
1
pages in cluster
/ai-visibility-check/
1
pages in cluster
/ai-search-local-business/
1
pages in cluster
/ai-search-agencies-nottingham/
1
pages in cluster
/how-to-get-cited-perplexity/
1
pages in cluster
/best-ai-search-agencies/
1
pages in cluster
/ai-search-agencies-ecommerce/
1
pages in cluster
/ai-seo-financial-uk/
1
pages in cluster
/compare/
67
pages in cluster
/top-ai-seo-agencies-law-firms/
1
pages in cluster
/agency-vs-tool/
1
pages in cluster
/how-to-rank-law-firm-chatgpt/
1
pages in cluster
/ai-search-agencies-sheffield/
1
pages in cluster
/ai-seo-travel-tourism-uk/
1
pages in cluster
/affordable-ai-seo-agencies/
1
pages in cluster
/best-geo-agencies-uk/
1
pages in cluster
/top-ai-seo-law-firms/
1
pages in cluster
/ai-seo-linkedin-b2b/
1
pages in cluster
/ai-seo-audit-bristol/
1
pages in cluster
/ai-search-agencies-newcastle/
1
pages in cluster
/chatgpt-seo-b2b/
1
pages in cluster
/ai-seo-london/
1
pages in cluster
/ai-seo-audit-edinburgh/
1
pages in cluster
/perplexity-optimisation-agencies/
1
pages in cluster
/best-ai-overviews-agencies-2026/
1
pages in cluster
/ai-search-agencies-small-business/
1
pages in cluster
/ai-search-agencies-b2b/
1
pages in cluster
/best-geo-agencies-saas/
1
pages in cluster
/ai-seo-audit-norwich/
1
pages in cluster
/uk-ai-seo-agency-market-benchmarks-2026/
1
pages in cluster
/top-10-ai-seo-tools/
1
pages in cluster
/ai-seo-audit-reading/
1
pages in cluster
/ai-seo-agency-london/
1
pages in cluster
/ai-search-agencies-australia/
1
pages in cluster
/ai-seo-agencies-b2b-uk/
1
pages in cluster
/best-ai-seo-agencies/
1
pages in cluster
/how-to-get-cited-chatgpt/
1
pages in cluster
/ai-seo-insurance/
1
pages in cluster
/best-gemini-seo-agencies/
1
pages in cluster
/ai-seo-saas-guide/
1
pages in cluster
/ai-search-agencies-bristol/
1
pages in cluster
/ai-seo-healthcare-uk/
1
pages in cluster
/ai-seo-agency-uk/
1
pages in cluster
/alternatives/
13
pages in cluster
/geo-for-fintech/
1
pages in cluster
/geo-vs-seo/
1
pages in cluster
/ai-visibility-score/
1
pages in cluster
/ai-seo-audit-manchester/
1
pages in cluster
/ai-search-agencies-cambridge/
1
pages in cluster
/data/
2
pages in cluster
/ai-seo-hotels-accommodation/
1
pages in cluster
/best-ai-seo-agencies-ecommerce/
1
pages in cluster
/ai-search-agencies-glasgow/
1
pages in cluster
/ai-search-glossary/
1
pages in cluster
/ai-seo-manchester/
1
pages in cluster
/ai-search-optimisation-cost/
1
pages in cluster
/ai-seo-recruitment/
1
pages in cluster
/ai-seo-audit-liverpool/
1
pages in cluster
/free-ai-visibility-audit/
1
pages in cluster
/ai-seo-financial-services/
1
pages in cluster
/how-to-get-cited-claude/
1
pages in cluster
/stats/
24
pages in cluster
/chatgpt-seo-agencies-uk/
1
pages in cluster
/chatgpt-seo-saas/
1
pages in cluster
/top-ai-seo-agencies-uk/
1
pages in cluster
/ecommerce-ai-overviews/
1
pages in cluster
/questions/
56
pages in cluster
/ai-seo-product-led-growth/
1
pages in cluster
/best-ai-seo-agencies-financial/
1
pages in cluster
/best-ai-seo-agencies-law-firms/
1
pages in cluster
/ai-search-agencies-manchester/
1
pages in cluster
/best-aeo-agencies-2026/
1
pages in cluster
/gemini-optimisation-agencies/
1
pages in cluster
Terminology Consistency (H1 Analysis)
2026
x (%)
seocompare
x (%)
agencies
x (%)
search
x (%)
best
x (%)
chatgpt
x (%)
agency
x (%)
visibility
x (%)
comparison
x (%)
side
x (%)
platform
x (%)
coverage
x (%)
compared
x (%)
optimisation
x (%)
perplexity
x (%)