The AI Bots That ~140 Million Websites Block the Most

We looked at the total number of websites blocking the bots. There are many ways to block bots with robots.txt, and this accounts for all of them including:

• Explicit blocks, where the bot is mentioned and disallowed
• General blocks, where all bots may be blocked
• Any instances where a directive allowed the bot, after blocking all bots

Caveats: this doesn’t include any other block types such as firewalls or IP blocks.

More top-trafficked sites began blocking AI bots in 2024, but the trend is decreasing towards the end of the year. It looks like the decrease mostly comes from generic blocks. The trend for AI bots themselves is increasing and I’ll show you that in a minute.

Here’s how it breaks down for each individual bot in different categories of websites. I was actually expecting news to be more blocked than other categories because there were a lot of stories about news sites blocking these bots, but arts & entertainment (45% blocked) and law & government (42% blocked) sites blocked them more.

The decision to block AI bots varies by industry. There can be a number of unique reasons for this. These are somewhat speculative:

• Arts and Entertainment: ethical aversions, reluctance to become training data.
• Books and Literature: copyright.
• Law and Government: legal worries, compliance.
• News and Media: prevent their articles from being used to train AI models that could compete with their journalism and take away from their revenue.
• Shopping: prevent price scraping or inventory monitoring by competitors.
• Sports: similar to news and media on the revenue fears.

For this measure, we’re looking only at cases where a particular bot is disallowed. It does not include any overall disallow statements or cases where only certain bots may be allowed. In these cases, website owners went out of their way to specifically block certain bots.

Again, GPTBot is the most targeted, followed closely by Common Crawl’s bot. Common Crawl data is likely used as a data source for most LLMs.

Here are the most blocked AI bots with websites specifically targeting them:

Here’s the data for the number of websites blocking them:

Here’s the data:

As you can see, AI bots are starting to be blocked by a lot more of the most trafficked websites.

The number of AI bots more than doubled in just over a year, from 10 in August 2023 to 21 in December 2024. More new entrants into the market mean more bots all using resources to crawl websites.

Claudebot had the fastest growth of any crawler in the last year.

Here’s the data:

It will be interesting to see how the block rate evolves as more and more of these crawlers start to use an ever-increasing amount of resources. Will they be able to fulfill that social contract with website owners and send them more traffic, or will they choose to keep that traffic for themselves?

I think if they go for the walled garden approach, more sites will end up blocking the bots and these systems will have to pay websites for access to their data, or the bots may end up breaking web standards and ignoring robots.txt blocks. There have been a few reports of some AI bots ignoring robots.txt blocks already, which sets a dangerous precedent.