\n<\/p>\n
The web is being crawled more than ever, but it\u2019s no longer just search engines doing the crawling. AI bots are soaking up all the data they can to train and power their AI assistants.<\/p>\n
The problem is that AI bots are new to web crawling. They\u2019re still making some silly crawl decisions, bumping around, and wasting a lot of resources. We\u2019ve already seen that AI bots are starting to be blocked<\/a> by a lot more of the most trafficked websites.<\/p>\n I have no doubt the AI bots will figure things out, but it will likely be a painful process. I don\u2019t think any of them have even started rendering content yet.<\/p>\n There was a lot of blowback on Bing in the past because of their aggressive crawling. They fixed it, started IndexNow<\/a>, and now they may be the most efficient web crawler.<\/p>\n Check out IndexNow<\/p>\n If you haven\u2019t set up IndexNow for your website, you should.<\/p>\n If you\u2019re a CDN, CMS, an AI bot, or Google, you should also be getting in on this. Save resources, save money, and save the planet. It\u2019s good for you, it\u2019s good for your users, it\u2019s good for website owners, it\u2019s good for website hosts, and it\u2019s good for the web and the planet.<\/p>\n It\u2019s estimated that 53% of crawler traffic is wasted effort<\/a>, and IndexNow can help cut that number down to almost nothing.<\/p>\n<\/div>\n<\/div>\n By looking at data from Cloudflare Radar<\/a>, which reflects a massive slice of internet traffic, we can see how AI bots compare to search engine bots and SEO bots. Cloudflare handles ~20% of all internet traffic<\/a>, so this is a large and representative dataset for overall web crawling.<\/p>\n Let\u2019s dive\u00a0in.<\/p>\n