If you are thinking about blocking ai crawlers from your website, hold that thought for just a second.<\/p>\n
As we saw at the latest Refinery89\u2019s Publisher At The Core<\/a> discussion session, AI crawlers are becoming a big concern for publishers. They use publisher content to gather information to train AI models and generate answers, often without permission or compensation. At the same time, blocking these bots entirely could reduce your visibility in search results, potentially affecting traffic and revenue. So, is blocking every bot really the right call?<\/p>\n Here’s the catch: different bots do different jobs. Some index your site for search, some ground an AI answer, and some crawl purely to train a model. Treat them as one item, and you’ll likely make the wrong decision in either direction: blocking traffic you actually wanted or allowing the exact crawling you meant to stop.<\/p>\n Before you fall into the temptation of blocking them altogether, let\u2019s take a step back and analyze the situation to decide whether it\u2019s worth banning bots from the lands of your website or letting them roam free.<\/p>\n <\/p>\n \u00a0<\/strong><\/p>\n In the past year, AI-powered search features have been reshaping discovery as we know it very quickly. Therefore, it has been taking a toll on publishers\u2019 traffic and ad monetization performance. A February 2026 Ahrefs study<\/a> found that when an AI Overview appears, click-through rates for top-ranking pages drop by 58% on average. That drop in clicks is also showing up in publisher revenue. Ozone data reported by Digiday<\/a> showed that ad inventory across publisher sites in the US and the UK fell by as much as 40% year-over-year in Q2 2026.<\/p>\n <\/p>\n \u00a0<\/strong><\/p>\n Publishers are already feeling the pressure of the new era of search, which is putting their visibility, revenue, and integrity at risk. The obvious fix to protect their site\u2019s content seems to be blocking bots. But this is not a one-size-fits-all solution since not all bots are used for training AI models or powering AI answers. Some search engine crawlers have the main purpose of helping index your page in search result, while others are used by demand partners to understand your ad inventory and maximize its value.<\/p>\n Not all bots serve the same purpose, and that distinction is what makes blocking risky. A single blanket block aimed at \u201cAI\u201d can end up catching everything at once, including the search crawler you still depend on for visibility.<\/p>\n \u00a0<\/strong><\/p>\n <\/p>\n Publishers are taking different approaches to this challenge, and they mostly depend on their size, growth strategies, and resources. During AdExchanger\u2019s Programmatic AI 2026<\/a>, Microsoft\u2019s VP of publisher product told publishers to let bots in and pursue licensing deals instead. This statement raised the opposite argument that blocking all AI bots first and then only allowing the relevant ones is what gives publishers the leverage to negotiate licensing deals. That strategy, however, tends to work best for publishers whose content is valuable enough that AI companies are willing to pay for access.<\/p>\n <\/p>\n <\/p>\n Blocking bots is not the unbreakable barrier it seems to be. As we explained in our blog on how to use robots.txt<\/a> as part of your website strategy, this is just a request and it\u2019s up to the bots to honor it or not. There\u2019s not 100% effective enforcement mechanism behind a disallow rule. Kind of like placing a \u201cDo not disturb\u201d sign rather than a locked door.<\/p>\n This detail is important because it changes the idea of what blocking accomplishes. A publisher can believe they\u2019ve shut a crawler out entirely, while it keeps crawling anyway, with no way to verify compliance. Compliance varies significantly by crawler, so a rule that works against one bot may do nothing against another wearing a different name. Let\u2019s see some ways to enforce the access decisions for your site.<\/p>\n <\/p>\n <\/p>\n New tools are being developed and updated to help publishers some control over which bots get access their content. However, none of them (at least not on their own, for now), will solve the whole issue. Here are some you can keep in mind:<\/p>\n \u00a0<\/strong><\/p>\n Activating this toggle means you will not receive traffic or impressions from Google\u2019s generative AI features (AI Overviews and AI Mode). However, this control is not a ranking signal for search results meaning you can still appear on the traditional search engine result pages.<\/p>\n \u00a0<\/strong><\/p>\n A robots.txt file tells search engine and AI crawlers which parts of a website they can or cannot access. It helps publishers control how their content is crawled and a decent bot will respect these instructions. However, as we mentioned earlier, it relies on compliant crawlers and is not a security measure that will protect your content from being scraped. You can read more about this in our blog.<\/a><\/p>\n <\/p>\n Cloudflare is an infrastructure provider used to improve website performance and security. It has been developing tools to help control bot access more efficiently. Realizing that their current solutions were too blunt due to the different types of bots that exists, the company decided to group them into three separate crawler categories:<\/p>\n Starting in September 2026, new Cloudflare sites will default to blocking Training and Agent bots. This only applies to pages with ads. Search bots will stay allowed, but crawlers like Googlebot fall into more than one category (Search and Training in this case), meaning they will be blocked. However, publishers can opt out of this. You can read the full announcement here.<\/a><\/p>\n <\/p>\n \u00a0<\/strong><\/p>\n Before writing any allow or disallow rules, check your server logs to see what bots are actually hitting your site and what their main job is (indexing, retrieval, or training) before deciding what to block.<\/p>\n \u00a0<\/strong><\/p>\n Tools that now separate search, agent, and training traffic let you block what doesn\u2019t add value. They so this without cutting off the search visibility you still depend on.<\/p>\n \u00a0<\/strong><\/p>\n Keep your CSS, JavaScript, and images crawlable, because search engines rely on them to understand how your page is built.<\/p>\n \u00a0<\/strong><\/p>\n A block only becomes a negotiating position if it’s paired with a licensing proposal. Otherwise, it’s traffic given up for nothing.<\/p>\nHow did we get here?<\/strong><\/h2>\n
Why you should think twice before blocking bots<\/strong><\/h2>\n
There\u2019s no industry consensus on what to do<\/strong><\/h2>\n
Why robots.txt isn\u2019t the wall you think it is<\/strong><\/h2>\n
How can publishers protect their content from AI bots?<\/strong><\/h2>\n
Google Search Console\u2019s AI opt-out toggle<\/strong><\/h3>\n
Robots.txt<\/strong><\/h3>\n
Cloudfare<\/strong><\/h3>\n
\n
Best Practices for Managing AI Crawlers and Protecting Publisher Revenue<\/strong><\/h2>\n
Audit your crawler logs<\/strong><\/h3>\n
Separate helpful bots from unwanted ones<\/strong><\/h3>\n
Do not block what your pages need to load<\/strong><\/h3>\n
Use blocking as leverage, not a goal<\/strong><\/h3>\n