Imagine if, back in 2004, someone had told B2B marketers to remove their site from Google search, the single biggest driver of online discovery at the time. Fast forward to today, and AI-powered assistants like ChatGPT, Gemini, Perplexity and Claude are becoming the new gateway to information. Blocking them is the modern equivalent of opting out of visibility in the future of search.
Last week, Cloudflare, one of the largest internet infrastructure providers, announced it would start blocking AI bots by default on new domains. If you use Cloudflare for DNS, firewall, or CDN services (and many companies do), your website might already be blocking AI bots without you realizing it.
And that could be a big problem.
Because in today’s AI-first discovery landscape, visibility doesn’t just mean being on Google. Imagine a buyer researching enterprise compliance software, they may start with a question in ChatGPT, compare vendors through a Claude summary, and click a Perplexity link to explore pricing. If your content isn’t accessible to those bots, you’re absent from that journey.
Cloudflare’s change is designed to give publishers power, which is good. But for B2B companies not already monetising content, putting up walls between your site and AI bots is a terrible move. These bots are not your enemies. They are your new discovery channel.
Think about your ideal buyer researching solutions. They are no longer just visiting Google or Bing, they’re asking ChatGPT, Perplexity, or Claude. Blocking those bots is like turning off your SEO without warning. And with Cloudflare’s new default, you might be doing that right now without knowing it.
Here are some of the bots you can now choose to block or allow, grouped by function to help you decide what stays open and what gets shut out.
General Purpose Crawlers |
AI Search Bots |
AI Training Bots |
|---|---|---|
|
|
|
Some of these bots are used for AI training, others for AI-powered search, some are multipurpose. But if you block all of them, you’re betting your future visibility on a rapidly shrinking slice of the web.
The big risk here is that those that don’t take action may be unwittingly handing the spotlight to their competitors. If your site isn’t part of the AI training data or retrieval layer, your voice, your brand, and your solutions just won’t appear.
Worse, your competitors might be there instead.
If you’re a marketer, web manager, or CTO using Cloudflare for anything, here’s what to do:
Decide which bots you want to allow (most B2B companies should start by allowing GPTBot, ClaudeBot, PerplexityBot, and Googlebot).
Use the Allow or Custom Rules to fine-tune your access preferences. A smart starting point: allow AI search bots (like Claude, Perplexity, and ChatGPT-User) for visibility while you could consider blocking or charging training-focused bots like GPTBot or CCBot to protect high-value content.
Bonus: You can monitor AI bot access via Cloudflare’s AI Crawler reports to see who’s been visiting and what they’ve been trying to read.
We’re entering a new era of search where generative AI sits between your content and your buyer. Don’t let technical defaults sabotage your discoverability. If you’re not selling content, your content should be selling you.
Blocking AI bots doesn’t protect your business. It just makes it harder to be found.
Make the bots your allies, or risk becoming invisible.
Want help planning how to maximise your visibility in the AI era? Drop us a note or book a meeting.
You can also check out our post on Navigating the Shift to AI Search..