Here is an uncomfortable possibility. You may have spent money on your website, kept your Google Business Profile tidy, and started thinking about how to get recommended by ChatGPT, and none of it matters because your site is telling the AI to go away. Not on purpose. Usually it is a default setting somebody ticked, or a line of text in a file you have never opened. But the result is the same: ChatGPT, Perplexity and Google’s AI tools are not allowed to read your pages, so they cannot describe you, and they will never name you.
The good news is you can check this yourself in about two minutes, with no technical skills, and the fix is usually just as quick. This article shows you how.
Why a blocked site can never be recommended
AI search tools build their answers from what they can read on the open web. When someone asks “who is a good roofer in Geelong?”, the AI is pulling together what it knows about roofing businesses in Geelong from their websites, their reviews and their listings. If it cannot read your website, it has nothing to work with. It might know your name exists from a directory, but it cannot say what you do, where you work or why you are worth calling. So it recommends someone it can describe.
This is different from normal Google. Google has been crawling your site for years and you have probably never thought about it. The AI companies use their own crawlers, with their own names, and a lot of websites and hosting platforms started blocking those crawlers by default over the past couple of years. Many tradies are blocked without knowing it. We covered how AI decides who to name in who AI search recommends for Australian trades. Everything in that article assumes the AI can see your site in the first place. This is the step before.
The four crawlers you care about
Every AI tool sends out a program called a crawler (or bot) to read web pages. Each one announces itself with a name. The ones that matter for a trade business are:
- GPTBot, which feeds ChatGPT. There is also OAI-SearchBot, used specifically for ChatGPT’s live search results, and ChatGPT-User, which fetches a page when a user asks about it directly.
- ClaudeBot, which feeds Claude (made by Anthropic).
- PerplexityBot, which feeds Perplexity, the AI search engine.
- Google-Extended, which controls whether Google can use your content in its Gemini AI products. Note that Google’s AI Overviews in normal search use the standard Googlebot, so blocking Google-Extended does not remove you from AI Overviews, but it does limit you elsewhere.
If any of those names appear in a block rule on your site, that tool cannot read your pages.
Check 1: Your robots.txt file (60 seconds)
Every website has, or should have, a small text file called robots.txt. It sits at the root of your domain and tells crawlers what they are and are not allowed to read. To see yours, open a browser and type your website address followed by /robots.txt. For example:
yourbusiness.com.au/robots.txt
You will see a plain text page. It might be very short. Now look for lines that start with User-agent: followed by one of the crawler names above, and then a Disallow: / line beneath it. That pairing means “this bot is not allowed to read anything.” Here is what a blocked site looks like:
User-agent: GPTBot
Disallow: /
If you see that for GPTBot, ClaudeBot, PerplexityBot or Google-Extended, you are blocked from that tool. If the file just has User-agent: * with either no Disallow lines or only a few specific folders like /admin/, you are fine. If the page does not load at all (a 404 error), that also means nothing is blocked, because no rules exist.
Quick tip: on a phone, use your browser’s find-in-page feature and search for “GPT”. If nothing comes up, you are almost certainly not blocked at this level.
Check 2: Cloudflare’s AI bot switch (60 seconds)
This is the one that catches people out. Cloudflare sits in front of a huge number of Australian small business websites, often without the owner knowing, because their web designer or host set it up. In 2025 Cloudflare started blocking AI crawlers by default for new sites, and it offers a one-click “Block AI bots” setting. Plenty of web designers switched it on thinking they were protecting their clients. For a tradie who wants to be found in AI search, it does the opposite.
The catch is that a Cloudflare block does not show up in robots.txt. Your robots.txt can look perfectly clean and the AI crawler still gets turned away at the door. So you need to check both.
To check: log in to your Cloudflare account (or ask whoever manages your website to do this), choose your domain, then go to Security and look for Bots or AI Crawl Control, depending on your plan. You are looking for a toggle labelled something like Block AI Bots or AI Scrapers and Crawlers. If it is on, that is your problem.
Not sure whether you even use Cloudflare? The quickest tell is your domain’s nameservers. If they end in cloudflare.com, you do. You can also ask your web designer directly: “Is our site on Cloudflare, and is the AI bot block turned on?” That one sentence is often enough.
Other places a block can hide
Two more to be aware of. First, some website builders and hosting platforms have their own AI crawler settings. If your site is on Squarespace, Wix, Shopify or a managed WordPress host, look in the settings for anything mentioning “AI crawlers” or “AI training” and make sure it allows them. Squarespace, for example, has a setting under Crawlers that blocks known AI bots and it is worth checking where it sits.
Second, security plugins on WordPress (Wordfence and similar) sometimes have bot-blocking rules that sweep up AI crawlers. If your robots.txt and Cloudflare look clean but AI tools still seem unable to describe your site, ask whoever manages your WordPress to check the firewall logs for GPTBot or ClaudeBot being denied.
How to fix it
For robots.txt: remove the block rules for the AI crawlers, or better, explicitly allow them. A tradie’s robots.txt rarely needs to block anything beyond an admin folder. A clean version that welcomes all the AI crawlers looks like this:
User-agent: *
Allow: /
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
Sitemap: https://yourbusiness.com.au/sitemap.xml
Your web designer can make this change in a few minutes. If you manage your own WordPress site, an SEO plugin like Yoast or Rank Math lets you edit robots.txt from the dashboard.
For Cloudflare: turn the “Block AI Bots” toggle off. If you are on a plan with AI Crawl Control, you can allow specific crawlers individually, so allow the four listed above at minimum. Cloudflare also lets you set robots.txt rules from inside its dashboard, so check those do not contradict the file on your server.
Then re-do both checks. Reload your robots.txt and confirm the block lines are gone, and confirm the Cloudflare toggle shows as off. Changes take effect straight away, although the AI tools may take days or weeks to re-crawl your site and update what they know about you.
Should a tradie ever block AI crawlers?
The blocking trend came from publishers and artists who did not want their work used to train AI models without payment. That is a fair argument for a newspaper. It is a terrible trade for a plumber. Your website exists for one reason: to get you found and hired. You want every tool a customer might use to know exactly what you do and where. Blocking AI crawlers protects nothing of value and removes you from a channel more of your customers are using every month.
If you are still uneasy, remember what is actually on your site: your services, your suburbs, your phone number and a few photos of finished jobs. That is exactly the information you want repeated as widely as possible.
Once you are unblocked, make it count
Being readable is the entry ticket, not the win. Once the AI can see your site, it needs to find clear, plain-English pages that say what you do, who you do it for, where, and roughly what it costs. That is what gets you quoted in an answer rather than skipped over. Our pillar guide, how tradies get found in Google and AI search, lays out the full method, and AI SEO for tradies goes into the specifics of writing pages that AI tools like to cite.
And do not forget the last step. If the AI recommends you and the call rings out while you are on the tools, you have lost the lead anyway. Missed-call text-back or an AI receptionist closes that gap. We cover it in how to use AI in your trades business.
Frequently Asked Questions
How do I check if my website is blocking ChatGPT?
Type your web address followed by /robots.txt into a browser and look for GPTBot, ClaudeBot, PerplexityBot or Google-Extended paired with a Disallow: / line. Then check whether your site is behind Cloudflare and, if so, whether the Block AI Bots setting is switched on. Both checks take about a minute each.
My robots.txt looks fine. Could I still be blocked?
Yes. Cloudflare’s AI bot block, some website builders’ crawler settings and WordPress security plugins can all turn AI crawlers away without any sign of it in robots.txt. Cloudflare is the most common culprit for Australian small business sites.
Does blocking AI crawlers hurt my normal Google rankings?
Blocking GPTBot, ClaudeBot or PerplexityBot has no effect on your Google rankings, because Google uses its own crawler. Blocking Google-Extended does not affect normal search results or AI Overviews either. The damage is limited to AI search tools, but that is exactly the channel you are trying to grow.
How long after unblocking will AI tools start using my site?
The change is live immediately, but each AI tool has to come back and re-read your pages. Expect days to a few weeks before answers start reflecting your content. You can speed things up by keeping your sitemap current and making sure your key service pages are clearly written.
Not sure if your site is blocked?
We check it for you as part of a free audit, along with everything else that decides whether Google and AI search recommend you. Book a call and we’ll show you exactly where you are invisible, and how to fix it.
Book Your Free Call →