Free tool
Check whether AI bots can see your site.
Enter a website address. I check robots.txt and how your server responds to ChatGPT, Perplexity, Claude, Google and other bots, then give you a ready robots.txt to copy.
Addresses you enter are not stored.
How to read the result.
robots.txt
The file with instructions for bots. I show whether each bot may access the homepage and which rule decides.
Server
For AI bots I send a request with their user agent. A rejection usually means a firewall or Cloudflare block. I do not test Googlebot, Bingbot or Applebot, because large sites reject anyone impersonating them and recognise the real bots by IP address.
Two groups of bots
Some search and cite pages in answers, others collect content to train models. You can allow the first and block the second.
Ready robots.txt.
Choose how you want to treat AI bots and copy the rules into your robots.txt.
Blocking Google-Extended also stops your content being used in Gemini app answers. It has no effect on Google Search or AI Overviews.
AI search bots vs model training bots.
When someone asks ChatGPT or Perplexity about a company in your field, the tool searches the web live. Bots like OAI-SearchBot, PerplexityBot and Claude-SearchBot do that work. Block them and AI cannot read your site, so it has nothing to cite.
The second group collects content to train future models: GPTBot, ClaudeBot, Google-Extended, CCBot. Blocking them does not remove you from AI search. It is a decision about whether your text can be used to train models.
User-triggered bots like ChatGPT-User and Perplexity-User are a separate case. They open a page when someone pastes a link or asks about a specific address. OpenAI and Perplexity say plainly that these visits may not follow robots.txt.
Should you block AI bots?.
If you want customers, it almost always pays to let AI search bots in. Otherwise you hand recommendations to competitors.
Training bots are a matter of choice. Publishers and authors often block them. Service businesses and stores usually neither gain nor lose much, so they can do whatever suits them.
The worst case is a block nobody knows about. It is often set by a plugin, a firewall or a default Cloudflare option rather than a deliberate decision.
robots.txt is fine, but bots still cannot get in.
robots.txt is only a request. A server, firewall or Cloudflare can reject bots regardless. If the Server column shows a block, check your hosting security settings or the AI bots section in your Cloudflare dashboard. The test sends requests with each bot's user agent but from a Cloudflare address, not from OpenAI or Google servers. A real bot may be treated differently, so take the server result as a hint.
Sources
- Overview of OpenAI crawlers OpenAI
- Perplexity crawlers Perplexity
- List of Google's common crawlers Google
- Anthropic crawlers Anthropic
- About Applebot Apple
AI bot questions.
How do I check whether ChatGPT can access my site?
Enter your address in the tool above. OAI-SearchBot matters most, because it decides whether you appear in ChatGPT search. If robots.txt or the server blocks it, ChatGPT cannot read the site.
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot collects content to train OpenAI models. ChatGPT search relies on OAI-SearchBot. You can block GPTBot and still appear in search-based answers.
Does Google-Extended affect Google rankings?
No. According to Google it is only a robots.txt token that controls Gemini training and use of content in Gemini apps. It has no effect on Google Search inclusion.
What is llms.txt?
A simple text file in your site root that describes the company in a few sentences and points to key pages. It is a proposed standard, not a requirement. It takes minutes to create, so there is little reason not to have one.
How quickly do AI bots notice a robots.txt change?
OpenAI and Perplexity say a change can take about 24 hours to take effect. Other bots read the file on their own schedule.
Want AI to recommend your company?.
Bot access is only the start. In a GEO audit I check what ChatGPT, Perplexity and AI Overviews say about your brand and who they recommend instead.