Technical SEO
What is robots.txt.
In short
A file in the root of a website that tells search engine crawlers and AI bots which parts of the site they may access. It blocks crawling, but does not guarantee a URL stays out of the index.
How it works
The robots.txt file lives at example.com/robots.txt. It contains groups of rules for specific crawlers (User-agent) listing paths they may (Allow) or may not (Disallow) crawl. It can also point to the sitemap. Well-behaved crawlers, such as Googlebot and bots from the big AI companies, follow these rules.
Example
User-agent: * means all crawlers, and Disallow: /cart/ blocks crawling of the cart. A separate group for User-agent: GPTBot with Disallow: / blocks the OpenAI training crawler without touching Googlebot. When a bot has its own group, Google applies only that group and ignores the rules for the asterisk.
What robots.txt does not do
- It does not remove pages from the index. A blocked URL can still appear in results, without a description, if links point to it. Use noindex to keep it out of results.
- It does not protect private data. The file is public and malicious bots ignore it.
- It has no effect on bots that do not respect it.
Common mistakes
- Disallow: / left over from a staging site, making the whole site disappear from Google.
- Blocking CSS and JavaScript files needed to render the page.
- Blocking a page that has a noindex tag, so Google cannot see it.
- Accidentally blocking AI search bots along with model training bots.