AI Crawler Checker and llms.txt: Can AI Systems Access Your Site?

What AI crawlers are

AI companies run their own crawlers, just as search engines do. Some collect pages to train language models. Others fetch pages so an AI assistant can find and cite them when someone asks a question. They identify themselves with a user-agent name, and reputable ones follow your robots.txt file.

The difference matters. Blocking a training crawler keeps your content out of future training data. Blocking a search crawler can stop your pages from appearing or being cited in that AI product's answers. Google-Extended does not affect Google Search, and Google's AI Overviews are part of Search, crawled by the normal Googlebot.

robots.txt and llms.txt are different things

robots.txt: the rules crawlers follow

robots.txt sits at the root of your domain and tells each crawler which paths it may fetch. It is an internet standard (RFC 9309), and the major AI companies say their crawlers respect it. A crawler follows the group with its own name, or the * group if no group names it.

User-agent: *
Disallow: /admin/

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

In this example GPTBot is blocked everywhere, while every other crawler, including OAI-SearchBot and Googlebot, can crawl everything except /admin/.

llms.txt: a summary file, not a permission

llms.txt is a proposal from 2024 for a Markdown file at /llms.txt that summarizes your site and lists your most useful pages for language models. It is not a standard and it does not grant or block access. Google has said it does not use llms.txt, and no major AI provider has said that having one makes your pages more likely to be cited. It is cheap to add, but treat it as optional. It never overrides robots.txt.

How to check whether AI crawlers can access your site

1. Read your robots.txt. Open yourdomain.com/robots.txt. Look for groups that name AI crawlers, and for a Disallow: / under User-agent: *, which blocks everyone.
2. Test real URLs. Paste the file into the Robots.txt Tester, choose each AI crawler in turn and test your homepage, a few key pages and your sitemap. The tester follows RFC 9309: wildcards, $, the longest matching rule and user-agent groups.
3. Check your server and firewall. robots.txt can allow a bot while your host, CDN or security plugin still blocks it. Many hosts and CDNs have "bot protection" or "block AI bots" switches. If an AI product says it cannot open your page, check these settings first.
4. Look at your server logs. If AI crawlers are reaching you, their user-agent names appear in your access logs. Requests that get 403 errors are being blocked by the server, not by robots.txt.
5. Keep your sitemap linked. A Sitemap: line in robots.txt helps every crawler find your pages. Check it with the Sitemap Validator.

A browser tool cannot fetch another site's robots.txt for you, because browsers block that for security. That is why the tester asks you to paste the file. It also cannot see firewall rules, so step 3 always needs a manual check.

Common configuration mistakes

• Blocking search crawlers by accident. A blanket rule meant for training bots also catches OAI-SearchBot or PerplexityBot, so your pages stop being cited.
• A leftover Disallow: / from a staging site. This blocks every crawler, AI and search alike.
• Allowed in robots.txt, blocked by the firewall. The rules say yes, the server says 403.
• Thinking Google-Extended controls AI Overviews. It does not; Overviews come from Google Search.
• Expecting llms.txt to fix access. It does not change what crawlers may fetch.
• Using robots.txt to hide pages. Blocked pages can still be listed if others link to them. Use noindex for that.

Building the right robots.txt

If you want to decide crawler by crawler, the Robots.txt Generator lets you block training crawlers and search crawlers separately, warns you before you block something important and adds your sitemap line. If you manage many sites and want to run these checks without a daily limit, Pro removes the limit and the ads on every tool.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT search?

No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, which follows its own robots.txt group.

Do I need an llms.txt file?

No. It is an optional proposal, not a standard, and it does not control access. Your robots.txt, sitemap and the quality of your pages matter far more.

Will blocking Google-Extended affect my Google rankings?

No. Google-Extended only controls use of your content for training Gemini models. It does not affect Google Search or AI Overviews.

Why does an AI tool say it cannot open my page when robots.txt allows it?

Usually a firewall, CDN or host-level bot protection is blocking the request. Check those settings and your server logs for 403 errors.