Find out whether the assistants can read your site.
Enter a domain. See how robots.txt treats each crawler behind ChatGPT, Claude, Perplexity and Google, and what a request as that crawler actually gets back. No account and no email address.
What this asks, and as whom
OAI-SearchBotClaude-SearchBotPerplexityBotGPTBotGoogle-Extended
One request per crawler, from this server, identifying as that crawler. The domain and the result are not written to a database, and no account is needed.
Selling physical products? Run the free product data check to see whether your product pages tell an assistant the same price your store charges.
What the check reads, and what it refuses to guess
Everything here is either a public file or one request this server made. Nothing about your business data is involved, which is why it is free and needs no account.
robots.txt, per token
Every assistant uses more than one crawler. The report reads the rules for each token and prints the line that decided it.
The wildcard group
A Disallow on the star group applies to every assistant that has no group of its own, which is the block most sites ship by accident.
A live request as the crawler
One request per citation crawler from this server, so a bot wall at the edge shows up as a refusal instead of staying invisible.
A browser baseline
The same page is requested first as an ordinary browser. If that fails too, nothing is blamed on the assistant.
Training switches
Google-Extended and Applebot-Extended are robots.txt switches, not crawlers. They are reported as a chosen opt-out, never as a blocked fetch.
The sitemap declaration
Whether robots.txt points the crawlers at a sitemap, which is how a new page gets found without a link.
Search indexing and assistant citation are two different doors
A search engine keeps a list of pages and shows links. An assistant keeps passages and quotes them in an answer. A site can be indexed for search and still be missing from every assistant answer, because the crawler that builds the citation store was blocked, throttled, or never allowed in.
The usual cause is not a deliberate block. It is a robots.txt written years ago for one search engine, a wildcard group that grew a Disallow line, a bot wall on a new host, or a training opt-out that the operator assumed was a citation opt-out. All three look the same from outside, which is why the report names the line instead of printing a number.
The part a public check cannot see is the part that decides whether the answer is right. An assistant can read every page on your site and still quote last quarter's revenue, because the real figure lives in your accounting system. CorpusIQ answers from those systems, inside ChatGPT and Claude, and names the source.
AI crawler access, answered
- What does this free AI crawler check actually do?
- It fetches your robots.txt, reads the rules for the crawlers each AI assistant uses, then sends one request to the homepage that identifies as that crawler. Every row names the robots.txt line that decided the answer and the HTTP status a request as that crawler received.
- Does a refusal mean ChatGPT or Claude is blocked from my site?
- Not on its own. The request comes from this server and identifies as the crawler, so a refusal proves this server was refused. The real crawler fetches from its own address range, and a site can refuse one and serve the other. A refusal is a strong signal that something at the edge is filtering automated traffic, which is worth checking in your own logs.
- Why does the report list several tokens for one assistant?
- Because they do different jobs. OpenAI answers a search question with OAI-SearchBot, trains on GPTBot and fetches a page a person asked about with ChatGPT-User. Anthropic and Perplexity split the same way. Blocking the training crawler does not block the citation crawler, and the reverse is also true.
- Is Google-Extended a crawler?
- No. Google-Extended is a robots.txt token that controls whether your site is used to improve Gemini and Vertex models. It is never sent as a user agent, so no fetch is made for it. Googlebot, which crawls for search, is a different token and is not affected by Google-Extended.
- Do you store the domain I check?
- No. The domain is used for that one request and the result is returned to your browser. The domain and the result are not written to a database, and no account or email address is needed.
- If the crawlers can read my site, is the AI answer about my business correct?
- Not necessarily. Being readable decides whether an assistant can fetch the page. Being right decides whether the number it gives is the number your systems hold. The second question needs your invoices, orders and inbox, which is what CorpusIQ answers from inside ChatGPT and Claude.
Readable is the start. Correct is the point.
Connect a source, ask a real question, and check the answer against the system it came from. The free trial includes the full report and the step by step fixes.
30-day free trial. No credit card required.
Connector directory