← All guides

SEARCH AND CRAWLERS

Can you block AI training without blocking AI search?

For ChatGPT Search, yes. You can allow OpenAI’s search crawler while keeping a training opt-out. Start with the public service page you want prospects to find. Check which crawler your site addresses and what the page gives a reader once it arrives.

Check which crawler your policy addresses

OAI-SearchBot helps surface websites in ChatGPT Search. GPTBot crawls content that may be used to train OpenAI’s models. The two robots.txt settings are independent. OpenAI also documents ChatGPT-User for certain visits triggered by a user, where robots.txt rules may not apply. OpenAI’s crawler guide.

Imagine a repair company wants its commercial oven repair page discoverable and chooses to disallow GPTBot across the site. These example rules express that choice for /services/oven-repair.

User-agent: OAI-SearchBot
Allow: /services/oven-repair

User-agent: GPTBot
Disallow: /

Ask your web or hosting owner to check both bot names against the exact path in your existing robots.txt, including any restrictions you want to keep. In AI Website Check, Crawler preferences (T02) records the declared policy it could read. A Review status can reflect a deliberate restriction, such as a training opt-out.

If the scanner receives 403 Forbidden without a recognisable challenge marker, it records the response and leaves the cause unknown. Give your hosting team the URL, time and response so they can check actual requests from the intended crawler. Search appearance depends on the provider’s systems.

Read the service page the way a simple fetch does

Suppose the repair page initially contains an empty <main id="app"></main> and JavaScript fills it with the service details. If the scanner extracts no main text, Content in initial HTML (T09) flags that gap. A reader that relies on the first response could miss what you repair or where you work. Google Search has a separate rendering stage, and readers vary in their ability to run JavaScript. Google’s JavaScript SEO guide.

Open the page source in your browser and look for a distinctive phrase from the service description in the HTML body. The repair company might say, “We repair commercial ovens for cafés in Helsinki.” If the phrase appears only after the browser runs JavaScript, ask your developer to include the essential copy in the initial response. Repeat the check on the same URL and look for the sentence in the new evidence. A pass tells you the scanner extracted some text. Read the excerpt to see whether it actually names the service, audience and area.

If the same page has <a href="/contact">Request a quote</a>, the scanner can record a link candidate under Contact facts discoverable (B04) while the criterion stays Not tested. Open the destination yourself and see whether a prospect can tell how to request a quote. If you need to verify form delivery or booking, arrange a separate test on a route you control. Form usability and contact completion are outside the free check’s scope.

Use the report to give someone one task

For the repair page, start with whoever owns the first gap you can verify. If that is the HTML, send your developer the service URL, the report excerpt and the sentence that should be visible. Agree to run the same check after the change and confirm the sentence is present.

The free pilot inspects the initial HTML of one public page and limited discovery metadata. Its report lists 38 criteria, with up to 11 evaluated checks depending on the evidence available. A Not tested result can mean the scan lacked evidence or the check is outside its scope. Assessment methodology.