miraks.biz

AI Crawler Control Guide: Allow or Block Each Bot

By Holifan, approved 2026-09-19. Drafted with AI from the sources listed below, checked against them automatically, with code examples tested, and approved by the site owner.

This guide is for web developers and site owners who want to control how OpenAI, Perplexity, and Google crawlers access their sites. You’ll learn how to allow or block search bots and training crawlers separately, handle user-triggered fetchers, configure WAF rules, and verify bot traffic.

Crawler Categories and Core Behaviors

AI crawlers fall into automatic crawlers and user-triggered fetchers. OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI. PerplexityBot is designed to surface and link websites in search results on Perplexity and is not used to crawl content for AI foundation models. Google uses general crawlers like Googlebot that always respect robots.txt rules for automatic crawling, and exception crawlers like AdsBot that may ignore the global * user agent. User-triggered fetchers such as ChatGPT-User act on behalf of users and are not used for automatic web crawling.

You can control automatic crawlers with robots.txt. Exception crawlers may need specific rules. User-triggered fetchers act on a user’s request, so they may not follow robots.txt. For example, Perplexity-User generally ignores robots.txt.

OpenAI Bots: Search, Training, Ads, and User Actions

OpenAI provides separate bots for search, training, and ads, and you can control each independently.

OAI-SearchBot is used to surface websites in search results in ChatGPT’s search features. Sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. To appear in search results, allow OAI-SearchBot.

GPTBot is used to crawl content that may be used in training OpenAI’s generative AI foundation models. Disallowing GPTBot indicates your site’s content should not be used for training generative AI foundation models.

OAI-AdsBot validates the safety of web pages submitted as ads on ChatGPT. It only visits pages submitted as ads, and the data collected by OAI-AdsBot is not used to train generative AI foundation models.

ChatGPT-User is used for certain user actions in ChatGPT and Custom GPTs. When users ask a question, it may visit a web page. It is not used for automatic web crawling.

Each robots.txt setting is independent. You can allow OAI-SearchBot while disallowing GPTBot. If you allow both OAI-SearchBot and GPTBot, OpenAI may use the results from just one crawl for both use cases to avoid duplicative crawling. For search results, it can take about 24 hours from a robots.txt update for OpenAI systems to adjust.

Here is a robots.txt example that allows search and ads but blocks training:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: OAI-AdsBot
Allow: /

Perplexity Bots: Search and User Requests

PerplexityBot is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models. To ensure your site appears in search results, allow PerplexityBot.

Perplexity-User supports user actions within Perplexity. When users ask Perplexity a question, it might visit a web page to help provide an accurate answer and include a link to the page in its response. It is not used for web crawling or to collect content for training AI foundation models. Since a user requested the fetch, Perplexity-User generally ignores robots.txt rules.

Perplexity states that it may take up to 24 hours for their systems to reflect robots.txt changes.

To allow only search crawling in robots.txt:

User-agent: PerplexityBot
Allow: /

To control Perplexity-User, use a Web Application Firewall (WAF). Perplexity recommends combining User-Agent and IP address conditions, using the official JSON endpoints for current IP ranges. For example, in Cloudflare WAF, create a rule that checks if the User Agent contains PerplexityBot or Perplexity-User and the IP Source Address is in the official IP list, then set the action to Allow.

Google Crawlers: General, Exception, and Identification

Googlebot is a general crawler that always respects robots.txt rules for automatic crawling. AdsBot is an exception crawler that may ignore the global robots.txt user agent (*) because of specific agreements between the site and the Google product.

Google crawlers identify themselves using the HTTP user-agent header, the source IP address, and the reverse DNS hostname of the source IP. They support HTTP/1.1 and HTTP/2, and by default crawl only the first 15MB of a file. If your site has trouble handling the request rate, you can lower Google’s crawl rate.

Here is a robots.txt example:

User-agent: Googlebot
Allow: /

User-agent: AdsBot
Disallow: /

Because AdsBot may ignore the global * rule, it is important to specify AdsBot separately if you want to block it.

Common Mistakes

Avoid these common errors:

How to Verify the Result

After you update robots.txt or WAF rules, verify that the right crawlers are allowed or blocked.

For Google crawlers, verify using the HTTP user-agent, source IP address, and reverse DNS hostname of the request.

For OpenAI bots, check the published IP address JSON endpoints for OAI-SearchBot, GPTBot, and OAI-AdsBot. Confirm that incoming requests match those IP ranges.

For Perplexity, configure WAF rules that combine User-Agent and IP from official endpoints, then monitor your logs to ensure legitimate bot traffic is allowed and unwanted traffic is blocked.

After updating robots.txt, wait up to 24 hours, then check search results or server logs to confirm the new behavior has taken effect.

Sources

Check your own site with the AI Crawler Access Checker.