CompleteSEOTools
Home › SEO Tools › AI Crawler Blocker

AI crawler blocker

Choose which AI bots may read your site and get the rules to enforce that choice.

About AI Crawler Blocker

Many companies run crawlers that collect web pages to train or feed AI models. This tool keeps a list of the best known ones, with the company behind each. Tick the bots you want to stop, and it writes the matching lines for a robots.txt file, an Apache .htaccess file, an Nginx server block or a Cloudflare firewall rule. The bot list lives in this page and follows the crawler names the companies have published, but new bots appear often, so review it now and then.

  • Keeping your writing out of training datasets.
  • Cutting server load from aggressive scrapers.
  • Allowing search crawlers while refusing others.
  • Protecting only one folder, such as a members area.

How to use AI Crawler Blocker

1

Step 1

Choose whether the ticked bots are blocked or allowed, and set the folder to protect.

2

Step 2

Tick or untick the bots in the list.

3

Step 3

Open the tab for your server type, copy the rules and put them where the note says.

Good to know

  • A robots.txt file is a request. Honest crawlers obey it, but nothing forces a crawler to. Server rules refuse the request, though a bot that lies about its name can still get in.
  • Google-Extended and Applebot-Extended are permission tokens, not crawlers, so they only work in robots.txt.
  • Blocking GPTBot does not remove content that was already collected.

Common questions

Will blocking these bots hurt my Google ranking?

Not by itself. Googlebot is a different crawler and is not on the list. Blocking Google-Extended only stops use of your pages for Gemini, not search.

Why does the server rule skip some bots?

Google-Extended and Applebot-Extended never appear in a request header, so a server cannot match them. Use robots.txt for those.

How often should I update the list?

New crawlers appear every few months. Check the vendors' documentation a few times a year and regenerate the rules.