Robots.txt creator
Write a robots.txt file and check which URLs it lets a crawler fetch.
About Robots.txt Creator and Tester
A robots.txt file sits at the root of your site and tells crawlers which paths to leave alone. This builder lets you add groups for different user agents, list allow and disallow paths, add a crawl delay and point to your sitemaps. Presets cover WordPress, Shopify-style and Magento-style stores and a staging block. A tester then takes a path and a user agent and tells you whether the rules allow it, using the longest matching rule.
- Keeping admin, cart and search pages out of the crawl.
- Blocking a staging site completely.
- Checking a rule before it goes live and blocks something important.
- Pointing crawlers to your sitemap files.
How to use Robots.txt Creator and Tester
Step 1
Pick a preset or start with your own rules.
Step 2
Edit the user-agent groups and add your sitemap URLs.
Step 3
Test a few important paths, then download the file and upload it to your site root.
Good to know
- A robots.txt file does not hide anything. A blocked page can still be listed in search if other sites link to it. Use a noindex tag or a login for that.
- Google ignores Crawl-delay. Bing and some other crawlers read it.
- Do not block the CSS and JavaScript files your pages need, or search engines may misread the layout.
Common questions
How does the tester pick a winning rule?
The longest matching path wins, and if an allow and a disallow are equally long, allow wins. This follows the Google and RFC 9309 behaviour.
Where do I put the file?
At the top level of your domain, so it is reachable at yourdomain.com/robots.txt. A subfolder will not work.
Can I block only one bot?
Yes. Add a group with that bot's name as the user agent and list its rules.