Build a robots.txt file without touching a line of code

It’s a small text file that sits at the root of your website and tells search engine bots which parts of your site they’re welcome to crawl, and which parts to leave alone. Think of it as a note left at your front door for visiting bots, not a lock. It’s a request, not an enforcement mechanism.

Your site will work fine without one, search engines will just crawl everything by default. But most sites have at least a few pages, like internal search results or admin folders, that add no value in search results. A simple robots.txt file keeps crawlers focused on the pages that actually matter.

Right at the root of your domain, not tucked into a subfolder. It has to load at yoursite.com/robots.txt exactly. A copy sitting at yoursite.com/files/robots.txt won’t be read by anything, no matter how correct its contents are.

Not always, and this trips people up. Blocking a page in robots.txt stops Google from crawling it, but if other sites link to that page, Google can still list the URL in search results without visiting it. If you need a page kept out of results completely, use a noindex tag on the page itself instead.

Disallow, in robots.txt, tells bots not to crawl a page. Noindex is a separate tag placed in a page’s own code that tells search engines not to list it in results, even if they do crawl it. If you disallow a page, Google can’t even see the noindex tag on it, so the two don’t combine the way people expect.

Search result pages on your own site can multiply endlessly, every search someone types creates a new URL. None of them offer anything a real page doesn’t already cover, and letting Google crawl thousands of them just wastes the time it could spend on your actual content.

It’s a good habit, and it costs nothing. Pointing directly to your sitemap from robots.txt gives search engines a fast, reliable way to find it, on top of whatever you’ve already submitted through Search Console. There’s no downside to including it.

Search engines will treat it as an open invitation and crawl your entire site. For a small, simple site that’s often not a problem at all. It only becomes worth fixing once you have folders, filters, or search pages you’d rather bots spent less time on.

Yes. Every rule in robots.txt applies to whichever user agent you name above it, so you can single out one bot by name, GPTBot for example, and give it different rules than everyone else. The custom user agent field in this generator’s advanced options is built for exactly that.

It asks a bot to wait a set number of seconds between each request to your server, which can help on a slow host getting hit hard by crawlers. The catch is that Google ignores it completely, it manages its own crawl rate through Search Console instead. Some other search engines and bots do still respect it.

Type yoursite.com/robots.txt into a browser first, you should see plain text, not a 404 page. From there, Google Search Console has a robots.txt report under Settings that shows the last version Google fetched and flags any parsing errors it noticed.

It’s rare, but yes, this is the one file where a small mistake has an outsized effect. A stray “Disallow: /” under “User-agent: *” tells every search engine to stop crawling your entire site. Always double-check the preview before you upload, and revisit yoursite.com/robots.txt afterward to confirm it looks the way you expect.