robots.txt
robots.txt is a public file in the root directory of a website that sets access rules for crawlers. The rules can apply to all bots or to a specific type of crawler.
What you can set in it
Allow and Disallow rules describe permitted and blocked paths. The file can also point to your sitemap. Not every crawler has to follow these rules.
Why it matters for visibility
An unintended block on a public part of your site can stop crawlers from fetching your content. Before you remove a block, find out which bot and which pages it affects.
robots.txt doesn’t protect private data and isn’t a reliable way to remove URLs from search results. Protecting content requires proper access control.
Why you should check a rule and a URL together
What a rule means depends on the group of crawlers and the path it applies to. Blocking a technical directory may be intentional, while a similar-looking rule that covers your service pages can stop them from being fetched. Evaluate specific public URLs instead of only checking that the file exists. After a site migration, check that no settings from a staging environment were left behind.
robots.txt rules apply to a specific protocol, host and port. The settings for one subdomain don’t automatically cover the others. After changing the rules, also check access to the page itself: a correct allow rule won’t help if the server still returns an error or requires a login. Access has several independent layers, and this file describes only one of them.
Before publishing a change, keep the previous version of the file and test a few representative URLs, including both your public offer and directories that are restricted on purpose. That way you catch an overly broad rule without guessing from a single line, and it’s easy to roll back a mistake.
Common mistake: using a block instead of removing a page from the index
A crawler can know a URL from links even when it isn’t allowed to fetch the content. That’s why a Disallow rule is not the same as telling a search engine not to show the page. Don’t keep secrets at a URL protected only by such a rule either, because the file is public. Treat indexing decisions and access protection as separate tasks, chosen to fit the purpose of each page.
Example
The rule “Disallow: /” tells the named bot not to fetch the entire site. “Disallow: /admin/” restricts only that section.
Key takeaway
Check specific rules instead of removing every restriction. Some blocks may be intentional and necessary.