This guide explains the concepts and practical checks behind the related Tolvix utility. Examples are illustrative; review the result in the service or platform you actually use.
Where to put the file
For a site at https://example.com, the usual location is https://example.com/robots.txt. A subdomain has its own robots.txt scope. A file inside /docs/ does not set rules for the whole hostname. Make sure the server returns the text file at the exact root path and that it is not blocked by a redirect or access rule.
Understand the directives
User-agent names the crawler group. Disallow asks that crawler not to request matching paths; Allow can specify accessible paths in supported implementations. Sitemap can point crawlers to an XML sitemap. Rule matching and wildcard support can vary, so use simple rules and test with the relevant search engine tools when available.
Do not block pages that need to be crawled
If a page is blocked from crawling, a crawler may be unable to see a noindex directive placed on that page. A blocked URL may still appear in search results based on external links or other signals. Decide whether you need crawl control or index control; they are different problems. Keep important CSS and JavaScript resources accessible when crawlers need them to understand the page.
Example: a cautious starting file
A small site might begin with a rule group like User-agent: *, Disallow: /admin/ and a separate Sitemap: https://example.com/sitemap.xml line. Replace these example paths with routes that actually exist on your own site. Do not copy a broad rule such as Disallow: / unless you intentionally want to ask all compliant crawlers not to crawl the entire hostname.
After uploading the file, open it directly at https://example.com/robots.txt and inspect the exact response. Test important public pages and resources, and remember that robots.txt is public guidance: it does not stop visitors or malicious bots from opening a URL.