Robots.txt is a text file at the root of a host that provides crawling instructions to compliant crawlers. It can help manage crawler access to parts of a website, but it is not a security mechanism and does not guarantee that a URL will stay out of search results.
Key Concepts
The User-agent field identifies the crawler a rule targets. Disallow can prevent crawling of specified paths, while Allow can provide exceptions where supported. Syntax and rule interpretation may vary between crawlers.
Practical Steps
Common mistakes include blocking the entire website with Disallow: /, blocking CSS or JavaScript needed to render pages, and treating robots.txt as protection for private content. Place the file at the correct host root, review it after changes, and protect sensitive information with authentication.