## What is a Robots.txt File?
A `robots.txt` file lives in the root directory of your website (e.g. `example.com/robots.txt`) and instructs search engine spiders which paths to skip.
Key Syntax Rules - `User-agent: *`: Applies to all web crawlers. - `Disallow: /admin/`: Prevents crawling of administrative routes. - `Allow: /`: Grants access to public pages. - `Sitemap: https://example.com/sitemap.xml`: Declares your XML sitemap location.
Build and validate your crawler directives instantly with the [Robots.txt Checker & Builder](/tools/robots-txt-checker).