Website diagnostics
Robots.txt Checker
Retrieve only the origin’s /robots.txt file with a strict body limit, then list basic Sitemap and User-agent directives.
Inspect robots.txt
Crawling controls are not indexing controls
robots.txt is primarily a request to compliant crawlers about which URLs they may fetch. It is public and is not access control. Blocking a URL does not reliably remove it from search results; an HTTP or HTML noindex instruction is an indexing control, but a crawler generally must be allowed to fetch the page to see it.