Paste your robots.txt content below to instantly see its parsed directives and catch rules that could be blocking your whole site.
| User-agent | Directive | Value |
|---|
Robots.txt is a plain text file at the root of your domain that tells search engine crawlers which parts of your site they're allowed to access. A single misplaced rule — like an accidental "Disallow: /" — can silently block your entire site from being crawled and indexed.
Because the file has no visual interface, errors often go unnoticed until organic traffic quietly drops. Checking it directly is the fastest way to catch a crawling issue at the source.
It tells the specified crawler not to access anything on the site starting from the root. Under 'User-agent: *', this blocks every well-behaved crawler from your entire site — a common accidental cause of pages disappearing from search results.
It's not required, but it's a widely recommended best practice. It helps search engines find your sitemap file even if it hasn't been manually submitted through Search Console.
No. Robots.txt only controls crawling, not indexing. A disallowed page can sometimes still appear in search results (without a description) if it's linked from elsewhere. Use a noindex meta tag to reliably prevent indexing.
Well-behaved crawlers like Googlebot and Bingbot respect it, but robots.txt is a voluntary standard — it does not technically prevent access, only requests that crawlers stay away.
50+ free marketing, SEO, and utility tools. No sign-up, ever.
Explore All Tools