Noindex and robots.txt checker: can Google index it?
Can Google index this page?
Paste up to 100 URLs. For each one the checker reads the robots.txt rule Googlebot follows, the robots meta tag, the X-Robots-Tag header and the canonical tag, then says in plain words whether Google can index the page and, if not, what stops it. A second mode tests any robots.txt file against search and AI crawlers.
Check your pages
Edit it, or paste a file of your own. Edits stay in your browser, so you can test a change before you publish it.
One per line, up to 50. Matching is case-sensitive and includes the query string.
What each result means
| Result | What it means |
|---|---|
| Noindex | The page tells Google to leave it out of search results. |
| Noindex Google can’t see | Has a noindex, but robots.txt stops Google from reading it. |
| Possible noindex in a merged header | A merged X-Robots-Tag header may hold a noindex for Googlebot. |
| Blocked by robots.txt | robots.txt stops Googlebot crawling the page. |
| robots.txt unavailable | robots.txt answers with an error, so Google pauses crawling the site. |
| Challenged by a firewall | The site’s firewall challenged our checker, so this may not be what Google gets. |
| Broken: 404 not found | The page is gone and Google drops it. Redirect it if the content moved. |
| Gone: 410 | Removed on purpose. Google treats it like a 404. |
| Blocked or broken (4xx) | A login wall, a bot block or a broken page. Google drops it like a 404. |
| Server error (5xx) | The server failed. Google slows its crawling and may drop the URL. |
| No response | Nothing answered: DNS, the connection or a timeout failed. |
| Looks like an error page | Answers 200 but reads like an error page. Google may call it a soft 404. |
| Canonical points elsewhere | Names another URL as the page’s main version. |
| Conflicting canonicals | More than one canonical, or the tag and the header disagree. |
| Nofollow | Google is told not to follow the page’s links. |
| Google can index this | Answers 200, Googlebot may crawl it, and nothing says noindex. |
Rows that redirect also show the redirect results, such as a redirect to the homepage. The redirect checker explains those.
How Google reads robots.txt
robots.txt tells crawlers what they may fetch. It controls crawling, not indexing. These are the rules Google applies (Google’s robots.txt specification, last updated August 31, 2026).
One file per host. The file lives at the root of each host and protocol, so https://www.example.com/robots.txt doesn’t cover https://shop.example.com.
One group per crawler. Googlebot obeys the group with the most specific user agent that matches it and ignores every other group. If the file has a group for Googlebot, the rules under User-agent: * don’t apply to Googlebot at all. Groups that name the same crawler are merged.
The longest rule wins. Inside the group, the rule with the longest path decides. When an Allow and a Disallow match with paths of the same length, Google uses the least restrictive one, so Allow wins.
Two wildcards. * matches any run of characters and $ marks the end of the URL. Rules are case-sensitive and match the path plus the query string, so Disallow: /search also blocks /search?q=shoes and /searching.
Only the first 500 KiB count. Google ignores anything after that point in the file.
The file’s status code matters. A 4xx response other than 429 counts as no robots.txt, so everything may be crawled. A 5xx or 429 makes Google stop crawling the site for the first 12 hours, then use the last good copy for up to 30 days. Google follows at least five redirects to find the file.
Google’s robots.txt tester is gone
Search Console used to have a robots.txt tester. Google retired it in November 2023 (Search Engine Land, published November 15, 2023) and replaced it with the robots.txt report, which lists the robots.txt files Google found for your site, when it last fetched each one and any problems it hit (Search Console Help: robots.txt report, read October 23, 2026). To check a single URL, Google points you to the URL Inspection tool. The tester above does the old tester’s job: paste or edit rules, test paths, and see the line that decides, for Googlebot and eight other crawlers and tokens.
Noindex in a meta tag or an HTTP header
In the HTML. <meta name="robots" content="noindex"> applies to every crawler. name="googlebot" applies only to Google. Google supports two user agent names in the tag, googlebot and googlebot-news, and ignores any other (Google’s robots meta tag specification, last updated March 24, 2026).
In the response headers. X-Robots-Tag: noindex does the same job and is the only way to noindex a PDF or another file that has no HTML. A header can name a crawler: X-Robots-Tag: googlebot: noindex.
When they disagree, the stricter rule wins. Google adds up the negative rules from every tag and header that applies to it. robots: nofollow plus googlebot: noindex means noindex and nofollow for Google.
Noindex takes effect on the next crawl. Google drops the page from its results once it recrawls the page and sees the rule (Google’s noindex guide, last updated December 10, 2025). Nofollow is a different rule: it tells Google not to follow the page’s links and doesn’t keep the page itself out.
Why blocking a noindexed page backfires
Adding a robots.txt block on top of noindex looks like extra protection, but it hides the noindex. Google can read a noindex only if it’s allowed to crawl the page. When robots.txt blocks the page, Google never sees the noindex, and the address can still appear in results (Google’s noindex guide, last updated December 10, 2025). Search Console reports these pages as “Indexed, though blocked by robots.txt”.
The checker flags this as Noindex Google can’t see. To keep the page out of Google, remove the robots.txt rule so Google can crawl the page and read the noindex. To have it rank, remove both.
Canonical tags
A canonical tag names the main version of a page: <link rel="canonical" href="https://example.com/page">. Google treats it as a strong signal, like a redirect, and still decides for itself which URL to show (Google’s canonical URL guide, last updated July 10, 2026). It only accepts the tag inside <head>. For a PDF or another file, send the same thing as a Link HTTP header. Don’t use robots.txt to pick a canonical.
After a relaunch, check that every canonical names the live page. A canonical that points at the staging host, the old domain or the homepage asks Google to index that page instead.
The crawlers in the robots.txt tester
| User agent | Run by | Kind | What it does |
|---|---|---|---|
| Googlebot | Search | Crawls for Google Search. It obeys the group that names it most specifically, and only that group. | |
| Bingbot | Microsoft | Search | Crawls for Bing search. |
| GPTBot | OpenAI | AI training | Collects pages that may be used to train OpenAI’s models. |
| OAI-SearchBot | OpenAI | AI search | Finds sites to show in ChatGPT’s search results. |
| ChatGPT-User | OpenAI | User request | Fetches a page when someone using ChatGPT asks for it. OpenAI says robots.txt rules may not apply, because a person started the request. |
| ClaudeBot | Anthropic | AI training | Collects pages that may be used to train Anthropic’s models. |
| Claude-SearchBot | Anthropic | AI search | Crawls to improve the search results Claude gives. |
| PerplexityBot | Perplexity | AI search | Crawls to show and link sites in Perplexity’s answers. Perplexity says it isn’t used to train AI models. |
| Google-Extended | Control token | Not a crawler: Google’s existing crawlers do the fetching, and this token tells Google whether it may use your pages to train Gemini models and for grounding. Google says it has no effect on inclusion or ranking in Google Search. |
Pages blocked that should rank?
Rescue writes a developer ticket for every indexing fix (noindex tags, robots.txt rules, canonicals), and our crawl confirms the whole site is indexable before you pay the second half. If you’re not sure that explains your drop, start with the free Cause Check: it also lines the drop up against Google’s updates and checks for AI answers.
The staging noindex that went live.
A new site is built on staging with noindex on every page, so Google won’t index it half-finished. The setting is still on at launch, and Google drops pages as it recrawls them. Launch Cover checks for it before launch and crawls the live site on launch day.
Launch CoverYou’re planning a redesign.
What breaks rankings in a redesign or migration, and the checks that catch it before launch.
Redesign without losing SEOOld URLs might be broken too.
Trace every old URL to where it lands now: broken pages, chains and redirects to the homepage.
Redirect checker
Noindex and robots.txt questions
How do I check if a page has a noindex tag?
Paste its URL above. The checker reads the robots meta tags and the X-Robots-Tag header and shows each one. By hand: view the page source and search for “noindex”, then check the response headers in your browser’s developer tools, because a header doesn’t show in the source.
What does “Excluded by ‘noindex’ tag” mean in Search Console?
Google crawled the page, found a noindex in its meta tags or headers, and left it out of the index. If the page should rank, remove the noindex, then use Request indexing in the URL Inspection tool. Google can index it again once it recrawls the page.
What’s the difference between noindex and nofollow?
Noindex keeps the page out of search results. Nofollow tells Google not to follow the links on the page. Either works alone. noindex, nofollow, or none, does both.
Does Google crawl noindex pages?
Yes. Google has to crawl a page to read its noindex, and it checks again on later crawls, which is how it notices when you remove the tag. That’s also why a robots.txt block hides the noindex.
What does “User-agent: * Disallow: /” mean?
It asks every crawler that has no group of its own to fetch nothing on the site. A crawler with its own group, such as a separate group for Googlebot, follows that group instead. On a live site, it usually means a staging robots.txt shipped by mistake.
How do I fix “Blocked by robots.txt”?
Find the rule that matches: test the URL in the robots.txt tester above and select the Googlebot cell to see the line. Remove or narrow that rule, publish the file, then use the robots.txt report in Search Console to ask Google to fetch it again.
Is robots.txt still used?
Yes. Google, OpenAI, Anthropic and Perplexity all document the robots.txt tokens their crawlers obey. It controls crawling, not indexing: to keep a page out of Google, use noindex and let Google crawl the page.
How do I block AI crawlers without leaving Google?
Give each AI crawler its own group, such as User-agent: GPTBot with Disallow: /, and leave Googlebot’s rules alone. To stop Google using your pages for Gemini, add a group for Google-Extended; Google says it doesn’t affect inclusion or ranking in Search. Test the file above before you publish it.
Do you store what I check?
No. Your list, the robots.txt files and the results stay in your browser. Our server fetches each page and robots.txt and sends back what it found, without saving your list or the results. robots.txt rules are evaluated in your browser.
More free tools
Redirect checker
Trace up to 100 URLs, or pull your old URLs from the Wayback Machine, and see which are broken, chained or sent to the homepage.
Traffic fell after a redesign, a new platform or a domain move.
Google update checker
Enter the date your traffic dropped, or load a Search Console export, and see which confirmed Google updates overlap. It runs in your browser.
Nothing changed on your site and the drop lines up with an update.