First, should this page be public?
Start with the exact URL and a clear decision that it should be in search. A preview, private document or unapproved page should not be opened to crawlers just to get traffic. For an approved page, record whether a signed-out visitor can open it, the final URL after redirects, and the response status. A login page, an error and a public page with a noindex rule are different problems; fix the first one that fails.
Google's three eligibility requirements
Google adds that meeting these requirements does not mean the page will be indexed; indexing is not guaranteed. [1]
| Requirement (Google wording) | How to check | Common cause of failure |
|---|---|---|
| Googlebot isn't blocked | Read robots.txt rules that apply to the path | A staging rule copied to the live site |
| The page works (HTTP 200) | Check the response status of the final URL | A redirect chain ending in an error, or a login wall |
| The page has indexable content | Check the page has real text and no noindex rule | A noindex meta tag or X-Robots-Tag header left over from preview |
Check noindex in both places
Google documents two ways to set noindex: a meta tag in the head, such as meta name robots content noindex (or name googlebot to target only Google), and an X-Robots-Tag HTTP response header with the value noindex or none. The header also works for non-HTML files such as PDFs and images. [2]
Check both. If the HTML no longer has noindex but the response still carries X-Robots-Tag: noindex, the edit did not remove the rule; a hosting or deployment setting is adding it. Ask whoever owns that setting to change it for the approved page only, not for previews that should stay excluded.
Why robots.txt can hide a noindex
Google warns that a page with noindex must not be blocked by robots.txt, because then the crawler never sees the rule and the page can still appear in results. [2]
Robots.txt is mainly for managing crawler load. Google says it is not a mechanism for keeping a page out of Google: a disallowed page can still be indexed if other sites link to it, shown without a description. Not every crawler follows robots.txt, and crawlers read its syntax differently. For anything private, use a password or other access control, not robots.txt. [3]
Worked example: a preview rule survived launch
Hypothetical: a public guide returns HTTP 200. Its HTML has no noindex tag, but the deployment adds an X-Robots-Tag: noindex header, and robots.txt blocks the guide path. Two of Google's requirements fail: Googlebot is blocked, and if it were not, it would read the header and drop the page.
Fix: change the deployment setting so the header is not sent for the approved guide, and narrow the robots.txt rule so it no longer covers that path. Leave previews protected. Then recheck the response header and the robots rule before asking Google to recrawl.
Verify, then wait
Google recommends the URL Inspection tool to see the HTML Googlebot received, and the Page Indexing report in Search Console to see pages where it found noindex. You can request a recrawl from URL Inspection. Google also says that, depending on the page's importance, it may take months for Googlebot to revisit. [2]
Record four states separately: publicly reachable, crawl allowed, no unintended noindex, and observed indexed. Leave the last one blank until Search Console shows it. An eligible page can still stay out of the index, and being indexed is not a promise of rankings or visitors.
Questions before you decide
Will blocking a page in robots.txt keep it out of Google?
Not reliably. Google says robots.txt is not a mechanism for keeping a page out of Google and that a disallowed page can still be indexed if linked from other sites. Use noindex on a crawlable page, or password protection.
How long after removing noindex will Google notice?
Googlebot has to recrawl the page first. Google says this may take months depending on the page's importance, and you can request a recrawl with the URL Inspection tool.
Sources & verification
Product details and prices can change. Check the linked provider before buying.
- Google Search Central: Google Search technical requirements Accessed 29 Oct 2026
- Google Search Central: Block Search indexing with noindex Accessed 29 Oct 2026
- Google Search Central: Introduction to robots.txt Accessed 29 Oct 2026
Sources link directly to providers. Product buttons may use separately labeled affiliate links. Read our disclosure.