Bulk noindex checker
Paste a list of URLs and find the ones Google is told to leave out: a noindex in the HTML or in an X-Robots-Tag header, a robots.txt rule that blocks Googlebot, a canonical that points somewhere else, or a redirect.
What is checked, in the order Google meets it
- 1. Redirects and the status code
- Every HTTP redirect is followed, up to 10 hops, as Googlebot does. A URL that redirects is never indexed under its own address; the checks that follow are about where it lands. A final 404, 410, 4xx or 5xx keeps the page out.
- 2. robots.txt
- The destination site's robots.txt, read with Google's rules: the most specific group for Googlebot, the longest matching rule, and Allow on a tie. The rule that decided is shown. If robots.txt answers with a server error, Google stops crawling the whole site, so that counts as blocked.
- 3. noindex
- A
robotsorgooglebotmeta tag in the head, and theX-Robots-Tagheader, which is the only place a PDF or an image can say it. Directives addressed to other crawlers are ignored, as Google ignores them. - 4. Canonical
- From a
<link rel="canonical">in the head or aLinkheader. Pointing to another URL is a strong hint that Google should index that one instead. Two different canonicals cancel each other out.
Two traps worth knowing
- Blocked and noindex at the same time
- If robots.txt blocks a page, Google never downloads it, so it never sees the noindex. The URL can stay in the index from links, with no description. To get it removed, allow the crawl and leave the noindex in place until it drops out.
- Indexable is not the same as indexed
- A clean result here means nothing stops Google. It does not mean Google has chosen to index the page; that depends on quality, links and crawl priority, and only Google's own results can confirm it.
Questions
- Does it run JavaScript?
- No. It reads the HTML as the server sends it. A noindex or canonical added later by a script is not seen here. Google does render pages, but a noindex already in the served HTML stops it before rendering.
- Why could Google see something different?
- Some sites answer search engines differently, by User-Agent or by IP. This tool says who it is instead of pretending to be Googlebot, and fetches from Cloudflare's network. For a page that matters, confirm with URL Inspection in Search Console.
- Is there a limit?
- 500 URLs a run, checked two at a time so no site gets a burst. robots.txt is fetched once per site. One quick check before the first batch keeps the tool from being used to flood other people's sites.
- Is my list stored?
- No. Pages are read up to the end of their head, the findings go back to your browser, and the judging, filtering and CSV happen there. Nothing is kept.