What do search spiders see?
Fetch any page the way a basic crawler does and see its status, redirects, robots rules, canonical, headings, links and the text that exists before JavaScript runs.
- Checks robots.txt, meta robots and X-Robots-Tag
- Shows the raw HTML, before JavaScript
- No signup needed
Page title crawlers read
Enter a URL to start
- URLWaiting for inputThe address you entered
- Response—Status code and redirects
- Crawl rules—What robots.txt allows
- Index signals—noindex and canonical
Crawl report
Enter a page URL above to see the full report: response and redirects, robots rules, canonical, metadata, headings outline, every link and the text crawlers can read.
Fair-use limit: 20 crawls per hour and 100 per day for each visitor IP address. Pages larger than 2 MB are read up to the first 2 MB.
What a spider simulator shows, and what it can't
Crawlers read pages in steps. This tool shows the first step, which decides whether the rest can happen.
The raw HTML comes first
Every crawler starts with the HTML your server sends. Google renders JavaScript later, in a second pass, and many other crawlers and AI bots never do. Anything important that is missing from the raw HTML is at risk of being missed.
Crawling and indexing are separate decisions
robots.txt decides whether a page may be fetched. Meta robots and X-Robots-Tag decide whether it may be indexed. A page blocked in robots.txt cannot show its noindex tag, so it can still appear in results as a bare URL.
Canonical is a request, not a command
A canonical tag tells search engines which URL you prefer when content is duplicated. Google usually follows it, but it can choose another URL when the signals disagree, for example when the canonical redirects or is blocked.
Links are how pages are discovered
Crawlers follow normal links with an href. Links that only work through JavaScript click handlers, or that are marked nofollow, help much less with discovery. The links list shows exactly which ones a crawler can follow.
Crawler directives at a glance
The rules this tool reads, where each one lives and the mistake people make most often.
| Directive | Where it lives | What it controls | Common mistake |
|---|---|---|---|
| robots.txt Disallow | A text file at /robots.txt | Whether crawlers may fetch a URL. It does not remove a URL from results. | Blocking a page you want removed, so Google can never see its noindex tag. |
| Meta robots | A <meta name="robots"> tag in the page head | Indexing and link following for that page, for example noindex or nofollow. | A noindex left over from a staging site after launch. |
| X-Robots-Tag | An HTTP response header | The same rules as meta robots, and it also works for PDFs and images. | A server or plugin adding it to every page without anyone noticing. |
| Canonical | A <link rel="canonical"> tag in the head | Which URL should be shown when several URLs have the same content. Google treats it as a hint. | Every page pointing to the home page, or to a URL that redirects. |
| Hreflang | <link rel="alternate" hreflang> tags | Which language or country version to show each searcher. | Missing return links, so the other version does not confirm the pair. |
| Link rel values | rel="nofollow", "ugc" or "sponsored" on links | Hints that a link should not pass ranking signals. | Marking your own internal links nofollow, which hides pages from discovery. |
How this tool works
What happens between entering a URL and seeing the report.
The URL is checked
Only http and https addresses on ports 80 and 443 are accepted. The domain is resolved in DNS, and private, internal or reserved IP addresses are refused before anything is fetched.
robots.txt is read
We fetch the site's robots.txt and apply the group for Googlebot, or the group for all crawlers (*) when there is none. The most specific matching rule decides, as in Google's documentation.
The page is fetched like a crawler
Our server requests the page with its own user agent, AllInOneToolsSpider, without cookies. It follows up to 5 redirects, checks every new address again, and stops after 8 seconds or 2 MB.
The HTML is parsed, not rendered
We read the title, meta tags, canonical, hreflang, headings, links and visible text straight from the HTML. JavaScript on the page is never run, so you see what exists before rendering.
You get the report
The result goes straight to your browser and is not saved on our server. Every part can be copied or downloaded.
What this tool does, and what it doesn't
A plain list, so you know exactly what to expect.
This tool does
- Fetch one page and show its status and redirect chain
- Test the URL against the site's robots.txt
- Read meta robots, X-Robots-Tag, canonical and hreflang
- List the headings outline and every link with its rel values
- Show the text and HTML a crawler receives
- Export the report, links and headings as CSV
This tool does not
- Run JavaScript or render the page
- Pretend to be Googlebot or use Google's IP addresses
- Tell you whether Google has indexed the page
- Crawl a whole site or follow the links it finds
- Fetch private, internal or logged-in pages
- Store the pages or results you check
Where your data goes
Our server fetches the URL you enter and that site's robots.txt, identifying itself as AllInOneToolsSpider. The site you check sees a request from our server, not from you, and no cookies are sent. The page and the result go straight to your browser and are not stored; robots.txt files are cached for 10 minutes. No third-party service is involved. Like any website, our server may keep normal access logs, as described in our privacy policy.
Want the official rules? Google documents how its crawlers read robots.txt, noindex and canonical tags.
Why your results might look unexpected
The six surprises people run into most, and what each one means.
The page looks almost empty
The site builds its content with JavaScript, so the raw HTML holds little more than a script tag. Server-side rendering or static generation puts the content into the HTML itself.
Error 403 or a challenge page
A firewall or bot protection blocked our server. Googlebot is usually allowed through, so check the site's firewall settings rather than assuming Google is blocked.
Too many redirects
The URL redirected more than five times or in a loop. Point links straight at the final address so crawlers and visitors reach it in one step.
robots.txt could not be read
The robots.txt request failed or returned a server error. When that lasts, Google pauses crawling of the whole site, so fix it first.
The canonical points to another URL
The page asks search engines to index a different address. That is correct for duplicates, but a mistake on a page you want to rank on its own.
Google shows something different
Some sites serve crawlers, mobile visitors or certain countries different content. Compare with the URL Inspection tool in Google Search Console.
Questions about crawling and this simulator
Is this the same as what Googlebot sees?
Not exactly. This tool reads the raw HTML your server sends, which is what every crawler starts with. Googlebot also renders pages with JavaScript later, uses its own user agent and IP addresses, and some sites serve it different content. For Google's own view of a page, use the URL Inspection tool in Google Search Console.
Why is some of my content missing from the result?
Content that is added by JavaScript after the page loads is not in the raw HTML, so this tool cannot see it. If important text, links or headings are missing here, crawlers that do not render JavaScript will miss them too, and Google will only find them after rendering, which can take longer.
Why does a page work in my browser but fail here?
Firewalls and bot protection, such as Cloudflare or a security plugin, often block requests from servers they do not recognise, and they may answer with 403 or a challenge page. That tells you how the site treats unknown bots, not necessarily how it treats Googlebot, which most protection services let through.
Does a robots.txt Disallow stop a page from appearing in Google?
No. Disallow stops crawling, not indexing. A blocked URL can still appear in results, usually without a description, if other pages link to it. To keep a page out of results, allow crawling and add a noindex meta tag or X-Robots-Tag header instead.
Does the tool follow redirects?
Yes, up to five redirects, the same limit Google uses before giving up. Every step is listed in the report with its status code, and each new address is checked again before it is fetched.
Can I check pages on localhost, an intranet or behind a login?
No. For safety the tool only fetches public web addresses on ports 80 and 443, and it refuses private, internal and reserved IP addresses. It also sends no cookies, so pages behind a login show what a logged-out crawler gets.
Are the URLs I check stored?
No. The fetched page and the result are sent to your browser and not saved on our server. Only robots.txt files are cached for 10 minutes so repeat checks of the same site are faster. A hashed, temporary copy of your IP address is used for rate limiting and expires within 24 hours. Like any website, our server may keep normal access logs, as described in our privacy policy.