SITEMAP CRAWLER

Create XML, HTML & TXT Sitemaps from Any Website

Crawl your website to discover internal pages and generate XML, HTML, or TXT sitemaps. Choose the format you need for search engines, visitors, or simple URL lists.

Live Crawler { }XML Format </>HTML Format #TXT Format Instant Download
Understanding Sitemaps

What Are Sitemaps and Why Do They Matter?

A sitemap is a file that lists your website's pages. There are different types, each serving a specific purpose.

Three Formats, Three Purposes

XML sitemaps are primarily for search engines and can include optional fields such as last modified date, change frequency, and priority. HTML sitemaps are for human visitors — regular webpages with clickable links. TXT sitemaps are the simplest format — one URL per line, also for search engines.

Why XML for Search Engines?

XML sitemaps help search engines discover URLs and provide additional information such as when a page was last significantly modified.

Why HTML for Visitors?

An HTML sitemap page improves navigation and accessibility. It gives users a complete overview of your site structure in one place.

How the Crawler Discovers Pages

The crawler reads the HTML of your starting page and follows every internal link it finds, page by page. Pages that are not linked from anywhere on the site — or that only exist behind JavaScript or login walls — will not be discovered.

Sitemap Generator

Developed & Maintained by Bhavin J. Sheth | Last Updated on August 10, 2026

Optional XML fields: lastmod is most useful when it reflects the page's real update date. changefreq and priority are included for sitemap compatibility — Google ignores them.

Privacy note: your website is fetched from our server during crawling. URLs are processed for this request and are not intentionally stored by this tool.

What the Generator Does

What This Sitemap Generator Checks

The crawler follows internal links and builds a sitemap. Here's what it actually does.

Discovers Internal Links

Crawls your website starting from the URL you provide, following all internal links it finds in the HTML.

{ }

Generates XML Metadata

For XML sitemaps, optionally includes last modified date, change frequency, and priority fields.

</>

Creates HTML Pages

For HTML sitemaps, generates a complete webpage with clickable links for human visitors.

#

Plain Text Lists

For TXT sitemaps, outputs one URL per line — the simplest format search engines accept.

Download Ready

Provides copy-to-clipboard and direct file download for all three formats.

Customizable Settings

Control crawl depth and optional XML metadata fields for your sitemap.

Honest by Design

This tool crawls your website and generates sitemaps. Here is exactly what it does and does not do:

  • ✓ Crawls publicly accessible HTML pages and follows internal links from your starting URL.
  • ✓ Does not bypass login-protected pages and does not execute JavaScript-rendered navigation.
  • ✓ Generates valid XML, HTML, or TXT output — you still upload the file and submit it to search engines yourself.
  • ✓ Does not guarantee search-engine indexing — a sitemap is a discovery tool, not an indexing guarantee.
  • ✓ Does not generate image, video, or news sitemap metadata — those require manual creation following Google's guides.
Format Comparison

XML vs HTML vs TXT — Which One Do You Need?

Each format serves a different purpose. Choose based on who will use the sitemap.

Sitemap
Main purpose
Best for
XML
Help search engines discover URLs and read optional metadata.
SEO / search engine submission
HTML
Help visitors navigate your website from one page.
Users / site navigation
TXT
Provide a simple list of URLs, one per line.
Simple URL submission

What if my website has more pages than the crawl limit?

This generator crawls up to the selected limit per run (anonymous visitors: max 250 pages). For larger websites, run separate crawls from different starting pages, or use a dedicated sitemap system that supports sitemap index files. Remember: the crawler only discovers pages that are reachable through internal HTML links.

Quick Guide

How to Generate and Use a Sitemap

Four steps from crawling to submitting your sitemap to search engines.

1

Choose Format

Select XML for search engines, HTML for visitors, or TXT for a simple list.

2

Enter URL & Settings

Paste your website URL and choose the crawl depth. For XML, set the optional metadata fields.

3

Generate & Download

The crawler discovers pages and builds the sitemap. Copy it or download the file when ready.

4

Upload & Submit

Upload the sitemap to your website and submit its URL through Google Search Console or another search engine's webmaster tools.

Troubleshooting Guide

Common Issues and What They Mean

If the crawl fails or the sitemap looks wrong, here's what to check.

×

No Pages Found

CRAWL FAILED

The crawler couldn't access your starting URL.

Solution: Check that the URL is public and accessible. If your site blocks crawlers, the tool cannot access it.

!

Too Few Pages

LIMITED CRAWL

The sitemap contains fewer pages than expected.

Solution: Increase the crawl limit, or check that your site's internal linking is complete. JavaScript-rendered pages won't be discovered.

Missing Metadata

XML SETTINGS

XML sitemap lacks last modified dates or priority.

Solution: Select a custom date in the settings if you know the real update dates. Remember these fields are optional.

Timeout Error

SERVER LIMIT

The crawl took too long and was interrupted.

Solution: Reduce the page limit to 50 or 100. The tool returns the pages discovered so far when the time limit is reached.

Site Blocks the Crawler

ACCESS BLOCKED

Some hosts or firewalls block automated crawlers.

Solution: If the crawl returns no pages, check that your hosting or a security plugin is not blocking crawler user-agents.

JavaScript Pages Missing

JS LIMIT

Pages that only exist via JavaScript navigation are not discovered.

Solution: Make sure important pages have regular HTML links (<a href>) pointing to them.

Decision Guide

When to Use a Sitemap Generator — and What to Do Next

Use the generator when you need to create or update a sitemap. The next steps depend on the format.

When you use it
What the generator shows
What to do next
New website launch
XML sitemap with all discovered pages and metadata.
Upload sitemap.xml to your root directory and submit to Google Search Console.
After site redesign
Updated sitemap reflecting the new URL structure.
Replace the old sitemap file and resubmit to search engines.
Improve user navigation
HTML sitemap page with clickable links.
Create a new page (e.g., /sitemap) and paste the HTML code. Link to it from your footer.
Simple search engine submission
TXT file with one URL per line.
Upload sitemap.txt to root and submit to search engine webmaster tools.
Regular maintenance
Fresh sitemap after adding new content.
Regenerate when you add or remove important URLs, then replace the file on your server.
Frequently Asked Questions

Sitemap Generator FAQs

Answers based strictly on what this tool does and how it works.

?

Where do I upload the sitemap.xml file?

Upload it to your website's root directory so it's accessible at https://yourdomain.com/sitemap.xml. Most hosting control panels have a file manager for this.

?

What's the difference between XML, HTML, and TXT?

XML includes optional metadata for search engines. HTML is a webpage with links for visitors. TXT is the simplest — just URLs, one per line.

?

What do I do after uploading the sitemap?

Submit the URL (e.g., https://yourdomain.com/sitemap.xml) to Google Search Console and Bing Webmaster Tools. This tells them where to find it.

?

Why doesn't the crawler find all my pages?

The crawler only follows links it finds in HTML. Pages behind JavaScript rendering, login walls, or with no internal links won't be discovered. Check your site's internal linking.

?

Does this tool create image or video sitemaps?

No. Image, video, and news sitemaps require specific metadata that can't be reliably crawled automatically. Use Google's official guides to create those manually.

?

How often should I regenerate my sitemap?

Regenerate the sitemap when you add or remove important URLs or make significant changes to your website. If your site changes frequently, regular regeneration helps keep the sitemap current.

?

Will this guarantee all pages get indexed?

No. Sitemaps help search engines discover pages, but indexing depends on content quality, robots.txt rules, and other factors. A sitemap is a discovery tool, not an indexing guarantee.

?

Is my website data sent to your server?

Yes. The crawler fetches your website from our server to discover pages. The tool does not intentionally store the crawled URLs after the request.