How to create a sitemap for a website and help search engines find it

What a missing or broken sitemap costs

A page that nothing links to, or one buried several clicks deep, can wait a long time before a crawler reaches it. A sitemap is an XML file that lists the pages you want crawled, each with an optional lastmod date, so search engines don't have to discover them link by link.

It only helps when crawlers can find and read it. The usual mistakes are a sitemap at an address that robots.txt never mentions, one file that mixes URLs from two domains, and changefreq and priority values filled in as if they raised rankings. Google ignores both fields. One file holds at most 50,000 URLs, so a large site needs several files and a sitemap index.

Build the file, link it, then check it

Build the file in the XML Sitemap Generator. Paste a list of absolute URLs, or use collect mode to fetch one page and pull up to 500 of its same-host links. Collect mode reads that one page only, doesn't follow links across the site, and misses links that JavaScript adds later. All URLs must belong to one host. If the file would go over 50,000 URLs or 50 MB, the tool splits it and writes a sitemap index. Upload the result to the root of your site.

Next, tell crawlers where the file is with the Robots.txt Generator. Enter your sitemap URL and it adds a Sitemap: line to the file, next to any Disallow rules for private folders. robots.txt lives at the root of the host, such as https://example.com/robots.txt, and each subdomain needs its own.

Then check the result in the Website Sitemap Extractor. It reads robots.txt, tries /sitemap.xml and four other common paths, and lists every sitemap it finds with its type, URL count, and newest lastmod. A sitemap that returns broken XML stays in the table with the reason in the Error column, so you can see which file needs fixing.