Robots.txt GeneratorDeveloper Tools
Creates robots.txt rules for search and AI crawlers and tests URLs against them.
Create an XML sitemap from a list of URLs in your browser, or collect links from one live page with a server-side check. Nothing is stored.
One absolute URL per line. Blank lines and duplicates are ignored. All URLs should share one host.
changefreq and priority.
They stay in the XML for other crawlers. lastmod is the optional field that still helps most.
Optional. Fetches one page on our server and lists same-host links (up to 500).
Tick the box to show you are not a robot, then press Collect.
In list mode, the tool turns each absolute URL into a <url> entry and escapes special characters so the XML stays valid. All URLs must belong to one host, because a sitemap cannot mix domains.
In collect mode, the check runs on our server. The page address is used only for this check and is not stored. The server fetches one page, collects same-host links, and returns up to 500 URLs for the sitemap.
If the final file would go beyond 50,000 URLs or 50 MB, the tool splits the output into several sitemap files and creates a sitemap index.
sitemap.xml from a pasted list of page URLs.changefreq and priority. You can include them, but they should not be treated as ranking controls.Yes. A sitemap file should list URLs from one host. Mixed domains should be split into separate sitemap files.
You can, but search engines may ignore them. lastmod is usually the more useful field when it is accurate.
No. It fetches one page and collects links from that page only.
Often used together with the XML Sitemap Generator.
Creates robots.txt rules for search and AI crawlers and tests URLs against them.
Reads sitemap files, follows sitemap indexes, and exports the URL list with lastmod values when available.
Validated hreflang tags, Link headers, and sitemap entries.