Sitemap and Robots.txt Basics

A sitemap lists important URLs that you want search engines to discover. A robots.txt file gives crawling instructions and can point to the sitemap.

Sitemap role

Sitemaps do not guarantee indexing, but they help search engines find pages, especially on new sites with few external links.

Robots.txt role

Robots.txt can allow or disallow crawling. A mistake in robots.txt can block important pages, so it should be simple and carefully checked.

Best practice

Include core pages, guides, and policy pages in the sitemap. Avoid listing broken or redirected URLs.