Sitemap XML: which URLs to include and how to check it?

9 min read

An XML sitemap must list the final, canonical and indexing URLs, not all the addresses produced by your site. To verify this, cross-reference your inventory with HTTP responses, indexing directives, and modification data. Submission comes next, when the file correctly describes the pages you want to make visible.

Remember: the sitemap is a discovery index, not a guarantee of indexing.^[21] Keep the useful and canonical versions, provide information lastmod only from a significant change, then check the processing of the file and the status of the priority pages.

Define the scope of the XML sitemap

Google specifies that a submission does not guarantee downloading of the sitemap nor its use to crawl URLs.^[21] Distinguish between the available file, the accessible page and the actually indexed page. Even if indexed, a page does not necessarily appear in every search.^[22]

The relevant scope includes the content you want to offer in the results: service pages, substantial editorial categories or public resources. It does not correspond to the history of URLs or to all the technical routes of CMS.

We propose an operational admission rule: retain a final URL that is accessible, indexable, consistent with the canonical version retained and useful to the target audience. This filter is a control method, not a new protocol requirement.

Build an exploitable canonical inventory

Start with the content published in CMS, then combine it with an exploration of the site and the existing sitemap. Spot missing pages, old routes, and campaign variants.

For each candidate URL, log:

• the absolute address and its page type;
• the status HTTP and the destination of a possible redirection;
• declared canonicality, guidelines noindex and exploration restrictions;
• the usefulness of the content, its responsible person and its publication status;
• the date of the last significant change, with its justification.

When multiple URLs provide access to the same content, Google recommends choosing the one you prefer and including it rather than all its variations.^[21] However, do not blindly delete each URL containing a parameter: analyze its function.

Also set your address conventions: domain, protocol, path and form used by the site. Keep the encountered URL and the final URL separately. This will allow you to explain why an old route disappears from the file without losing track of its replacement.

Deciding with a completed admission table

Explicitly fictitious example: the site atelier.example publishes services and maintains campaign URLs. The paths below are abbreviated for reading; XML entries should use absolute addresses.

The last line requires a business and technical decision: is the page a voluntary duplicate, or does its canonicality come from a bad model setting? Don't export a contradiction just because the CMS provides a URL. Other access controls remain necessary.

Add to the register the justification, the person responsible for correction and the expected proof. For a redirect, this proof will be the response from the destination; for voluntary exclusion, the decision to keep noindex. “Suspend” means to wait for this arbitration, not to automatically remove the page from the site.

Check the responses before producing the file

Apply the checks in this order, before each commissioning of a generator or after a model modification:

1. Resolution HTTP. Query the exact address. For your admission rule, wait for a direct answer 200. Replace redirected URLs with their eligible destination; discard deleted pages and deal with server errors.
2. Access and indexability. Check offline access, rules robots.txt, the robots tag and the header X-Robots-Tag. Content intended for indexing should not be exported with a conflicting exclusion.
3. Canonical consistency. Compare the final address and the declared canonical. If they differ, check the content and configuration before deciding which version to keep.
4. Actual usefulness. Open the page: a satisfactory code does not prove that an offer still exists or that the main content is present.

Check all URLs automatically for recoverable responses and directives, then manually review exceptions. A check limited to the home page does not validate other models.

Archive the tested export, the HTTP responses and the directives noted. A correction is technically validated when the new control confirms the expected behavior. This does not constitute proof of indexing by Google.

Produce clean XML and credible lastmod

The file must be encoded in UTF-8 and contain absolute URLs. XML values must be escaped: one character & in an address becomes & in the file.^[21] Test the document with an XML parser, not just by opening it in a browser.

Here is a usable entry for our fictitious example, without an unjustified modification date:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://atelier.example/services/audit/</loc>
  </url>
</urlset>

Google uses lastmod when this value is regularly accurate and verifiable. It should reflect a significant change to the main content, structured data or links, not a simple copyright update.^[21]

Therefore link this value to a documented event: revision of an offer, substantial modification of a resource or change of relevant links. The sitemap export date and the automatic saving date of CMS are not sufficient to establish this change.

If your tool does not distinguish a real modification from a technical operation, it is better to omit lastmod that broadcast a misleading date. Keep track of the remembered event so you can compare the field to the content. Google otherwise ignores priority and changefreq : Do not use them as indexing priority levers.^[21]

Organize submission without blind automation

Generate the file from the validated inventory, rather than by stacking several extensions whose rules contradict each other. Verify that posting, removal, and change of address update the affected entries.

A sitemap is limited to uncompressed 50 MB or 50 000 URLs. Beyond that, it must be cut; a sitemap index can group files together.^[21] Without reaching these limits, subsitemaps by model, such as services and resources, can facilitate your monitoring if their admission criteria remain the same. This division is an organizational choice, not a guarantee SEO.

Preferably place the file at the root. Without submission in Search Console, its scope is limited to descendants of its parent directory.^[21] Also check that the URL of the file returns XML, and not a HTML connection or error page.

Then submit the sitemap or its index to the Sitemaps report in Search Console. You can also indicate their address in robots.txt, for example Sitemap: https://atelier.example/sitemap.xml.^[21] Log submitted path and generator configuration.

Validate corrections with the correct reports

Start with the Sitemaps report: Google indicates its file accesses and potential processing errors.^[21] An XML error calls for a correction of the document; an inaccessible file calls for a hosting check or access restrictions. Resubmitting the same file without correcting the cause does not replace these checks.

Then consult the page indexing report. Analyze patterns, not just total unindexed URLs: not all duplicates and intentional exclusions require fixing. Totals may also differ from your inventory.^[22]

For a priority service page, use URL inspection and compare the known state of Google with the live test. This provides information on current access; it does not prove that the page is already indexed.^[22] To explore this case in more detail, consult our diagnosis of a page missing from Google.

Also note the canonicality retained by Google in the URL inspection.^[22] If it differs from your preference, check for duplicates and internal links before editing the builder. The sitemap expresses a preference, it does not impose your choice.^[21]

Keep proof of closure: corrected file, successful XML check, responses from affected URLs and report status after processing. If an eligible page remains missing, open a targeted diagnostic rather than a succession of submissions. No indexation period can be guaranteed.

Frequently asked questions

Can an orphan page appear in the sitemap?

Yes, if it meets your admission criteria. But its registration does not replace useful navigation: find out why no internal link serves it. For a public offer, add a contextual link from a relevant page and verify that visitors can find it. The sitemap completes this discovery, without solving editorial isolation on its own.

Should a URL be removed after a temporary outage?

Not automatically. For a URL already accepted, distinguish between temporary unavailability and lasting withdrawal. Note the incident, check the recovery of the expected response and avoid transforming this failure into a permanent exclusion. Do not modify lastmod to report a simple re-upload without a change in content. Reexamine the admission if the offer or its address actually changes.

Is removing the sitemap enough to hide a private page?

No. Google can discover an address through other links; its absence from the sitemap does not prohibit indexing.^[22] For confidential content, use appropriate access control, not the sitemap as protection. A public page that you want to exclude from results must have a directive noindex robot readable. Never export URLs containing access tokens or personal data.

Conclusion: validate before submitting

Your next action: return to the admission table and check a service page, its campaign variant and an old redirected address. Correct the generator rules, then validate the file before submitting it. If the CMS or the models produce conflicting signals, you can contact our web services and SEO to frame the necessary checks and corrections.

Sources

• Build and Submit a Sitemap | Google Search Central | Documentation | Google for Developers (accessed 2026-10-08)
• Page indexing report - Search Console Help (accessed 2026-10-08)