Written by 4SYSTEMPublished on Updated on
7 min read
A method for separating filters that help shoppers from pages worth crawling, without confusing crawl blocking with removal from the index.
To manage faceted navigation and SEO, first choose which filter combinations should be eligible to become search landing pages, then limit crawling of the rest. Do not let every colour, size, sort order and pagination option define your strategy by default. Google recommends preventing crawling of faceted URLs when indexing them is unnecessary, and carefully structuring those that should remain accessible to crawlers.[1] The task is therefore to assign a purpose to each URL family before choosing a technical rule.
Separate usability, crawling and indexing
A filter can help shoppers without deserving a place in search results. Crawling means a crawler visiting a URL; indexing concerns its inclusion in the index. Google explains that facets can generate a very large URL space, consume resources and slow discovery of useful content.[1] The aim is not to remove filters, but to control the pages they expose for discovery.
Start by listing parameters: product criteria, sorting, display, pagination and marketing tracking. Identify those that genuinely change the selection and those that merely reorder the same products. For the buying journey, retain clear shop navigation. This SEO decision concerns the URLs exposed to search engines, not the systematic removal of features that customers find useful.
Assign a policy to each URL family
The following grid helps document a decision, rather than applying the same directive everywhere. A page considered for indexing should address an identifiable intention and offer a usable selection. Also assess your ability to maintain its content and internal links. Accumulating almost identical pages is not an editorial strategy.
| Family | Objective | Treatment to define |
|---|---|---|
| Main category | Lasting entry point | Crawling allowed and consistent content |
| Selected combination | Answer to a distinct need | Stable URL, maintained page and relevant links |
| Sorting or combination without search value | Navigation feature only | Limit crawling according to the architecture |
| Impossible combination or non-existent pagination | Avoid creating a falsely useful page | HTTP 404 response at the affected URL |
Build rules without confusing their effects
Google distinguishes, among other options, robots.txt blocking and URL-fragment-based filters for facets whose indexing is not required. Its documentation regards canonical and nofollow signals as generally less effective over the long term for controlling this crawling.[1] Do not turn that guidance into an immediate catalogue-wide change: first check the current state and dependencies.
- Collect real examples for each URL family, including parameters, HTTP status, internal incoming links and filtering behaviour.
- Define a limited list of pages to retain as search destinations, then justify each choice by its usefulness and maintenance requirements.
- Choose the right mechanism for the other families, distinguishing reduced crawling from removal of an already indexed page.
- Standardise URL generation: consistent filter order in paths, no duplicate filters and the standard separator for parameters.
- Check category links, pagination rules and responses for empty combinations before extending the changes.
A noindex directive has to be read to take effect. Google specifies that a page blocked by robots.txt prevents the crawler from seeing that directive and can still appear in results if linked elsewhere.[2] If facets are already indexed, plan how to address that state before any later blocking. The guide to robots.txt and noindex helps keep the distinction clear.
Fictional example: available filters, selected pages
Entirely fictional example: a shop offers 8 colours, 6 sizes and 5 materials. If each criterion is optional and allows only one value at a time, the combinations already produce 378 possible states, including the unfiltered state. This illustrative calculation excludes sorting and pagination; it is not a measurement of a real shop’s URL count. Allowing several values per criterion would change the scope again.
The team selects “waterproof jackets” as a lasting destination because it corresponds to a product range and a clearly stated need. It does not automatically turn every size and colour into a search landing page. Filters remain available for shopping. An incoherent combination should receive the planned treatment for that case, without a redirect to a generic page that conceals the problem.
Check responses and discovery paths
In a validation environment, check the rules with allowed and blocked URLs, as well as similar variations. Google recommends the standard parameter separator, consistent ordering in paths and HTTP 404 status for combinations with no results, incoherent or duplicate filters and non-existent pagination.[1] Distinguish these URLs from a lasting category whose product range is temporarily affected: its policy deserves a separate decision.
- Check the HTTP status and directives actually served, rather than relying solely on interface settings.
- Verify that a rule intended for filters does not block categories, products or a necessary resource.
- Test different parameter orders and repeated values to identify duplicate URL generation.
- Then examine available logs and reports, separating deployed changes, new crawling and changes in indexing.
Frequently asked questions
Should all facets be blocked?
No. Some selections can make good landing pages if they address a distinct need and remain maintained. What matters is choosing them explicitly. For the rest, a filter’s usefulness to visitors does not establish the value of having every combination crawled.
Is a canonical enough to reduce crawling?
Do not treat it as a block. Google says it can gradually reduce crawling of non-canonical versions, while presenting other methods as generally more effective over the long term. Choose the mechanism according to your actual objective and check what it changes in the relevant architecture.[1]
Can blocking and noindex be combined without care?
No. If robots.txt prevents access to the page, the crawler cannot discover its noindex. Handling already indexed URLs therefore needs to be planned as a transition, with appropriate checks. A rule written in a file is not proof that removal from results has occurred.[2]
Make facets an architectural decision
Consistent faceted navigation and SEO depend on an inventory, a policy for each family and controlled URL generation. Start by protecting useful destinations and clarifying already indexed cases. To align this logic with your catalogue and shop’s technical constraints, 4SYSTEM services provide a basis for discussing the diagnosis and checks required, without promising a ranking outcome.