Google’s official guidance
What Google documents about faceted URLs
Google recommends preventing crawling of faceted-navigation URLs when they do not need to appear in Google Search or other Google products. If filtered URLs do need to be potentially crawled and indexed, Google recommends making their URL patterns consistent and warns that crawling them can consume server resources and slow discovery of useful URLs. The size and impact of this risk depend on the site and its crawl demand. [1]
For facet URLs, Google documents standard key=value parameters separated with an ampersand (&), consistent ordering when filters are encoded in the path, and no duplicate filters. It also recommends returning HTTP 404 for empty, duplicate, nonsensical, or nonexistent filter and pagination combinations. [1]
Google says URL fragments do not affect crawling of faceted filters. Its broader URL guidance says Google Search generally does not use fragments to represent different page content. A fragment can keep a UI state out of the crawlable URL space, but it is not a way to publish a separate search landing page. [1] [3]
Google describes robots.txt as a way to control which URLs crawlers can request, not a way to guarantee that a page is absent from Search. A disallowed URL may still appear if Google learns about it elsewhere. A robots meta noindex directive can only be read when the page is crawlable, so a robots.txt block can hide the noindex instruction from the crawler. [5] [6]
A canonical link is a preference signal for duplicate or very similar URLs, not a rule that guarantees Google will select that URL. Nofollow is also not a crawl block: a URL may still be crawled if it is linked from somewhere that does not mark the link nofollow. [4] [7]
Google’s crawl-budget documentation is aimed mainly at large or frequently updated sites and notes that most smaller sites do not need advanced crawl-budget work. For affected sites, unnecessary URLs can take crawler time that could otherwise be spent discovering new or updated useful pages; this is a possible consequence, not a guaranteed delay for every store. [2]
URL inventory worksheet
Estimate the address space before you tune it
List the independent facet groups, count their allowed values, and note whether each control is single-select or multi-select. For four single-select facets where one value or no value can be chosen, the theoretical number of filtered states is the product of (value count + 1) for each facet, minus one unfiltered state. Then multiply by the combinations of independent sort, view, and page-size settings that also create URLs.
Illustration only: a category with 4 product types, 30 brands, 12 colors, and 8 sizes would have (4 + 1) × (30 + 1) × (12 + 1) × (8 + 1) − 1 = 18,134 filtered combinations if each facet allows at most one selection. If 3 sort orders, 2 view modes, and 2 page sizes each produce a URL, the theoretical total across filtered and unfiltered states becomes 18,135 × 3 × 2 × 2 = 217,620 URL states.
These are sample inputs and a theoretical ceiling, not a count of a real catalog, search demand, crawl volume, or an SEO forecast. Replace them with the values and rules from the store. Impossible combinations, empty results, mutually exclusive values, and URL normalization can reduce the actual total; multi-select facets need a different combinations count.
Facet classification
Classify the state each control creates
Product type, brand, color, and size
Behavior: Changes which products appear in the result set.
Default: Candidate for evaluation; not automatically indexable.
These attributes can create a distinct selection, but only a subset is likely to be useful as a stable search destination. Judge each candidate against demand, product selection, and intent before allowing it into Search.
Sort order
Behavior: Reorders substantially the same products.
Default: Non-indexable by default.
“Price low to high” or “newest” is usually a browsing preference rather than a distinct search destination. Prefer not to expose separate crawlable URLs for each ordering.
View mode
Behavior: Changes presentation, such as grid versus list.
Default: Non-indexable by default.
A layout preference does not normally create a different product selection or a separate search task. Store it in the interface or user preference when possible.
Items per page
Behavior: Changes how many items are shown at once.
Default: Non-indexable by default.
Page-size settings often overlap the same inventory and create alternate renderings. Keep the setting from multiplying crawlable versions of a listing.
Price sliders
Behavior: Can create a large range of arbitrary minimum and maximum values.
Default: Non-indexable by default.
A slider can create many thin or one-off ranges. If evidence supports a valuable price intent, consider a small, deliberately maintained price band as its own landing page instead of indexing every slider position.
Stock and pickup availability
Behavior: May change with inventory, store location, or a visitor’s session.
Default: Transient availability and session states are non-indexable by default.
A visitor-specific or frequently changing state is rarely a durable landing page. A stable local pickup destination could be evaluated separately where it serves a distinct local intent; that decision depends on the store.
Session and tracking values
Behavior: Identifies a visit, campaign, or measurement event rather than a product selection.
Default: Do not index; avoid creating crawlable URL variants.
These values do not make a new search destination. Keep them out of internal browse links where possible and use the clean destination consistently for canonicals and sitemaps.
Unbounded user-entered values
Behavior: Can accept arbitrary text or values with no fixed catalog vocabulary.
Default: Do not index by default; avoid exposing an unbounded crawlable URL space.
Free-form values can generate endless URLs with no stable product set. Keep these as on-site interaction states unless a team deliberately curates and maintains a finite set of useful destinations.
Site-dependent recommendation
Decide whether a filtered selection deserves a search landing page
Treat indexation as a deliberate editorial and technical choice. A facet changing the product list is necessary for a distinct page, but it is not enough by itself. Review each candidate against all three criteria:
Genuine search demand
There is credible evidence that people search for this product selection or intent. Validate it against the store’s own search evidence and suitable keyword research; do not infer demand just because a filter exists.
A distinct, useful product selection
The filtered assortment is meaningfully different from the parent category and remains useful as products change. Empty, tiny, or nearly identical selections are weak landing-page candidates.
A matching search intent
The page helps someone who arrived for that specific selection, rather than merely displaying a temporary state from the filter interface.
Use examples such as 'Men's black running shoes', 'Women's waterproof jackets in size XL', and 'Black Nike Men's Running Shoes Size 10' as candidate intents to investigate, not as claims that those exact searches have proven demand. Some single-attribute facets may qualify; most multi-filter combinations rarely do.
For a valuable combination, create a stable dedicated landing page with a useful product selection, unique content that helps the shopper, and internal links from relevant category or navigation pages. Keep the page’s URL, canonical, and internal links aligned with the decision. Do not publish every possible combination just because a URL can be generated.
Treat sort order, view mode, items-per-page, price-slider positions, transient stock or pickup availability, session values, tracking values, and infinite user-entered variants as non-indexable by default. A site may have an exception, but it should be a separately reviewed, maintained destination rather than an automatic consequence of a control.
Site-dependent implementation choices
Match each control to the outcome you want
These controls have different purposes. Choose one policy per URL family after checking what a crawler can access and what the store wants in Search.
robots.txt disallow
Prevent crawling of URL patterns that do not need to be crawled.
Use only after checking that the pattern cannot include valuable pages you want crawled. This controls access, not guaranteed removal from Search; URLs discovered elsewhere may still appear without their page content.
robots meta noindex
Keep a crawlable page out of search results.
Use when the crawler needs to read the page-level exclusion. Do not block the same URL in robots.txt before Google has been able to see the noindex directive. Noindex does not stop the request from being crawled.
rel="canonical"
Signal a preferred representative for duplicate or very similar URLs.
Use a canonical when the filtered view is genuinely duplicate or highly similar to the intended representative. It is a hint, not a directive, and it is not a substitute for crawl control or noindex when those are the actual goals.
rel="nofollow" on links
Qualify links that point to a URL.
Do not treat nofollow as a crawl block. The URL can still be discovered through another link that is not nofollow, and Google’s faceted-navigation documentation describes it as a less effective signal than robots.txt or fragments.
URL fragments
Represent a UI state without creating a crawlable facet URL.
This can suit filters whose result states should not be separate Search destinations. Google generally does not use fragments as distinct page content, so do not use a fragment when the intent is to make a filter landing page discoverable and indexable.
HTTP 404 for empty or invalid combinations
Tell clients and crawlers that the requested result does not exist.
Return a real 404 for empty, duplicate, nonsensical, or nonexistent filter and pagination states rather than serving a successful category page or redirecting every case to one generic error URL.
Repeatable audit
Run the facet audit from inventory to release
Inventory generated patterns, not just example URLs
Start with a representative category and exercise every filter and display control. Record the base URL, query keys, path segments, accepted values, single- or multi-select behavior, whether values are bounded, and whether the result changes the product set or only its presentation.
Compare the observed patterns with internal links, a site crawl, server logs, and the application’s URL-building rules. Include tracking and session values, empty results, duplicate values, combinations in different orders, and text entered by a visitor. A short sample can reveal patterns, but record the rule that can generate the full family.
Estimate the possible URL growth
For each bounded single-select facet, count the allowed values and add one for “not selected.” Multiply those counts across independent facets, then subtract one if you want only filtered states. Add separate multipliers for sort orders, view modes, page sizes, or other settings that alter the URL. For a multi-select facet with n independent values, count its allowed subsets according to the actual UI rules instead of using the single-select formula.
Write down assumptions next to the result: whether combinations are mutually exclusive, whether empty results still get URLs, whether values can repeat, and whether the system normalizes order. Treat the estimate as a way to understand the address space, not as a forecast of how many URLs Google will crawl or index.
Classify each parameter or path segment by behavior
Mark whether a value changes which products are present, changes only order or presentation, is temporary or visitor-specific, or can take an unlimited number of values. Apply one classification to the URL family and document exceptions as named landing pages; avoid vague rules such as “all parameters are bad.”
Make URL generation deterministic
For query URLs, use ordinary key=value pairs joined with ampersands, such as /shoes?brand=nike&color=black&size=10. Keep parameter names and value encoding consistent. If a filter is represented in the path, always put attributes in the same logical order, such as /shoes/brand/nike/color/black/size/10. Reject or normalize duplicate filters so selecting the same value twice cannot create a second URL for the same state.
When a value is not part of the state that defines the page, avoid adding it to internal filter links. Keep campaign tracking and session handling separate from the stable destination URL. For any intended landing page, use the same chosen URL in internal links, canonical metadata, and the sitemap.
Choose one crawl and index goal for each family
Label each family as a potential search landing page, a crawlable page that should not be indexed, a URL that should not be crawled, or an invalid state that should return 404. Then select controls that support that goal. Robots.txt, noindex, canonical, nofollow, and fragments solve different problems; combining them without checking which signals remain crawlable can cancel out the result you intended.
Test representative URLs after release
Check at least one intended landing page, one redundant combination, one sort or display variant, one tracked URL, and one empty or invalid combination. Confirm the response status, rendered product selection, canonical, robots directives, internal links, and sitemap membership match the chosen policy. Recheck URL generation after a filter or catalog release so the rules still match the live interface.
References
Primary Google documentation
- Google Crawling Infrastructure: Managing crawling of faceted navigation URLs
- Google Crawling Infrastructure: Optimize your crawl budget
- Google Search Central: URL structure best practices
- Google Search Central: What is URL canonicalization?
- Google Search Central: Introduction to robots.txt
- Google Search Central: Robots meta tag specifications
- Google Crawling Infrastructure: Myths and facts about crawling