Overview
Our team resolved a complex SEO issue where a WooCommerce-based product catalog was generating tens of thousands of unique URLs through its filtering system. These URLs — produced dynamically when users applied multiple product filters — were being indexed by search engines, creating duplicate content concerns and drastically inflating the site’s page count.
The solution involved refining how the filter plugin handled URL parameters and implementing targeted robots.txt rules with wildcard support to prevent unnecessary pages from being crawled or indexed.
Issue background
The website’s product catalog used the Filter Everything Pro plugin to provide users with dynamic product filtering (e.g., by frequency, voltage, or range). However, each filter combination generated a unique URL such as:
/products/receivers/bwi-frequency-2-4-ghz/
/products/receivers/bwi-range-5000-foot/
As users applied multiple filters, these combinations produced thousands of unique paths. Over time, search engines indexed these URLs, leading to:
- Over 40,000 indexed pages detected in Google Search Console.
- Duplicate and thin-content issues impacting the site’s SEO.
- Slower crawl efficiency and diluted link equity across unnecessary URLs.
Diagnosis
The development team confirmed that the issue was caused by the Filter Everything Pro plugin’s method of generating clean URLs for each filter state. Instead of using query strings (e.g., ?filter=color:red), the plugin produced full URL paths, which search engines treated as unique pages.
A review of the plugin’s configuration revealed that indexing was already disabled in the plugin’s internal SEO settings — meaning the issue had to be addressed at the site level, not within the plugin itself.
Key goals identified during diagnosis:
- Prevent search engines from crawling dynamically generated filter URLs.
- Maintain proper indexing for main product category and product pages.
- Avoid breaking front-end functionality or user experience.
Resolution steps
- Audit and inventory filter-generated URLs: Using site crawl data, the team identified patterns in the filter URLs — many contained consistent prefixes such as
/bwi-frequency-or/bwi-range-. - Reviewed plugin-generated URL behavior: Developers confirmed that these URLs were produced by AJAX filter logic in Filter Everything Pro, which still created crawlable paths despite being designed for user-side filtering.
- Designed a robots.txt exclusion strategy: To ensure search engines would stop indexing these URLs, the following wildcard-based disallow rule was proposed:
User-agent: *
Disallow: */fepf*
This rule blocks all URLs generated by the filter logic while keeping primary category pages indexable.
- Added custom URL prefixes: A custom code snippet modified the plugin to prepend a unique prefix (e.g.,
fepf-) to all dynamically generated filter URLs. This made it easier to target and disallow these pages inrobots.txtwithout affecting legitimate product or category pages. - Updated robots.txt and verified exclusions: The updated file was deployed, and a new crawl test in Google Search Console confirmed that filtered URLs were now properly blocked.
- Preserved key product category visibility: The following key category pages were explicitly kept indexable:
/products/
/products/transceivers/
/products/analog-systems/
/products/receivers/
/products/wireless-e-stops/
/products/voltage-input-transmitters/
/products/handheld-transmitters/
/products/contact-input-transmitters/
Final outcome
After implementation, search engine crawlers began excluding filter-generated URLs from indexing. The website’s indexed page count dropped significantly, resolving duplicate content warnings in Google Search Console.
All core product and category pages remained visible and unaffected, and the filter functionality continued to perform seamlessly for users.
Key takeaways
- Dynamic filter URLs can unintentionally inflate indexed page counts — especially on large WooCommerce catalogs.
- Adding custom URL prefixes allows for precise control in
robots.txtand prevents blocking valid pages. - Balancing SEO and UX requires careful coordination between plugin configuration, server rules, and search engine crawling behavior.
Is your WooCommerce site struggling with excessive indexed pages or duplicate content issues?
👉 Contact Freshy — our experts can optimize your filters, indexing rules, and SEO setup to keep your site performing efficiently.
