Doofinder indexing can stop even when its crawler is allowed through Cloudflare and the WordPress site remains fully accessible. In this case, the site’s page-caching system appended an HTML snippet to the Doofinder product feed, which was expected to return pure JSON. Adding a cache exclusion for API responses and manually triggering a new Doofinder index restored the product data.
Issue background
A WordPress site using Doofinder reported that product indexing had stopped. The last successful index had occurred several days earlier, and Doofinder support suggested checking Cloudflare rules and confirming that its crawler IP addresses were allowed.
The investigation initially focused on recent caching and performance changes because the site had recently changed its optimization configuration. The Doofinder crawler used the following user agent:
Doofinder/4.0 (+http://www.doofinder.com/)The crawler IPs provided during troubleshooting were:
- Europe:
54.171.4.216 - USA:
52.2.218.41 - Asia:
18.143.220.25
Although allowlisting was an important check, the indexing failure was ultimately not caused by Cloudflare.
Diagnosis
The Doofinder product feed was inspected directly. Instead of returning only valid JSON, the response contained an additional HTML snippet appended by the site’s page-caching layer.
Doofinder expects a machine-readable product feed. Even a small amount of HTML before or after the JSON payload makes the response invalid and prevents the indexer from parsing the catalog.
This explained why the public website appeared normal while Doofinder stopped updating. Standard page requests could tolerate the injected markup, but the API consumer could not.
The Cloudflare configuration and crawler access were reviewed, but no blocking rule was identified. The response format—not network access—was the root cause.
Resolution steps
- Confirm the indexing timeline. Check the Doofinder dashboard for the last successful index, the current index status, and any feed parsing messages.
- Verify crawler access. Review Cloudflare, firewall, security plugin, and host-level rules. Confirm that the Doofinder user agent and required IP addresses are not being blocked or challenged.
- Request the feed directly. Open the product feed or API endpoint outside the Doofinder dashboard. Use browser developer tools,
curl, or another HTTP client to inspect the complete response body and headers. - Validate the JSON response. Confirm that the response begins and ends with valid JSON and contains no banner, optimization marker, debug output, HTML comment, script, or page fragment.
- Compare cached and uncached responses. Bypass the cache or temporarily disable the suspected caching layer on a staging environment. If the feed becomes valid JSON, the caching system is modifying the response.
- Exclude API and feed endpoints. Add narrowly scoped exclusions so the page cache, HTML optimization, or performance plugin does not process the Doofinder feed or other JSON API responses.
- Clear all caching layers. Purge the page cache, object cache, CDN cache, and any server-level cache so the malformed response is no longer served.
- Retest the endpoint. Request the feed again and pass the response through a JSON validator. Confirm that the HTML fragment is gone.
- Trigger a manual Doofinder reindex. Start a new index from the Doofinder dashboard after the feed returns clean JSON.
- Verify catalog completeness. Confirm that product counts, availability, pricing, categories, and other indexed fields match the current WooCommerce catalog.
Cache exclusions should target only the affected API paths whenever possible. Disabling caching for the entire website may resolve the immediate problem but sacrifices performance unnecessarily.
Final outcome
After the caching system was prevented from modifying API responses, the Doofinder feed returned valid JSON again. A manual reindex completed successfully, and the product catalog began pulling into Doofinder correctly.
The key lesson is that third-party indexing failures are not always crawler-access problems. When an external service can reach a WordPress endpoint but cannot process it, inspect the exact response body. Page caches, optimization plugins, debug output, and injected HTML can silently corrupt JSON feeds while leaving ordinary frontend pages unaffected.
For help troubleshooting Doofinder indexing, malformed WordPress API responses, or caching conflicts with third-party integrations, contact Freshy.
