WP Engine Blocks AI Crawlers, Causing 429 Errors for ClaudeBot and Others
Patch WP Engine firewall rules to whitelist current AI crawler UAs (ClaudeBot, GPTBot, Amazonbot) and remove outdated blocklist entries.
Update WP Engine firewall rules to whitelist current AI crawler UAs (ClaudeBot, GPTBot, Amazonbot) and remove outdated blocklist entries.
Summary
An investigation into why ClaudeBot and other AI crawlers were returning 429 errors on a WordPress site revealed that the issue was not Cloudflare or a security plugin, but WP Engine’s own firewall. The logs showed that 65.8% of bot traffic was AI bots, with ClaudeBot, GPTBot, Amazonbot, and Bytespider all receiving rate‑limit or blocked responses. Despite disabling Solid Security’s bot blocklist and checking Sucuri WAF, the 429s persisted. A series of curl tests confirmed that the block was triggered by a UA‑based rule in WP Engine’s platform, which only allowed a dated set of AI crawler UAs from mid‑2024. The older Anthropic‑AI UA and the Common Crawl bot were still permitted, exposing a gap in the blocklist. Cached responses were served to ClaudeBot, but cache‑miss requests hit the origin and were blocked, explaining the mixed 200/429 outcomes. The investigation highlighted that WP Engine’s firewall is the source of the problem, not Cloudflare or a plugin, and that the blocklist is out of date. Site owners must update the bot filtering rules to allow legitimate AI crawlers or contact WP Engine support for assistance.
Key changes
- WP Engine’s firewall blocked AI crawlers (ClaudeBot, GPTBot, Amazonbot, Bytespider) with 429 responses.
- The blocklist targets AI training crawlers from mid‑2024, ignoring newer UAs like the current Anthropic‑AI UA.
- Cached responses bypass the block, but cache‑miss requests are blocked, causing mixed 200/429 outcomes.
- Cloudflare, Solid Security, and Sucuri WAF were ruled out as the source of the block.
- The block is UA‑based, not path‑ or rate‑based, and originates from WP Engine’s platform layer.
- The older Anthropic‑AI UA and Common Crawl bot are still allowed, exposing a gap in the blocklist.