Bot Spoofing Exposed: Only 6 of 33 AI Assistant Visits Verified
Validate bot requests by matching IP ranges against vendor lists.
Implement an IP‑range verification check for all bot user‑agents in your logs.
Summary
The author launched CitationIQ.com and discovered that the majority of bot visits reported in server logs were spoofed. Out of 33 requests claiming to be AI assistants, only six were verified against vendor IP ranges, while 27 were spoofed. The author also found that of 799 Googlebot requests, only 107 came from verified Google IPs, with 692 spoofed. Verification uses vendor JSON lists and IP range matching. Spoofed bots often target credential files like .env.production, indicating scanning activity. The author built a lightweight Python script that loads vendor JSON lists, extracts IP ranges, and checks each request’s IP against those ranges. The study highlights the prevalence of impersonated bots and the importance of IP‑range verification for accurate traffic analysis. Common Crawl bots frequently spoofed, requiring additional checks.
Key changes
- 33 AI assistant requests logged; only 6 verified via vendor IP ranges.
- 27 requests were spoofed, targeting sensitive files like .env.production.
- 799 Googlebot requests; only 107 verified, 692 spoofed.
- Verification uses vendor JSON lists and IP range matching.
- Spoofed bots often target credential files, indicating scanning activity.
- Script uses ipaddress, json, urllib.request to load ranges.
- IP‑range verification is essential for accurate traffic analysis.
- Common Crawl bots frequently spoofed, requiring additional checks.