⚙️ DevOps & Platform Eng

Your Site's Silent Bandwidth Thief: Blocking AI Crawlers Before They Drain You Dry

Imagine opening your hosting bill to find AI bots ate half your quota. One Reddit post exposed Meta's crawler hammering a site 7.9 million times—now devs everywhere are scrambling.

DevTools Feed Apr 07, 2026 4 min read

``` It's tiny—under 2KB—GDPR clean. But pair it with raw logs. Human traffic baseline sharpens; bot deltas glow red. Plausible's close cousin. Hosted from $9/month, or self-host free. Even sleeker dash. API for scripting alerts. ``` ``` Fathom? Polished, $15/month start, no self-host. Rock-solid. None solo-snag server-hammering crawlers. But that human delta? Gold for spotting invasions early. | Feature | Umami | Plausible | Fathom | | --- | --- | --- | --- | | Self-hosted | Yes | Yes | No | | Open source | Yes | Yes | No | | GDPR (no cookies) | Yes | Yes | Yes | | Free tier | Self-host | Self-host | No | | Hosted pricing | N/A | $9/mo | $15/mo | | API | Yes | Yes | Yes | Quick comparison shows why devs flock here over GA bloat. ## Does robots.txt Actually Stop AI Crawlers—or Just Waste Your Time? It's a polite note on your door: "AI bots, stay out." ``` User-agent: Meta-ExternalAgent Disallow: / User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / # Good bots welcome User-agent: Googlebot Allow: / User-agent: bingbot Allow: / ``` Meta claims respect. OpenAI's GPTBot sometimes listens. But others? Laugh it off. Damage piles before rules bite. Plus—unique angle here—remember the Great Web Scraping Wars of the 2010s? Sites begged via robots.txt; scrapers ignored, built empires on pilfered data. LinkedIn sued hiQ; courts split hairs on CFAA. Today? AI firms scale that chaos 1,000x, no subpoenas needed. History whispers: voluntary blocks fail against hunger this big. ## The Real Wall: Server Configs That Actually Say No Nginx first. Map user-agents, slam 403s. ``` # /etc/nginx/conf.d/block-ai-crawlers.conf map $http_user_agent $is_ai_crawler { default 0; ~*Meta-ExternalAgent 1; ~*GPTBot 1; ~*ClaudeBot 1; ~*CCBot 1; ~*Google-Extended 1; ~*Bytespider 1; ~*Amazonbot 1; ~*anthropic-ai 1; ~*Applebot-Extended 1; } server { if ($is_ai_crawler) { return 403; } } ``` Apache? .htaccess rewrite. ``` RewriteEngine On RewriteCond %{HTTP_USER_AGENT} (Meta-ExternalAgent|GPTBot|ClaudeBot|CCBot|Google-Extended) [NC] RewriteRule .* - [F,L] ``` (Yeah, incomplete in source—finish with your full list.) Layer Cloudflare? Rules tab blocks by agent string. Free tier handles it. Rate-limit extras: 429s for politeness. But wait—architectural shift brewing. These aren't bandaids; they're portents. Web's shifting to authenticated access by default. Paywalls for crawlers? OAuth for bots? Indies can't wait for that; block now, watch standards evolve. Test ruthlessly. Curl your site with fake agents: ``` curl -A "Meta-ExternalAgent" https://yoursite.com ``` Expect 403. Tweak till solid. ## How Bad Is the Crawler Zoo—and Who's Worst? Meta-ExternalAgent leads pack. GPTBot ubiquitous. ClaudeBot sneaky. Bytespider (TikTok), Amazonbot, Applebot-Extended—all hoover for LLMs. Costs? Small site: $50-200/month extra. Scale up, thousands vanish. Performance? Humans flee slow loads—SEO tanks. My prediction: By 2025, browsers bake bot-blocks native. Like adblockers rose against trackers. But till then, you're the gatekeeper. Corporate spin? AI giants cry "public data fair game." Bull. They built no opt-in; just feast. ## Monitoring the Trenches: Alerts That Wake You Logs alone? Passive. Pipe to ELK stack or simpler: GoAccess for real-time. Script cron jobs scanning /access.log for agent matches—Slack pings on spikes. Umami/Plausible APIs feed dashboards. Anomalies auto-highlight. Pro tip: Block *before* analytics. Cleaner data forever. --- ### 🧬 Related Insights - **Read more:** [Power BI's Secret Weapon: Merging Messy Data Sources into Analytics Gold](https://devtoolsfeed.com/article/power-bis-secret-weapon-merging-messy-data-sources-into-analytics-gold/) - **Read more:** [Rails Magic Methods Finally Work in Plain Ruby Scripts — No Rails Bloat Needed](https://devtoolsfeed.com/article/rails-magic-methods-finally-work-in-plain-ruby-scripts-no-rails-bloat-needed/) Frequently Asked Questions **How do I block AI crawlers on nginx?** Use the map block above—add agents, return 403. Reload nginx. **What analytics tool shows AI bot traffic best?** Umami or Plausible for human baseline, plus raw server logs. GA hides it. **Does robots.txt block Meta's crawler?** Usually yes—but act fast; it's advisory only.