What are LLM scraper bots?

Automated crawlers from AI companies like OpenAI that slurp web content to train large language models. They're polite in name only — volume crushes servers.

How do I block LLM scraper bots on my site?

Add user-agent blocks to robots.txt or server config (e.g., 'GPTBot', 'anthropic-ai'). Use Cloudflare Bot Management or Nginx limit_req for IP throttling.

Will LLM scrapers break my hobby site?

If HTTPS is weak and traffic spikes, yes — expect outages like acme.com's. Monitor logs, harden ports, consider HTTP-only for non-critical pages.

🤖 Large Language Models

One Dev's HTTPS Server Buckles Under LLM Scraper Bots — Port 443 Shutdown Ends the Nightmare

At 1 a.m., staring at yet another outage, he killed port 443. The flood of LLM scraper bots stopped cold, and his server breathed easy for the first time in a month.

theAIcatchup Apr 08, 2026 4 min read

Server traffic graph spiking from LLM scraper bots on acme.com HTTPS port

⚡ Key Takeaways

LLM scraper bots caused a month's outages on acme.com by overwhelming its HTTPS server, fixed instantly by closing port 443. 𝕏
This isn't isolated — AI firms' indiscriminate crawling hits hobbyists hardest, echoing 90s spam waves. 𝕏
Defend with robots.txt blocks, rate-limiting, and CDNs; industry standards for 'respectful scraping' are overdue. 𝕏

Published by

theAIcatchup

Ship faster. Build smarter.

#AI crawling #AI web scraping #HTTPS overload #LLM scraper bots #LLM scrapers #server outages

Worth sharing?

Get the best Developer Tools stories of the week in your inbox — no noise, no spam.

Originally reported by Hacker News

⚡ Key Takeaways

The 60-Second TL;DR

theAIcatchup

Share this article

Worth sharing?

Related Stories

Anthropic Unearths 27-Year-Old OpenBSD Bug with Claude Mythos – Then Locks It Up

Claude Code's Memory Gamble: 250ms LLM Queries Trump Vector DBs

Claude Hacked Its Own Chat Window—and Sparked a Debate on Consciousness

Reverse-Engineering Claude Code: The CLI AI Coder That Fixes Its Biggest Flaws

Stay in the loop