Block AI Crawlers with One Click: Vercel's New AI Bot Managed Ruleset.

As a developer, I've been watching with growing concern as AI crawlers steadily consume more and more bandwidth across the web. These bots from Open AI, Anthropic and Perplexity are now generating more traffic than actual users on many popular sites which causing infrastructure costs and raising serious questions about data usage.

What's particularly frustrating 😤 is how many of these crawlers simply ignore robots.txt directives. The traditional ways of controlling crawler behavior just aren't cutting it anymore.

That's why I was excited to see Vercel's new AI bot managed ruleset that launched yesterday. With a single toggle, it blocks unauthorized AI crawlers like GPTBot, ClaudeBot, and PerplexityBot. 🚫🤖

🚀 How to Enable It

🔹 Go to your project dashboard and select the Firewall tab
🔹 Click Configure
🔹 In the Bot Protection section, choose Log or Challenge mode
🔹 Review your changes and hit Publish ✅

ℹ️ You'll need Developer or Viewer permissions to view the Firewall page, and Project administrator or Team member status to save and apply configurations.

Publish By Piyush Kashyap