Better Robots.txt Premium v3.0.1

Better Robots.txt Premium is a WordPress plugin that provides guided control over which bots can crawl, train, or quote your website content. It eliminates manual robots.txt editing by offering preset configurations, per-crawler rules, and machine-readable policies including llms.txt and AI usage signals.

Key Features

• Guided Robots.txt Generation - Provides preset configurations (Essential, AI-First, Fortress, Custom) to generate robots.txt files without manual text editing
• Per-Crawler Rules - Allows customized rules for specific bots including GPTBot, ClaudeBot, Google-Extended, Applebot, and others
• llms.txt Publishing - Enables publishing of llms.txt files to communicate site offerings to AI systems
• AI Usage Policy Configuration - Lets you define your stance on training, quoting, and indexing for AI crawlers
• Crawl Waste Reduction - Blocks bad bots and unnecessary paths to reduce wasted crawl budget
• AI Usage Signals - Provides machine-readable signals about how your content can be used by AI systems
• Preset-Based Workflow - Uses guided presets and explanations to make robots.txt management accessible to non-specialists
• Review Step - Includes a final review stage before publishing policy changes
• Machine Governance Support - Enables different policies for search engines, AI bots, archive services, SEO tools, and bad bots rather than applying one default policy
• Operational Documentation - Maintains understandable, publishable policies that remain clear for future reference

Best For

This plugin is designed for WordPress teams managing AI visibility and bot access without robots.txt expertise. It serves organizations that need to control how AI systems, search engines, and other bots interact with their content, and those implementing comprehensive AI visibility strategies that include source page design, snippet governance, and crawler behavior measurement.