AI Crawler Blocker Plugin

AI Crawler Blocker adds crawler rules to the robots.txt file generated by Publii. You can block the 160 AI-related crawler identifiers included with the plugin, allow selected crawlers, or add identifiers that are missing from the bundled list.

The plugin preserves the rules already present in robots.txt. It places its rules inside a marked section and replaces that section when Publii generates the website again. Rules created by Publii or maintained through File Manager remain outside the managed section.

AI Crawler Blocker can also add noai and noimageai metadata to generated HTML pages. These values provide an extra preference signal without changing existing index, follow, or noindex directives.

Important: robots.txt asks compliant crawlers to follow the published rules. It does not authenticate visitors or prevent a service from requesting a public page. A crawler that ignores robots.txt can still access the website.

Enabling the Plugin

AI Crawler Blocker requires Publii 0.47.0 or newer. Open Publii and go to Tools & Plugins. Find AI Crawler Blocker and activate it using the toggle in the bottom-left corner of the plugin tile.

The default configuration uses the Strict profile. It blocks all 160 crawler identifiers in the bundled snapshot. The plugin also adds the noai and noimageai metadata and includes diagnostic comments in robots.txt. Emergency Lockdown and local Preview generation remain off.

Save the plugin settings, then generate or synchronize the website. The published file is available at:

https://your-domain.com/robots.txt

File Manager: The plugin writes to Publii's generated output directory. It does not create or modify a source file in File Manager, so the managed version of robots.txt will not appear there.

How the Plugin Updates robots.txt

Publii creates or copies robots.txt before AI Crawler Blocker runs. The plugin reads that output file, removes its previous managed section, checks the remaining rules, and appends a new section for the current configuration.

If the generated output has no robots.txt content, the plugin starts with a base that allows crawling:

User-agent: *
Disallow:

When Publii has sitemap generation enabled and the website uses a public domain, the plugin also keeps the sitemap address in that base file.

A shortened block for the Training only profile can look like this:

# BEGIN AI CRAWLER BLOCKER
# Generated by AI Crawler Blocker 1.0.0
# Policy: training-only
# Blocked user agents: 72
# Dataset: ai.robots.txt @ 2f5d7ccf39b2
# robots.txt is a crawler preference signal, not access control.

# AI training, data collection, and scraping
User-agent: AI2Bot
Disallow: /
User-agent: Ai2Bot-Dolma
Disallow: /
# ...
User-agent: GPTBot
Disallow: /
# ...

# END AI CRAWLER BLOCKER

The exact count can be lower than the profile total. AI Crawler Blocker does not add a second full block for a crawler that is already blocked outside its managed section. Allowed crawler exceptions and disabled individual controls also reduce the count.

The plugin writes the finished file through a temporary file and then renames it to robots.txt. Repeated site generations keep one managed section rather than adding another copy.

Existing Rules and Duplicate Detection

AI Crawler Blocker preserves custom paths, sitemap lines, comments, and crawler groups found outside these markers:

# BEGIN AI CRAWLER BLOCKER
# END AI CRAWLER BLOCKER

The plugin treats a crawler as already blocked when its existing group contains Disallow: / and no non-empty Allow rule. It does not treat Disallow: /private/ as a full block. A group that blocks the root but contains an allowed path also remains eligible for a plugin rule.

This detection includes rules created through Publii settings and rules copied from File Manager. If Publii already blocks GPTBot, ChatGPT-User, or CCBot, the plugin leaves the existing rule in place and does not repeat it inside the managed section.

An existing User-agent: * group with Disallow: / covers crawlers that have no separate group, so the plugin avoids adding redundant entries for them. If the file contains an explicit crawler group that allows access despite the wildcard, AI Crawler Blocker can still add a full block for that crawler.

Protection Profiles

The Protection profile setting controls which bundled crawler identifiers receive Disallow: /. Choose:

Strict to select all 160 identifiers in the bundled snapshot. This is the default profile. It covers the training, AI search, agent, and other categories without adding a wildcard rule. Allowed crawler exceptions can remove selected identifiers from the plugin-managed block.
Training only to select the 72 crawlers classified as model training, dataset, data collection, or scraping services. The bundled AI search, assistant, agent, and other categories remain available.
Individual crawlers to control all 160 bundled identifiers with separate switches. Every switch starts on. Select a category to filter the controls, then turn off blocking for crawlers you want to allow. The category filter does not reset choices saved elsewhere.
Custom to choose from four categories: training and data collection, AI search and answer engines, assistants and autonomous agents, and other or unclassified crawlers. All four category switches start on. Turn off the categories you want to allow.

Enabled entries under Additional crawlers to block apply to every profile. If you turn off all four categories in Custom, the plugin adds no bundled crawler rules, but enabled custom crawlers and Emergency Lockdown still apply.

Profile settings and individual switches affect the plugin-managed section. They do not remove matching rules that already exist in Publii's robots.txt output or in a custom File Manager copy.

Bundled Crawler Categories

The current plugin snapshot contains 160 case-insensitive crawler identifiers divided into these categories:

Training, datasets, and AI scrapers: 72
AI search and answer engines: 19
Assistants and autonomous agents: 51
Other or unclassified crawlers: 18

The bundled data comes from the MIT-licensed ai.robots.txt project.

AI Crawler Blocker makes no network request to update this list during Preview, site generation, or deployment. A later plugin release can include a newer snapshot. Use Additional crawlers to block when you need an identifier that is not present in the installed version.

All 160 Bundled Crawlers

AddSearchBot
AgentTimes
AI2Bot
AI2Bot-DeepResearchEval
Ai2Bot-Dolma
aiHitBot
AIWebIndex
amazon-kendra
amazon-QBusiness
Amazonbot
AmazonBuyForMe
Amzn-SearchBot
Amzn-User
Andibot
Anomura
anthropic-ai
ApifyBot
ApifyWebsiteContentCrawler
Applebot
Applebot-Extended
Aranet-SearchBot
atlassian-bot
Awario
AzureAI-SearchBot
bedrockbot
bigsur.ai
Bravebot
Brightbot
Brightbot 1.0
BuddyBot
Bytespider
CCBot
Channel3Bot
ChatGLM-Spider
ChatGPT Agent
ChatGPT-User
Claude-Code
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
Cloudflare-AutoRAG
CloudVertexBot
Code
cohere-ai
cohere-training-data-crawler
Cotoyogi
CragCrawler
Crawl4AI
Crawlspace
Cursor
Datenbank Crawler
DeepSeekBot
Devin
Diffbot
DuckAssistBot
Echobot Bot
EchoboxBot
ExaBot
FacebookBot
facebookexternalhit
Factset_spyderbot
FirecrawlAgent
FriendlyCrawler
GeistHaus-PageFetcher
Gemini-Deep-Research
Google-Agent
Google-CloudVertexBot
Google-Extended
Google-Firebase
Google-Gemini-CLI
Google-NotebookLM
GoogleAgent-Mariner
GoogleAgent-URLContext
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
HenkBot
iAskBot
iaskspider
iaskspider/2.0
ICC-Crawler
ImagesiftBot
imageSpider
img2dataset
ISSCyberRiskCrawler
kagi-fetcher
Kangaroo Bot
Kimi-User
KlaviyoAIBot
KunatoCrawler
laion-huggingface-processor
LAIONDownloader
LCC
LinerBot
Linguee Bot
LinkupBot
Manus-User
Meta-ExternalAgent
Meta-ExternalFetcher
meta-webindexer
MistralAI-User
MistralAI-User/1.0
Mozilla-Tabstack
MyCentralAIScraperBot
NagetBot
netEstate Imprint Crawler
newsai
NotebookLM
NovaAct
OAI-SearchBot
omgili
omgilibot
OpenAI
opencode
Operator
PanguBot
Panscient
panscient.com
Perplexity-User
PerplexityBot
PetalBot
PhindBot
Poggio-Citations
Poseidon Research Crawler
QualifiedBot
Querit-SearchBot
QueritBot
QuillBot
quillbot.com
SBIntuitionsBot
Scrapy
SemrushBot-OCOB
SemrushBot-SWA
Shap-User
ShapBot
Sidetrade indexer bot
Spider
TavilyBot
Terra Cotta
TerraCotta
Thinkbot
TikTokSpider
Timpibot
TongyiBot
Trae
TwinAgent
UseAI
VelenPublicWebCrawler
WARDBot
Webzio-Extended
wpbot
WRTNBot
YaK
YandexAdditional
YandexAdditionalBot
YiyanBot
YouBot
ZanistaBot

Emergency Lockdown

Emergency lockdown: block every compliant crawler adds this rule:

User-agent: *
Disallow: /

This setting applies in addition to the selected protection profile. It asks all compliant crawlers to stop crawling the website, including traditional search crawlers.

Warning: Emergency Lockdown can remove the website from search results after search engines process the rule. Use it when blocking all compliant crawlers is the intended result.

Allowed crawler exceptions do not override the wildcard created by Emergency Lockdown. Disable Emergency Lockdown and regenerate the website when selected crawlers should regain access.

The normal Strict, Training only, Individual crawlers, and Custom profiles do not add a wildcard. The current bundled snapshot does not contain the standard Googlebot or Bingbot identifiers. Existing rules and entries added through Additional crawlers to block can still target them.

Additional Signals

Add noai and noimageai Meta Directives

This option is enabled by default. It adds the following tag to generated HTML pages that contain a <head> element:

<meta name="robots" content="noai, noimageai" data-ai-crawler-blocker>

The plugin uses Publii's publiiHead insertion for the tag and checks the completed HTML to prevent duplicates. Existing robots metadata remains unchanged:

<meta name="robots" content="index, follow">
<meta name="robots" content="noai, noimageai" data-ai-crawler-blocker>

If another robots tag already contains noai, the plugin adds only noimageai. If both values already exist, it does not add another tag.

noai and noimageai are experimental values. Services do not support them in a consistent way, and the tags do not enforce access restrictions. Disable this option if you do not want the plugin to add these values.

Advanced Exceptions

Allowed Crawler Exceptions

Use Allowed crawler exceptions to remove exact user-agent identifiers from the plugin-managed block. Add the product token sent by the crawler, such as OAI-SearchBot.

Matching ignores letter case, so OAI-SearchBot and oai-searchbot refer to the same identifier. An exception applies to bundled crawlers and entries under Additional crawlers to block, regardless of the selected protection profile.

The optional Private note stays in the plugin settings. AI Crawler Blocker does not write it to robots.txt.

An exception prevents AI Crawler Blocker from adding its own rule. It does not remove a matching block created by Publii, File Manager, or another tool. It also cannot override Emergency Lockdown.

The repeater accepts up to 250 exceptions.

Custom Crawlers

Additional Crawlers to Block

Use this repeater for crawler identifiers missing from the bundled snapshot. Each entry contains:

Enabled, which controls whether the plugin processes the entry
User-agent, which contains the exact crawler product token
Private note, which remains in the settings and does not appear in robots.txt

Enabled custom crawlers apply to Strict, Training only, Individual crawlers, and Custom. The selected bundled category does not limit them. Add an allowed crawler exception when a custom entry should remain configured but should not receive a block.

The repeater accepts up to 250 entries. Each user-agent may contain ASCII letters, numbers, spaces, hyphens, underscores, periods, and forward slashes. The first character must be a letter or number, and the complete value cannot exceed 200 characters.

The plugin ignores empty entries, disabled entries, case-insensitive duplicates, wildcard values, and names containing colons, line breaks, comments, or control characters. Use Emergency Lockdown for the * wildcard.

Advanced Settings

Include Policy and Dataset Comments in robots.txt

This option is enabled by default. The comments record:

the AI Crawler Blocker version
the selected protection profile
the number of crawler rules added by the plugin
the first 12 characters of the bundled dataset commit
category headings and the reminder that robots.txt is not access control

Comments do not change crawler behavior. Disable the option when you want a shorter managed section. The BEGIN and END markers remain because the plugin needs them to replace its previous output.

Generate robots.txt During Local Preview

This option is disabled by default. Turn it on to let AI Crawler Blocker update robots.txt in the local Preview output.

Preview generation is intended for inspection and testing. Normal site generation remains supported regardless of this setting.

When the option is off, AI Crawler Blocker does not modify the Preview copy of robots.txt. It can still add the noai metadata to rendered Preview pages while Add noai and noimageai meta directives remains enabled.

Checking the Generated Output

To inspect the plugin before publishing:

Enable Generate robots.txt during local Preview.
Save the settings and run Preview again.
Open the Preview output directory and locate robots.txt at its root.
Check for # BEGIN AI CRAWLER BLOCKER and # END AI CRAWLER BLOCKER.
Open the source of a generated HTML page and search for noai if the metadata option is enabled.

After synchronization, open https://your-domain.com/robots.txt. The published file should contain the managed section along with the rules and sitemap lines already produced by Publii.

The diagnostic count shows the number of records added inside the current managed section. It does not count crawlers already blocked by rules outside that section.

Search Visibility and Crawler Access

Strict includes AI search and user-triggered agent categories, so those services receive a full-site disallow rule when they respect robots.txt. This can reduce retrieval, citations, links, or referral traffic from those products.

Training only leaves the bundled AI search and agent categories available. Use Individual crawlers, Custom, or Allowed crawler exceptions when the website needs a more specific policy.

Emergency Lockdown also targets traditional search crawlers. Existing wildcard rules or custom entries can have the same effect even when Emergency Lockdown is off. Check the complete published file when Google, Bing, or another service appears blocked.

Limits of robots.txt Blocking

AI Crawler Blocker publishes crawler preferences. It does not inspect requests, verify bot identity, block IP addresses, apply rate limits, or return HTTP access errors.

A service can ignore the rules, change its user-agent, or request a page through an ordinary browser. Use controls provided by a CDN, web application firewall, reverse proxy, or web server when the website requires enforced access restrictions.

Troubleshooting

robots.txt Does Not Appear in File Manager

This is expected. AI Crawler Blocker writes to Publii's generated output, not to the source files shown in File Manager. Inspect the Preview output after enabling Preview generation, or open your-domain.com/robots.txt after synchronization.

The Managed Section Is Missing from Preview

Enable Generate robots.txt during local Preview, save the plugin settings, and run Preview again. The setting is off by default.

The Blocked User-agent Count Is Lower Than Expected

Check the full file for crawler groups outside the managed section. The plugin skips crawlers that already have a full Disallow: / rule without an allowed path. Allowed crawler exceptions, disabled individual switches, and excluded Custom categories also reduce the count.

An existing wildcard block can reduce the managed count to zero because separate crawler groups would add no restriction for crawlers without an explicit override.

A Crawler Remains Blocked After You Allow It

Allowed crawler exceptions and disabled individual switches affect the AI Crawler Blocker section. They do not remove rules outside that section. Check Publii's crawler settings, the File Manager copy of robots.txt, and Emergency Lockdown.

A Custom Crawler Does Not Appear

Confirm that the entry is enabled and contains a valid user-agent. AI Crawler Blocker ignores duplicates, *, values longer than 200 characters, and entries containing colons, line breaks, comments, or control characters. An Allowed crawler exception can also remove the entry from the managed block.

A Crawler Still Visits the Website

Open the published robots.txt and confirm that the exact user-agent appears with Disallow: /. If the rule is present, the service may use another identifier or ignore robots.txt. The plugin does not provide enforced request blocking.

Google or Bing Appears Blocked

Check for User-agent: * followed by Disallow: /. Disable Emergency Lockdown if the wildcard belongs to AI Crawler Blocker. If it appears outside the managed section, update the Publii or File Manager rule that created it.

Also check Additional crawlers to block and any existing explicit groups for Googlebot or Bingbot.

noai Metadata Is Missing

Confirm that Add noai and noimageai meta directives is enabled. View the source of a complete HTML page that contains a <head> element. If another robots tag already contains both values, the plugin does not add a duplicate tag.

Old Plugin Rules Remain After a Settings Change

Generate the website again. AI Crawler Blocker removes the content between its BEGIN and END markers and writes a new managed section. Rules outside those markers remain because the plugin treats them as user or Publii content.