We block certain User Agents at the server level because they have repeatedly caused excessive load on customers' websites and servers in the past.
The block is enabled by default on all our web hosting and managed server products. Its primary purpose is to limit automated access by bots that crawl websites extensively and cause load issues. This also includes various crawlers operated by AI providers and search services.
The blocking is based on our experience with increased server load and is intended to protect the availability and stability of your website.
Blocked User Agents
The following User Agents are blocked by default:
| User Agent | Operator | Typical Use |
|---|---|---|
Amazonbot |
Amazon | Crawling web content to improve Amazon products and services; according to Amazon, the collected content may also be used to train AI models |
Bytespider |
ByteDance | Crawling web content, including for AI-related applications |
ClaudeBot |
Anthropic | Collecting public web content that may potentially contribute to the training of AI models |
GPTBot |
OpenAI | Crawling publicly accessible content that may be used to train OpenAI models |
MJ12bot |
Majestic | Building and maintaining a link database for SEO and search engine analysis |
meta-externalagent |
Meta | Crawling web content for AI-related applications |
Scrapy |
Open-source project | Web crawling framework for developing custom crawlers |
SemrushBot |
Semrush | SEO, website, and marketing analysis |
The purposes listed above describe the typical use or the use specified by the respective operator. In the case of Scrapy in particular, the User Agent alone does not indicate the specific operator or purpose of an access, as Scrapy is a freely available framework for developing custom crawlers.
Difference Between Automated Crawling and User-Initiated Access
Some providers use different User Agents for different purposes. This makes it possible, for example, to distinguish between the automated collection of content for model development, search, and a specific retrieval initiated by a user.
OpenAI
OpenAI uses several User Agents, including the following:
| User Agent | Purpose | Status with us |
|---|---|---|
GPTBot |
Automatically crawling publicly accessible content that may be used to train OpenAI models | Blocked |
OAI-SearchBot |
Crawling web content for ChatGPT search features | Not blocked |
ChatGPT-User |
Retrieving web content in connection with certain user-initiated actions | Not blocked |
OpenAI describes GPTBot as a separate crawler for content that may be used to train models. OAI-SearchBot, on the other hand, is used to make websites discoverable for ChatGPT search features.
Since OAI-SearchBot is not blocked, publicly accessible content from your websites can still be included in ChatGPT search results.
ChatGPT-User is used by OpenAI for certain user-initiated actions. For example, if a user asks ChatGPT to access a website, ChatGPT-User can be used for this purpose.
Anthropic (Claude)
Anthropic also distinguishes between different User Agents:
| User Agent | Purpose | Status with us |
|---|---|---|
ClaudeBot |
Collecting public web content that may potentially contribute to the training of AI models | Blocked |
Claude-SearchBot |
Crawling the web to improve the quality and relevance of search results | Not blocked |
Claude-User |
Retrieving web content in response to a specific user request | Not blocked |
Anthropic describes ClaudeBot as a crawler for web content that may potentially contribute to AI model training data. The other two User Agents are used for search and for retrieving content at the user's request.
Blocking ClaudeBot prevents automated collection of content by this User Agent. Since Claude-SearchBot and Claude-User are not blocked, access for the corresponding search and user-initiated functions remains possible in principle. Anthropic notes that blocking Claude-SearchBot may affect a website's visibility in search results, while blocking Claude-User prevents content from being retrieved in response to user requests.
What Does the Blocking Mean for Your Website?
The blocking of the listed User Agents only affects the respective accesses that are blocked at the server level.
In particular, it does not mean that a website can no longer be found by search engines or AI services in general. What matters is which User Agents a particular provider uses for its different functions and whether those User Agents are included in our blocklist.
For example, with OpenAI, OAI-SearchBot remains allowed even though GPTBot is blocked. This means that the website can still be crawled for ChatGPT search. User-initiated access via ChatGPT-User is also not blocked by our GPTBot restriction.
Anthropic uses a similar distinction between ClaudeBot, Claude-SearchBot, and Claude-User.
Server-Level Blocking and robots.txt
Our block is enforced at the server level (HTTP response 403) and is therefore independent of the rules specified in robots.txt.
The robots.txt file is a way of telling automated crawlers which areas of a website they should not crawl. However, it is not a technical access control mechanism. A bot that does not follow these instructions can still access a website and cause excessive server load.
Server-level blocking therefore provides an additional technical safeguard against the User Agents we have identified.
Can I Customize the List of Blocked User Agents Myself?
No, you cannot customize the blocklist yourself. Our managed server support team can adjust the list for you.
If you want to allow User Agents from the list above to access your website, please submit an authorized support request through konsoleH. In your request, specify which User Agents you want to allow for which domain.