cloudflare · jinhee-c-lee · Jun 26, 2026 · Jun 26, 2026 · Jun 29, 2026 · Jun 30, 2026
@@ -441,6 +441,7 @@
 /support/other-languages/deutsch/cloudflare-bot/ /bots/troubleshooting/ 301
 /bots/reference/verified-bot-categories/ /bots/concepts/bot/verified-bots/#categories 301
 /bots/reference/verified-bot-policy/ /bots/concepts/bot/verified-bots/policy/ 301
+/bots/concepts/bot/signed-agents/ /bots/concepts/bot/verified-bots/ 301
 /bots/concepts/challenge-solve-rate/ /cloudflare-challenges/reference/challenge-solve-rate/ 301
 /bots/concepts/detection-ids/ /bots/additional-configurations/detection-ids/ 301
 /bots/concepts/ja3-ja4-fingerprint/ /bots/additional-configurations/ja3-ja4-fingerprint/ 301
@@ -2985,3 +2986,6 @@
 
 # Security Insights (moved from Security Center to Security)
 /security-center/security-insights/* /security/security-insights/:splat 301
+
+# Bots: signed agents deprecated -> verified bots
+/bots/concepts/bot/signed-agents/* /bots/concepts/bot/verified-bots/ 301
@@ -8,7 +8,7 @@ date: 2026-03-10
 
 _Edit: this post has been edited to clarify crawling behavior with respect to site guidance._
 
-You can now crawl an entire website with a single API call using [Browser Rendering](/browser-run/)'s new [`/crawl` endpoint](/browser-run/quick-actions/crawl-endpoint/), available in open beta. Submit a starting URL, and pages are automatically discovered, rendered in a headless browser, and returned in multiple formats, including HTML, Markdown, and structured JSON. The endpoint is a [signed-agent](https://developers.cloudflare.com/bots/concepts/bot/signed-agents/) that respects robots.txt and [AI Crawl Control](https://www.cloudflare.com/ai-crawl-control/) by default, making it easy for developers to comply with website rules, and making it less likely for crawlers to ignore web-owner guidance. This is great for training models, building RAG pipelines, and researching or monitoring content across a site.
+You can now crawl an entire website with a single API call using [Browser Rendering](/browser-run/)'s new [`/crawl` endpoint](/browser-run/quick-actions/crawl-endpoint/), available in open beta. Submit a starting URL, and pages are automatically discovered, rendered in a headless browser, and returned in multiple formats, including HTML, Markdown, and structured JSON. The endpoint is a [verified bot (intermediary agent)](/bots/concepts/bot/verified-bots/) that respects robots.txt and [AI Crawl Control](https://www.cloudflare.com/ai-crawl-control/) by default, making it easy for developers to comply with website rules, and making it less likely for crawlers to ignore web-owner guidance. This is great for training models, building RAG pipelines, and researching or monitoring content across a site.
 
 Crawl jobs run asynchronously. You submit a URL, receive a job ID, and check back for results as pages are processed.
 

@@ -493,6 +493,11 @@
 		"deeplink": "/?to=/:account/:zone/analytics/traffic",
 		"parent": ["Analytics & logs"]
 	},
+	{
+		"name": "Attribution Business Insights",
+		"deeplink": "/?to=/:account/:zone/analytics/attribution-business-insights",
+		"parent": ["Analytics & logs"]
+	},
 	{
 		"name": "Web analytics",
 		"deeplink": "/?to=/:account/:zone/analytics/web/overview",

@@ -7,7 +7,7 @@ products:
 tags:
   - Account takeover
 sidebar:
-  order: 6
+  order: 8
   label: Account Abuse Protection
   badge:
     text: Early Access

@@ -18,21 +18,31 @@ import { Render, Steps, DashButton } from "~/components"
 The **Block AI bots** feature is only available in the new [application security dashboard](/security/).
 :::
 
-You can choose to block AI bots by activating **Block AI bots**. Activating this setting will block [verified bots](/bots/concepts/bot/verified-bots/) that are classified as AI crawlers, as well as a number of unverified bots that behave similarly.
+You can choose to block AI bots based on their [behavior](/bots/concepts/bot/#ai-bots). Instead of a single "Block AI bots" switch, Cloudflare offers presets for the most common AI behaviors so you can allow the traffic that helps your business and block the traffic that harms it:
 
-To block [AI bots](/bots/concepts/bot/#ai-bots):
+- **Search**: crawlers that collect or index your content to answer questions about it later. Site owners should expect referral traffic or other equitable compensation in return.
+- **Agent**: automated activity acting in real time on a person's behalf, such as chat fetch bots and browser-use agents.
+- **Training**: crawlers taking your content to train or fine-tune a model, including multi-purpose crawlers such as Googlebot, Applebot, and Bingbot.
+
+Each preset blocks both [verified bots](/bots/concepts/bot/verified-bots/) classified with that behavior and a number of unverified bots that behave similarly.
+
+:::note[New defaults on September 15, 2026]
+These options are available now. On **September 15, 2026**, Cloudflare will set new defaults for new domains: **Training** and **Agent** blocked on pages that display ads, and **Search** allowed. Multi-purpose crawlers that combine Search and Training (such as Googlebot, Applebot, and Bingbot) are blocked when you block **Training**. Existing configurations are not changed automatically — before September 15 you can [opt out of the new defaults](https://dash.cloudflare.com/?to=/:account/:zone/security/settings).
+:::
+
+To configure AI bot blocking:
 
 <Steps>
   1. In the Cloudflare dashboard, go to the **Security Settings** page.
 
       <DashButton url="/?to=/:account/:zone/security/settings" />
   2. Filter by **Bot traffic**.
-  3. Go to **Block AI bots**.
-  4. Under **Configurations**, select the edit icon. Choose from:
-      - **Only block on hostnames with ads**: Use this option if you wish to block AI bots only on portions of your site that show ads. Cloudflare automatically detects whether ads are present on a subdomain, and only block on hostnames that contain those ad units.
-      - **Block on all pages**: Use this option if you wish to block AI bots on all your pages.
-      - **Do not block (off)**: Use this option if you wish to allow AI bots on all your pages.
+  3. Go to the AI bot presets (**Search**, **Agent**, and **Training**).
+  4. For each preset, select the edit icon. Choose from:
+      - **Only block on hostnames with ads**: Use this option if you wish to block this behavior only on portions of your site that show ads. Cloudflare automatically detects whether ads are present on a subdomain, and only blocks on hostnames that contain those ad units.
+      - **Block on all pages**: Use this option if you wish to block this behavior on all your pages.
+      - **Do not block (off)**: Use this option if you wish to allow this behavior on all your pages.
   5. Select **Save** to save your configuration.
 </Steps>
 
-To block individual AI crawlers (rather than blocking all crawlers), use [AI Crawl Control](/ai-crawl-control/).
+To block individual AI crawlers (rather than blocking by behavior), use [AI Crawl Control](/ai-crawl-control/). An individual block in AI Crawl Control can override an allowed preset.
@@ -0,0 +1,41 @@
+---
+pcx_content_type: concept
+title: Attribution Business Insights
+description: Understand which bots help or harm your business with crawl-to-referral ratios and behavior-based classification.
+products:
+  - bots
+tags:
+  - AI
+  - Bots
+sidebar:
+  order: 7
+head:
+  - tag: title
+    content: Attribution Business Insights
+---
+
+import { DashButton } from "~/components"
+
+**Attribution Business Insights** is a dashboard designed for business decision-makers and content owners, delivering a targeted view of bot traffic flowing to your website. Analyze crawler patterns to your website in the last 24 hours, 7 days, or 30 days.
+
+## Availability
+
+Attribution Business Insights is available to all [Bot Management Enterprise](/bots/get-started/bot-management/) customers.
+
+This dashboard is meant for visibility for a new set of stakeholders, and does not provide a new control plane. To mitigate certain bots, website owners can use [Security rules](/security/rules/) or the [new AI bot mitigation options](/bots/additional-configurations/block-ai-bots/).
+
+## Access
+
+<DashButton url="/?to=/:account/:zone/analytics/attribution-business-insights" />
+
+You can also reach the dashboard from your zone-level **Analytics** > **Attribution Business Insights** in the Cloudflare dashboard.
+
+## Definitions
+
+The dashboard surfaces both existing and new metrics that help you evaluate AI traffic. In the current version, we use the following definitions for the metrics shown on the dashboard:
+
+- **Content pages**: Content is initially defined as HTML pages on your website.
+- **Crawl-to-referral ratio, per bot operator**: The average crawl-to-referral ratio (number of crawls sent by this company, vs. the number of visitors who visit you through a referral link from that company, tracked through UTM parameters) for a given company, in the selected time period.
+- **Crawl-to-referral ratio, site-wide**: The average crawl-to-referral ratio (number of crawls sent by this company, vs. the number of visitors who visit you through a referral link from that company, tracked through UTM parameters) across all activity on your zone, in the selected time period.
+- **Classification**: Each crawler is classified with Cloudflare's updated taxonomy. See [Verified bot classifications](/bots/concepts/bot/verified-bots/) for more information. If the company has at least 1 bot with an AI use case, we label the operator with the "AI" label, plus provide this as a filter.
+- **Action**: Action reflects whether requests from this company are Blocked, Allowed, or Partially blocked. Companies that have some bots blocked but at least 1 bot allowed will be marked as "Partially Blocked", and configuration can be confirmed in [Security rules](/security/rules/).
@@ -0,0 +1,60 @@
+---
+pcx_content_type: concept
+title: BotBase
+description: Browse Cloudflare's directory of all known bots and agents, with behavior-based classification, directly in the dashboard.
+products:
+  - bots
+tags:
+  - AI
+  - Bots
+sidebar:
+  order: 6
+head:
+  - tag: title
+    content: BotBase
+---
+
+import { GlossaryTooltip } from "~/components"
+
+BotBase is Cloudflare's directory of all known bots, including [verified bots and agents](/bots/concepts/bot/verified-bots/). It provides a comprehensive, searchable view of the entire bot directory directly in the Cloudflare dashboard, where you can see how Cloudflare classifies each bot and target individual bots in your security configuration.
+
+BotBase currently provides visibility into tracked bots. To mitigate traffic, use [Security rules](/security/rules/) or the [AI bot presets](/bots/concepts/bot/#ai-bots).
+
+## Availability
+
+BotBase is available to [Enterprise Bot Management](/bots/get-started/bot-management/) customers.
+
+## Access
+
+To open BotBase, go to the Bot Management configuration card in the Cloudflare dashboard. BotBase opens in a dedicated page that lists every bot Cloudflare tracks.
+
+## What you can do
+
+- Browse the full catalogue of all verified bots and agents, and see the behavior or behaviors each one is classified under.
+- Search and filter the directory to find a specific bot or group of bots.
+- Filter your own traffic to a specific bot to investigate its activity on your zone.
+- Copy a bot's <GlossaryTooltip term="detection ID">detection ID</GlossaryTooltip> to target it in [Security rules](/security/rules/).
+
+## Classification by behavior
+
+BotBase classifies each tracked bot by its behavior — what the bot may do on your site. A single bot can have one or more of the following behaviors:
+
+| Behavior | Description |
+| --- | --- |
+| Search | Crawling to build search indexes or RAG databases. |
+| Agent | User-directed agents visiting a page on behalf of a human. |
+| Training | Crawling to train or fine-tune models. |
+| Transact | Checkout or other transaction actions on behalf of users. |
+| Data Collection | Price scraping, competitive intelligence gathering, and third-party analytics. |
+| Security Testing | Vulnerability scanning and penetration testing. |
+| SEO | SEO crawling, site auditing, and accessibility checks. |
+| Ads Verification | Ad placement verification and ad fraud detection. |
+| Social / Link Preview | Link previews for social platforms and messaging apps. |
+| Feed Fetching | RSS readers, podcast aggregators, and news feed bots. |
+| Monitoring & Operations | Uptime monitoring, webhooks, and health checks. |
+
+Search, Agent, and Training are also available as managed presets you can act on across all plans. For more information, refer to [AI bots](/bots/concepts/bot/#ai-bots).
+
+## Radar's public-facing BotBase
+
+Every bot tracked in BotBase, along with select metadata, is available publicly in [Cloudflare Radar's bots and agents directory](https://radar.cloudflare.com/bots/directory).
@@ -24,44 +24,47 @@ Bots can be used for good (chatbots, search engine crawlers) or for evil (invent
 For more background, refer to [What is a bot?](https://www.cloudflare.com/learning/bots/what-is-a-bot/).
 :::
 
-## Verified bots and signed agents
+## Verified bots and agents
 
 <Render file="verified-bots" product="bots" />
 
 :::note
-The method for allowing or blocking verified bots depends on [your plan](/bots/concepts/bot/verified-bots/#availability). 
+The method for allowing or blocking verified bots depends on [your plan](/bots/concepts/bot/verified-bots/#availability).
 :::
 
 ## AI bots
 
-To prevent AI-related usage of your site content (such as training language models or generating search answers), you can turn on a managed rule that blocks known AI crawlers that use data for training models ("AI Bots"). A managed rule is a rule that Cloudflare maintains and updates — you turn it on, but you do not write or edit the rule yourself.
+AI crawlers and agents interact with your site for very different reasons, and you may want to treat those reasons differently. Rather than relying on a single "AI bot" label, Cloudflare classifies bots by **behavior** — what a bot does on your site — so you can allow the behavior that helps your business and block the behavior that harms it. A single bot can have more than one behavior.
 
-### Which bots are blocked
+### Behavior-based classification
 
-When you enable this feature, Cloudflare will block the following bots:
+Cloudflare lets all customers manage three AI-related use cases directly:
 
-- `Amazonbot` (Amazon)
-- `Applebot` (Apple)
-- `Bytespider` (ByteDance)
-- `ClaudeBot` (Anthropic)
-- `DuckAssistBot` (DuckDuckGo)
-- `Google-CloudVertexBot` (Google)
-- `GoogleOther` (Google)
-- `GPTBot` (OpenAI)
-- `Meta-ExternalAgent` (Meta)
-- `PetalBot` (Huawei)
-- `TikTokSpider` (ByteDance)
-- `CCBot` (Common Crawl)
+| Behavior | What it does | Examples |
+| --- | --- | --- |
+| **Search** | Collects or indexes your content so it can answer questions about it later. Site owners should expect referral traffic or other equitable compensation in return. | OAI-SearchBot, Claude-Search |
+| **Agent** | Automated activity acting in real time on a person's behalf to get something done, such as chat fetch bots and browser-use agents. | ChatGPT-User, browser-use agents |
+| **Training** | Crawls your content to train or fine-tune a model, permanently absorbing your data into the model. | GPTBot, ClaudeBot |
 
-In addition to this list, [verified bots](https://radar.cloudflare.com/bots#verified-bots) that are classified as AI crawlers, as well as a number of unverified bots that behave similarly, are included in the rule. This rule does not include verified bots that fall into the `Search Engine` categories.
+Cloudflare classifies many other behaviors too — including Transact, Data Collection, Security Testing, SEO, Ads Verification, Social / Link Preview, Feed Fetching, and Monitoring & Operations. Enterprise Bot Management customers can browse the full behavior classification of every tracked bot in [BotBase](/bots/botbase/).
 
-These categories, and the bots classified in these categories, may change from time to time.
+### Managing AI traffic
 
-If you are a bot operator and feel your bot may have been incorrectly categorized, [add your bot to the list of verified bots](https://dash.cloudflare.com/?to=/:account/configurations/verified-bots).
+The options to manage **Search**, **Agent**, and **Training** traffic are available to all customers, including on the Free plan. To turn them on, refer to [Block AI Bots](/bots/additional-configurations/block-ai-bots/). To express crawl preferences in your `robots.txt`, refer to the [managed `robots.txt` setting](/bots/additional-configurations/managed-robots-txt/).
+
+### New defaults on September 15, 2026
+
+On **September 15, 2026**, Cloudflare will set new defaults for these three classifications. For new domains onboarding to Cloudflare, **Training** and **Agent** will be blocked by default **on pages that display ads**, while **Search** will remain allowed by default. An ad signals that a page is meant for a person to land on and see, so on those pages Cloudflare keeps off the automated traffic that does not send readers back.
+
+Also on September 15, multi-purpose crawlers — those that combine **Search** with **Training** — will be blocked according to *all* of their behaviors. Because defaults are enforced by the most restrictive applicable rule, multi-purpose crawlers such as `Googlebot`, `Applebot`, and `Bingbot` will be blocked for customers who block **Training**, whether through the [options to manage AI traffic](/bots/additional-configurations/block-ai-bots/) or the legacy Block AI bots service.
+
+:::note
+Existing configurations are not changed automatically. Before September 15, you can [opt out of the new defaults](https://dash.cloudflare.com/?to=/:account/:zone/security/settings) to confirm that you want no changes to Training crawlers that also crawl for Search. You can block or allow individual crawlers at any time with [AI Crawl Control](/ai-crawl-control/).
+:::
 
 ### How it works
 
-When you enable this feature, Cloudflare detects and blocks two categories of AI bots:
+When you enable AI bot blocking, Cloudflare detects and blocks two categories of AI bots:
 
 - **Well-behaved AI crawlers** that comply with `robots.txt`, respect crawl rates, and do not hide their behavior from your website.
 - **Evasive AI crawlers** that do not follow these conventions but are detected through additional signatures.