Bots: Content Independence Day 2026 — new taxonomy, AI presets, Attribution Business Insights#31793
Open
jinhee-c-lee wants to merge 8 commits into
Open
Bots: Content Independence Day 2026 — new taxonomy, AI presets, Attribution Business Insights#31793jinhee-c-lee wants to merge 8 commits into
jinhee-c-lee wants to merge 8 commits into
Conversation
- Redefine Verified bots (honest + non-abusive) and add Direct/Intermediary labels - Deprecate Signed agents classification (pages kept as deprecation notices) - Rewrite AI bots section: Training/Search/User presets, pages-with-ads defaults - Update Block AI bots page to the new presets - Add Attribution Business Insights section (between Bot Analytics and Account Abuse Protection) - Glossary + verified-bots partial updates
- Add 301 redirects: /bots/concepts/bot/signed-agents(/*) -> /bots/concepts/bot/verified-bots/ - Delete signed-agents index + policy pages (removes them from the auto-generated sidebar) - Repoint inbound links (bot-verification, web-bot-auth, bot-management-variables, fields) to verified bots
- Document Block Training (all) vs Block Training (only if it doesn't affect search) - Note the latter is equivalent to the legacy Block AI bots option - Call out Googlebot, Applebot, and Bingbot as the 3 tracked bots that do not separate their crawler: included in Block Training (all), excluded from Block Training (only if it doesn't affect search)
…d Definitions - Remove subtitle + 'for decades' framing paragraphs - Tighten intro; add 24h/7d/30d time windows - Reframe Availability as visibility-only; link Security rules and AI bot mitigation options - Update Access dash deep-link and zone-level navigation path - Remove What you can see / Per-operator view / How it relates sections - Add Definitions section (content pages, crawl-to-referral ratios, classification, action)
jinhee-c-lee
requested review from
a team,
elithrar and
marinaelmore
as code owners
Contributor
Review
Code ReviewThis code review is in beta and may not always be helpful — use your judgment. No code review issues found. ConventionsChecks PR title, description, and redirect checklist. No convention issues found. Style Guide ReviewWarnings (1)
Suggestions (8)
RedirectsNo missing redirect entries found. CommandsOnly codeowners can run commands. Post a comment with the command to trigger it.
|
Contributor
|
This PR changes current filenames or deletes current files. Make sure you have redirects set up to cover the following paths:
|
Contributor
|
This pull request requires reviews from CODEOWNERS as it changes files that match the following patterns:
|
Contributor
|
Here is the review for PR #31793. SummaryThis is a well-structured documentation PR that updates the Bots product for the Content Independence Day 2026 (July 1) launch. The changes accurately reflect the new taxonomy (Direct vs. Intermediary), deprecate the standalone "signed agent" concept, introduce behavior-based AI bot presets (Training / Search / User), and add the new Attribution Business Insights page. Frontmatter, sidebar ordering, glossary additions, and dashboard deep-link registration all look correct. Labels applied: Issues requiring attention1. Redirect source missing trailing slash (CI failure risk)File: The first redirect line lacks the required trailing slash on the source path. Per repo conventions, source URLs in -/bots/concepts/bot/signed-agents /bots/concepts/bot/verified-bots/ 301
+/bots/concepts/bot/signed-agents/ /bots/concepts/bot/verified-bots/ 3012. Stale internal link in changelog (will 404 after merge)File: This changelog still links to -The endpoint is a [signed-agent](https://developers.cloudflare.com/bots/concepts/bot/signed-agents/) that respects robots.txt
+The endpoint is a [verified bot (intermediary agent)](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) that respects robots.txt(You can also use the internal path 3. Stale description in bot-verification indexFile: The description still references the deprecated "signed agents" term, while the body text below was already updated. -description: Validation methods Cloudflare uses to verify bots and signed agents.
+description: Validation methods Cloudflare uses to verify bots and agents.Minor notes & nits
What looks good ✅
|
- Add BotBase page (sidebar between Bot Analytics and Attribution Business Insights); bump Attribution (6->7) and Account Abuse Protection (7->8) order - Rename the User behavior/preset to Agent across AI bots, verified bots, and Block AI Bots - Fix redirects: drop unslashed signed-agents source; move wildcard to the dynamic redirect section (resolves validate-redirects failures) - Repoint stale signed-agents changelog link to verified bots - Update bot-verification index description (signed agents -> agents)
- New defaults are enforced on September 15, 2026 (not July 1); AI traffic management OPTIONS are available now - Replace the 'two Training options' framing with the blog's model: blocking Training also blocks multi-purpose crawlers (Googlebot/Applebot/Bingbot) that combine Search + Training, via the new options or the legacy Block AI bots service; customers can opt out of the new defaults before Sept 15 - Order classifications as Search, Agent, Training across AI bots, Block AI Bots, BotBase, and verified bots - Clarify Verified now makes a bot allowable through its behavior category, with baselines taking effect Sept 15
Contributor
…l-endpoint/ Resolves the Post Build link-validation failure (/browser-rendering/ is not a valid page in this repo).
elithrar
approved these changes
Jun 30, 2026
marinaelmore
approved these changes
Jun 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documentation updates for Content Independence Day 2026, aligned to the "Your site, your rules: new AI traffic options for all customers" blog (source of truth).
Changes
/bots/concepts/bot/verified-bots/): redefine "verified" as honest self-identification + non-abusive behavior; a verified bot is now allowable through its behavior category (baselines take effect September 15, 2026); add Direct vs Intermediary labels./bots/concepts/bot/signed-agents(and/*) →/bots/concepts/bot/verified-bots/; inbound links repointed./bots/concepts/bot/#ai-bots): behavior-based Search / Agent / Training classification. The options to manage AI traffic are available now; new defaults are enforced on September 15, 2026 (Training and Agent blocked on pages with ads, Search allowed). On Sept 15, multi-purpose crawlers that combine Search + Training (Googlebot, Applebot, Bingbot) are blocked when blocking Training; customers can opt out before Sept 15./bots/botbase/), ordered between Bot Analytics and Attribution Business Insights; registered its dashboard route. (Attribution Business Insights and Account Abuse Protectionsidebar.orderbumped accordingly.)/bots/attribution-business-insights/).Notes