Infrastructure

Cloudflare Now Writes Your AI Training Rules for You

Cloudflare's new Bot Preference Sync auto-syncs robots.txt with your AI bot settings for Search, Agent, and Training, from Free to Enterprise plans.

InfrastructureBy Servola Tech Desk2026-08-224 min read

AI-assisted, edited by humans. | Editor-in-Chief: Leon Soliman | Editorial standards

Cloudflare Now Writes Your AI Training Rules for You

How Bot Preference Sync Keeps robots.txt Current

Cloudflare announced Bot Preference Sync on its official blog on August 22, 2026, describing a feature that automatically keeps a website's robots.txt file aligned with the AI bot policies an owner has already configured in the Cloudflare dashboard. The tool covers three bot categories - Search, Agent, and Training - so one dashboard choice can update the rules that govern all three crawler types at once. Cloudflare generates or updates the robots.txt file based on the site's zone-level configuration, meaning the settings already stored for that domain drive the resulting text file automatically.

New directives are prepended to the existing robots.txt file, which preserves any Disallow rules an owner had already written by hand. Cloudflare also keeps the file current by drawing on its own BotBase bot-classification list, so entries update automatically as new bots are added to blocked or disallowed categories. The feature can be switched on or off at any time, and it is available to every Cloudflare customer, from the Free tier through Enterprise, rolling out within the week following the announcement.

What Happens Automatically for New and Ad-Monetized Sites

Brand-new Cloudflare customers start with no bot blocks at all under this system, leaving the entire policy choice to the site owner from the first setup. Sites that identify themselves as ad-monetized, meaning they tell Cloudflare they monetize pages with ads, can instead select a preset that sets the Training category to Disallow by default. That single distinction already shows how a site's stated business model, and not a deliberate technical decision, can steer its AI-training exposure before the owner writes a single rule.

The Training category carries four distinct configuration options: Allow, Block on pages that serve ads, Block everywhere, or Disallow, which writes a no-training preference directly into the robots.txt file. Search and Agent traffic remain separately controllable from Training, so an owner can keep a site fully open to search indexing while restricting or blocking crawlers that harvest content for AI-training datasets. Because these choices sit inside a dashboard rather than a manually maintained file, changing a policy takes effect the next time Cloudflare syncs the robots.txt output.

From Manual Dashboard Toggle to Automatic Enforcement

Cloudflare first gave site owners granular control over AI bot traffic on July 1, 2026, letting them separately allow or block crawlers across the same three categories, Search, Agent, and Training, from the dashboard. Bot Preference Sync, announced seven weeks later on August 22, is the automation layer that keeps the resulting robots.txt file matched to those dashboard choices. Owners no longer need to hand-edit a static text file every time a preference changes; Cloudflare handles that step automatically.

The shift moves control of a website's AI-crawling permissions out of a document most owners never open and into a product Cloudflare is actively building out over successive releases. Once the default behavior of robots.txt is generated by dashboard settings and not by whoever last edited the file, the practical AI-training policy for a large share of the web starts to depend on what Cloudflare ships as its defaults. A mixed-use crawler that does both search-indexing and AI-training must meet four specific criteria, including respecting robots.txt preferences and providing URL-level visibility into what was crawled and why, to avoid being blocked once an owner disallows Training.

The New Checkbox Decision for EU and UK Site Owners

Any EU or UK business running a website now faces an AI-training decision that resolves to a dashboard checkbox. Most site owners never open their Cloudflare bot settings page, which means the platform's chosen defaults, such as the ad-monetized preset that disallows Training, will end up setting policy for a large number of sites regardless of whether the owner ever makes an active choice. For the first time, staying silent produces a specific, automatically enforced outcome.

Owners who want a deliberate policy, whether that means allowing AI-training crawlers, blocking them everywhere, or restricting them only on ad-monetized pages, can now set that policy in minutes through the same dashboard used for Search and Agent traffic. The four Training options give enough range to match most editorial or commercial positions without touching a text file directly. Businesses that skip the settings page are not opting out of a decision; they are simply accepting whichever default Cloudflare or their own site profile assigns to them.

Frequently asked questions

What is Cloudflare Bot Preference Sync?

It is a Cloudflare feature, announced on August 22, 2026, that automatically keeps a website's robots.txt file synced with the AI bot policies an owner has configured in the Cloudflare dashboard, covering the Search, Agent, and Training bot categories.

Which Cloudflare plans get this feature?

Bot Preference Sync is available to all Cloudflare customers, from the Free tier through Enterprise, and is rolling out within the week following the announcement rather than being restricted to higher-tier plans.

What happens by default if I do nothing?

Brand-new customers get no bot blocks by default and keep full control over their own policy, while sites that identify as ad-monetized can be set to disallow the Training category automatically as part of that preset.

How is this different from editing robots.txt manually?

Cloudflare generates or updates the file automatically from the dashboard configuration, prepending new directives while preserving existing Disallow rules and keeping entries current against its BotBase bot-classification list. An owner no longer needs to open and hand-edit the static text file for routine policy changes.

Bot Preference Sync is a small technical feature with an outsized effect: it moves the default AI-training posture of a large share of the web from a choice site owners actively make to a setting that infrastructure makes for them, and the July-to-August progression shows Cloudflare treating this as a deliberate product line rather than a one-off release. Servola's view is that any business running a website should treat the Cloudflare bot dashboard as a five-minute task worth doing this week, because the alternative is no longer silence, it is accepting someone else's default.

CloudflareAI botsrobots.txtAI training datapublishers

Post-Quantum Now Needs OpenSSL 3.5 on Your Origin

Cloudflare switched on post-quantum authentication to customer origin servers on 29 July. It is the first part of its post-quantum work that does nothing until you change a machine you run yourself.

5 min read

Cybersecurity

CLDAP Floods Surged 580% as DDoS Tactics Shift

Cloudflare's H1 2026 DDoS Threat Report shows attackers moving from raw botnets to DNS and CLDAP reflection attacks that exploit other organizations' misconfigured servers. What that shift means for EU and UK security budgets, and why the government sector's 20-rank jump in targeting is a live warning.

5 min read

Decisions

The Third Pivot Was the Warning Magic Leap Missed

Magic Leap filed a WARN notice for 193 layoffs effective 1 October 2026, its fourth strategic pivot after $4 billion raised. The lesson is the pattern, not the pivot.

4 min read

← All articles