Cloudflare pilot may raise ai visibility of more web pages

What happened

Cloudflare announced a research pilot with OpenAI on July 8. The companies say the pilot has helped OpenAI discover and index hundreds of millions of previously unknown web pages, increasing the material ChatGPT can draw on, according to Cloudflare’s October 15 blog post coauthored by two OpenAI staff.

The pilot ran on a participating cohort of roughly 1% of Cloudflare’s total network. Cloudflare says its network signals were used to help OpenAI’s search crawler find pages with value and to reduce unnecessary requests.

Cloudflare and OpenAI gave examples of pages surfaced by the pilot, such as a washing machine user guide, up-to-date passport fees and requirements, and a restaurant’s menu and opening hours. The companies also said the signals can flag potentially harmful pages and help identify pages that should be kept out of an index.

The October post reiterates that a signal about a page does not grant permission to access it and points readers to Cloudflare’s AI Crawl Control, which Cloudflare says is available on every plan.

Why it matters for businesses

If AI assistants can discover more pages, businesses may find that additional pages from their sites are visible to users through those assistants. That could include utility pages such as product manuals, updated service requirements or menus and hours that users ask about.

More discovery does not mean automatic citation or inclusion. OpenAI independently evaluates and ranks pages for relevance and helpfulness, and the pilot’s signals are also intended to filter out low-value or harmful pages before they enter an index.

The project shows two practical implications: signals about recency, traffic quality and the nature of page changes can influence whether crawlers consider a page worth visiting, and site owners can use tools such as Cloudflare’s AI Crawl Control to manage how those signals are presented.

Sites that explicitly block OpenAI’s search crawler can avoid being included in ChatGPT’s search results, although the blog post notes that blocked sites may still appear as navigational links. Businesses should therefore check both crawler access settings and their public content to manage ai visibility.

What the AI assistants say today

We checked several AI assistants to see how they reflect the Cloudflare–OpenAI pilot. Some assistants already summarise the pilot’s purpose and effects while others omit or date items inconsistently.

The assistants that reflected the pilot typically cited Cloudflare’s blog and OpenAI developer documents as sources for the pilot’s scope, examples of surfaced pages, the roughly 1% cohort and the distinction between signals and permission to access a page.

What to do now

If you want to manage how your site appears to AI assistants, consider these concrete steps.

1. Check your robots.txt for directives that target OAI-SearchBot and GPTBot and decide whether to allow or block those bots according to your visibility goals.
2. Use Cloudflare’s AI Crawl Control or equivalent tools to review which signals are exposed and adjust settings for recency, change detection and traffic-quality signals.
3. Audit utility pages (manuals, menus, passport or policy pages) for accuracy and clear structure so crawlers and assistants can find and interpret key facts.
4. Monitor server logs and bot traffic to see whether crawlers are requesting new pages and whether those requests are helpful or causing unnecessary load.
5. Implement filters or meta tags on pages you prefer not to be indexed by assistant search crawlers, and test the result by simulating bot access where possible.
6. Periodically review the search and training robot directives in your site documentation to ensure your choices about discovery and training use are still correct.