AI's area of concentration: data flow organized by objective and industry

In a recent development, Cloudflare Radar has introduced industry-specific data on AI bot traffic, providing a comprehensive breakdown of traffic by purpose. This new feature aims to help publishers better understand the composition of active bots and crawl-to-refer ratios within their respective industries.

One of the standout findings is the dominance of certain AI bots in specific industries. For instance, in the computer and electronics sector, GPTBot emerges as the most active AI bot, while Amazonbot moves up to second place, accounting for over 40% of crawling traffic. Conversely, in the news and publications industry, the top 5 bots have a much tighter distribution of traffic, ranging from 14.9% to 17.4%.

The cryptocurrency industry, falling under the finance vertical, presents a unique scenario. The first week of August saw three-quarters of crawling traffic concentrated on just four bots, with 80% of that used for model training data collection. This indicates a heavy reliance on AI bots for data in this sector.

Another concerning trend is the high crawl-to-refer ratio, which measures the number of crawling requests from a platform to the actual page views delivered. The company developing the chatbot platform with the highest Crawl-to-Refer ratio, as published by Cloudflare Radar on July 1, 2023, is OpenAI. This means that platforms like ChatGPT are massively scraping content without providing publishers with significant traffic.

To help publishers compare their metrics with their peer group, Cloudflare Radar allows users to select an industry from a dropdown list. Users can also filter traffic by categories such as training, search, user action, and undeclared purposes in the dashboard. Additionally, users can break down the data by User-Agent and filter by Crawl Purpose to examine traffic trends across a larger group of bots.

Moreover, Cloudflare Radar's "Data Explorer" view allows users to compare their metrics with earlier time periods and break down AI bots and crawlers data by Crawl Purpose. This feature provides a valuable tool for publishers to track changes in AI bot activity over time.

The issue of AI-driven answers replacing classic clicks on sources, leading to no traffic and no revenue, is a topic of concern for many publishers. Cloudflare acknowledges this issue and has committed to continuing its analyses of AI crawler activities through Radar.

In conclusion, Cloudflare Radar's new feature offering industry-specific data on AI bot traffic provides publishers with a valuable resource to understand and manage the impact of AI bots on their traffic and revenue.