rss-bridge 2026-01-06T08:40:00+00:00

Search engine bots crawled so AI bots could run

Ryan hosts Akamai data scientist Robert Lester on the show to discuss how the growth of AI bots affects internet traffic, the ways these AI bots differ from the original search engine optimization ones, and why you might not want to mitigate AI bots on your websites.

January 6, 2026

Search engine bots crawled so AI bots could run

Akamai is a CDN, full-stack cloud computing, and cybersecurity company that keeps experiences closer to users and threats further away using the world’s most distributed compute platform.

Connect with Robert on LinkedIn and check out his AI Pulse blogs.

Today’s shoutout goes to user Evan Phoenix for winning a Populist badge for their answer to llvm ir back to human-readable source language?.

TRANSCRIPT

[Intro Music]

Ryan Donovan: Hello everyone, and welcome to the Stack Overflow Podcast, a place to talk all things software and technology. My name is Ryan Donovan, and today we're talking about all the AI bots, and the traffic, and the effects that it has on the internet. And my guest for that is Akamai Data Scientist, Robert Lester. So, welcome to the show, Robert.

Robert Lester: Thank you, Ryan. Happy to be here. Been a big fan of Stack Overflow and the work done there for a long time. Very exciting.

Ryan Donovan: We love to hear that. And as a site that is concerned about the traffic we receive on the internet, this is a topic close to our heart. But before we get to that, [we] would like to get to know you. How did you get into software and technology?

Robert Lester: I did not start my academic and professional career here. I actually started in ancient languages. That was a big interest of mine for a long time, and studied that, and it led me to a natural evolution towards language and logic problems that got me into computer science and engineering; and it's led me towards data science where I get to do a blend of engineering and problem solving, but also data storytelling and crafting. So, I really like both of that.

Ryan Donovan: So, what was your favorite ancient language?

Robert Lester: I spent a lot of time reading ancient Greek and Latin poetry, primarily.

Ryan Donovan: The classics. There you go. So, today we're gonna be talking about the AI bots on the internet, and we've always had bots crawling the internet for search indexing, and such. But from what I've heard, it seems like the bots that the AI companies have sent out are sort of another level of traffic. Can you give us a sort of overview of the research that you did on this?

Robert Lester: If we back up, it kind of starts with classification and taking a look at where we are in the evolution of all this stuff. So, when we think about the tech giants, the traditional ones, especially those that already have scrubbed the internet for a large majority of its data, like Google or Amazon, those who already have these sort of products built out, they've already got these massive internal repositories of data, as well as the infrastructure in place already to be scraping the internet daily, updating their indexes and all of that. So, from a training presence, we only classify really as the AI bots in this space, those kind of adjunct research bots like you might see, for example, the Google Vertex lab. It's really difficult to engage with a customer sometimes and say, Google bot, traditionally, you want to rank high in search rankings. This is something that has been going on for 15 years on the internet, but then, at the same time, the same data is getting mixed with their AI training data. So, it's difficult to draw that line. But then, from another perspective, you also see, getting away from that are primarily for training data, we get over towards like user-driven activity, like 'fetchers,' is what we classify them as. And these are invocations of external fetching when users are using the model. And for someone like OpenAI, they don't have that search index already entirely built, or Anthropic. And then we see with something like Google AI overview, they're able to make internal fetches towards their already indexed results. So, it's a question of presence and categorization rather than these companies taking over.

Ryan Donovan: The fetching is like when you do a research query, or it pulls in just-in-time data it does inference on the fly, right?

Robert Lester: Exactly. Yeah. That's how we're classifying them.

Ryan Donovan: So, my sense was that the AI bots are putting a lot more bot traffic on pages. Is that born out by your research?

Robert Lester: So, it's not a massive needle mover yet, but the growth rate is what we're more interested in. As far as the raw numbers, we're still only looking at this stuff as about a percent of all of the validated bot traffic that we see on a daily basis. But this is a massive growth over what we were seeing at the beginning of the year, or last year at this time, where we've gone up, I think, 400% across all industries. So, it's been a pretty incredible increase and something that we're definitely keeping our eye on.

Ryan Donovan: Yeah. The way that AI companies have used data has changed in the last year. The beginning was just all for training data, and now it is that sort of reasoning model chain of thought, like age agentic stuff. Do you see the agentic stuff sort of increasing that traffic load even more?

Robert Lester: So, part of the problem here is drawing lines between what's bot, what's not a bot. If you think about what an agent is, it's automated like a bot, but it's reasoning in a more intelligent manner than a bot, and it's non-deterministic in that fashion, a lot of the time. And so, the behavior isn't quite the same. Similar to classifying these user-driven fetchers, It's hard to draw that line. And so, what we're kind of moving towards is more of identification and intent of these bots, or whatever you wanna call them, these entities that your online products are interacting with, and moving away from 'bot or not.' 'Cause that is, seemingly, a less important question at this point.

Ryan Donovan: I think I remember that site on the early internet, Bot or Not.

Robert Lester: Yeah. It's very different. It's rapidly evolving, and it's pretty cool.

Ryan Donovan: Yeah, because when you have an AI agent, it's almost like giving everybody their own sort of bot.

Robert Lester: In a way. Absolutely. Or it's something that these large language models are doing as well, is increasing access for people. So, while we're seeing this rise in AI bots, there's also been increased internet activity across the board.

Ryan Donovan: So, you'd said the big majors have everything already indexed. Basically, they have a copy of the internet on their servers.

Robert Lester: Something like that. And I won't speak for them necessarily, but they have a lot of data at their disposal. And the key thing is also they're reusing infrastructure in a lot of cases to where if you wanna classify it as an AI bot, sure, you absolutely can. And in some cases, that makes sense, but it also makes sense to classify it as a traditional search engine optimization bot.

Ryan Donovan: Do you think the other AI companies will start doing this? Should they do this? Is there a reason that they don't?

[...]

Original source