# The AI Rankings Just Got Reshuffled — Transcript (2026-10-29) https://aidailybrief.ai/e/2026-10-29 · Listen: https://pod.link/1680633614 --- [00:00:00] so frequently on this show, you will hear me ramble off a bunch of numbers that are supposed to tell you something significant about a new model that has just come out and more often than not, those benchmark numbers are in some ways related to Artificial Analysis, the company that has become the leading provider of the overarching benchmark that people use to judge AI models and as AI models become more significant, the benchmarks that tell us how they perform relative to one another consequently get more important Today we are looking at a major update to artificial analysis's intelligence index as well as a new leader for the benchmark which comes, believe it or not, from Google The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI. All right, friends, quick announcements before we dive in First of all, thank you to today's sponsors, KPMG, Cdata, Granola, and Blitzy Get an ad free version of the show, go to patreon.com/aidailybrief, or you can subscribe on Apple Podcasts And if you [00:01:00] wanna learn more about sponsoring the show, send us a note at sponsors at aidailybrief.ai. On aidailybrief.ai, you can also find links to everything else going on in the ecosystem, free trainings, paid trainings like our new Champions program at Superintelligent, or of course, the new How We Got Here podcast. and the first episode out this week called How Tech Lost America Again, you can find all of those links from aidailybrief.ai Welcome back to the AI Welcome back to the AI Daily Brief headlines edition, all the daily AI news you need in around five minutes. We kick off today with a story that could have easily been the main, and honestly probably would have been if more people had had their chance to get their hands on this model Google has unveiled their next frontier model, Gemini 4 Carbon, just two weeks after the release of Gemini 4 Argon. As I have mentioned on this show a number of times since Argon arrived, having a new base model means Google can iterate quickly with reinforcement learning rather than needing to complete a lengthy pre-training run to put out a new model. And this is definitely a quick pace, one of the shortest gaps we've seen between versions of ultra-large models CEO Sundar [00:02:00] Pichai even suggested the pace would increase from here, commenting on Wednesday night's earnings call, " This rapid progress underscores our incredible model momentum. We are at the point where AI is helping us increase our training and launch velocity. I would expect the next 12 months to see continued acceleration because of RSI." so so Google is iterating at a record pace but I'm sure you are sitting there asking yourself the same question that everyone is, which is, is the model any good? In particular, is it a significant improvement on the underwhelming release of Argon? Certainly when it comes to the benchmarks, the answer is a definitive yes In coding, Carbon gained eight points on TerminalBench 4.0 and six points on Frontier Suite. Cybersecurity saw a 12-point increase on Exploit Gym and a three-point increase on CWE Bench. For computer use, Carbon picked up seven points on OSWorld and eight points on Agent's Last Exam. On six of those eight key benchmarks, Gemini 4 Carbon is now the top of the leaderboard, ahead of Astra and Opus 5-5. Gemini 4 Carbon is now the front runner on the new artificial analysis index V5, which we will be discussing in the [00:03:00] main part of the episode. It scored 59 points, representing a nine-point jump on the new scale and placing it ahead of Opus 5.5 in first place On the AA Coding Index, Carbon landed in second place, tied with Sonnet 55 and three points behind Opus 55. Now, these are huge jumps for Google When Argon was released two weeks ago, some felt that Google had missed their window. The model might have been impressive a month or two earlier with benchmark scores topping Fable, but the release of Opus 55 and GPT-61 Solhad pushed the frontier sufficiently that it left Google in the second tier alongside Meta This release, if people's experience with it matches the benchmarks, puts Google genuinely back in the race for the frontier Cost is basically unchanged compared to Argon with artificial analysis measuring cost per task at seven dollars and seventy-six cents. for reference, that's twenty-three percent cheaper than Opus 5.5 and twenty percent more expensive than Astra, so right in the middle of the road for frontier models Now here we come to the less great part. We don't have many first-hand testimonials as Google unveiled the [00:04:00] model before it's ready for general release. They're following the administration's pre-release model testing guidelines and said they're taking what they call a deliberate approach to the rollout. The model will be made available to a subset of cyber defenders beginning this week, with access extended to trusted partners in their cyber defense program. As for the rest of us, Google wrote, " We're continuing to gather feedback and iterate on guardrails before making Carbon available to developers, businesses, and consumers via our product services as soon as possible." one of the one of the few groups talking about their experience with the new model was Arena, who had access for testing purposes they put out a video review full of render and game design tests that were generally slammed for not being as pretty as Opus 5 five I, now, as I have said before, I am not particularly convinced that 3D design is the right yardstick for LLMs,no matter how much social media users like to focus on that. But certainly, at least when it comes to that particular use case, Carbon is not at the same level as its rivals Arena's Peter Gastev did give a few more details, however, posting, " I've used Carbon a bit before the release, and I can tell you it feels a lot better than Argon. This is a real deal from Google [00:05:00] this time. I hope they keep up with the rapid pace of releases from now on." And that does seem to be the plan. responding to TR Taxes commenting that Fable 5 isn't here yet though, DeepMind researcher Zachary Natto wrote, " Gemini redacted isn't here yet though Chubby on X commented, " Okay, now things get crazy. They have another unreleased Gemini model that's even better than Carbon, and Carbon has already reached number one in the AI Intelligence Index today Gemini post-training researcher, Xiao Zhang added that Google AI is out of their funk and operating like a well-oiled machine Less than a year ago, he wrote, " Launching Gemini 3 Pro was a big bang event where the whole team celebrated number one benchmark numbers. Today, Gemini 4 Carbon hits number one on theAA index, yet the feeling inside is distinctly different. We've been dogfooding continuously for weeks, and we've almost adapted to the new intelligence, taking ambitious code rewrites for granted, leaving loops overnight that launch dozens of experiments to repo, root cause, and fix mysterious bugs, and using swarms to inspect and fix RL data. Launches just feel like periodic checkpoints now." and despite taking 10 months off [00:06:00] the early adopter AI conversation. You can see that Google still has lots of people's benefit of the doubt Crusader Bench's Hessem writes, " Why is Google suddenly dropping models like they invented transformers, own half the Internet's data, and sit on an ocean of TPUs? Oh, wait." Latent Spaces Twix writes, Gemini number one on AAV5? This is making me open up anti-gravity for the first time in a long time. If Google actually pulls this off, it is officially comeback season with Meta, Grok, and now GDM being cool again." For my part, I think we will suspend judgment until we get our hands on the actual model but it looks promising, and that in and of itself is exciting Now speaking Now speaking of Google, the other story was in their and Microsoft's earnings reports. Both companies delivered another incredibly strong quarter in cloud growth Which is where, of course, they contain their AI revenue. Microsoft said their Azure division grew at 48% over the past year to reach 33 billion This is the first time Microsoft has broken out Azure revenue as a separate line item, which seems like a very good decision with growth this strong. Azure drove overall revenue growth of [00:07:00] nineteen percent to reach ninety-two point three billion for the quarter. Each of those figures was a slight beat over analyst estimates. Microsoft also disclosed that OpenAI contributed ten billion to Azure revenue, around a third of the total business Microsoft stock was up as much as four point eight percent to reach a new all-time high in after-hours markets Google's Cloud division, meanwhile, doubled revenue compared to year-ago figures, reaching thirty point seven billion for the quarter, and meaning they've almost entirely closed the gap with Microsoft. This was well above Wall Street's high expectations. However, as David Wagner of Aptus Capital Advisors put it, " The problem is everything else." Search revenue growth came in at twelve percent, a massive slowdown from seventeen percent in the previous quarter and nineteen percent the quarter before that YouTube revenue also added a third continuous growth slowdown to reach 9.2%. Even TPU sales was a bit of a question mark, with executives dodging a question about profit margins for the chips during the earnings call. Overall revenue growth still came in at twenty-five percent, roughly in line with expectations But to some analysts, it looks like Google might have reached the moment where they're forced to cannibalize their own [00:08:00] search business because it's no longer a reliable growth driver In fact, CEO Sundar Pichai acknowledged this dynamic during the earnings call, commenting, It's worth noting that the strong growth in AI mode queries had an impact because of the lower monetization we're seeing in AI mode." in addition, Google missed on bottom line earnings, largely due to a markdown on their SpaceX stock tech analyst Dan Ives, ever the optimist, tried to strike a positive tone, writing, " This is the cloud print we have been waiting for, and in our view, it matters more than a lighter search number the bears will lean on tonight." And lean on it they did, sending the stock plummeting as much as 10% in after-hours markets Mi-much of the coverage focused on the cloud race between the two companies Which are now neck and neck, but the pace of the data center build-out might be even more consequential. Both companies held CapEx forecast steady for this year and next, but actual CapEx spend for the quarter came in under expectations. Google, by way of example, is guiding a fifty percent increase in CapEx next year to three hundred billion, but they're currently undershooting this year's numbers. Oracle and Meta are in a similar position. So it [00:09:00] looks like the AI build-out may have reached peak velocity That's not at all surprising given physical constraints around components, power, and labor, let alone community support running on empty Now, we Now, we are quite early in this trend, and Google said that they would have more in-depth discussions on CapEx during the next quarterly earnings. But what makes this interesting is that many AI bears suggested that a CapEx slowdown would be the catalyst that pricks the AI bubble what we're seeing, however, doesn't exactly line up with that theory. The logic was that demand would falter and force the hyperscalers to slow down. Instead, it looks like the hyperscalers, if they had their druthers, would like to continue accelerating their data center build out, but are running into more fundamental and physical constraints Tech analyst Luke Longo thought the slowdown was more about finance but are running into more fundamental and physical constraints. constraints. Lastly Lastly today, more antitrust scrutiny for NVIDIA as the DOJ queries their acquisition of Hugging Face. Reuters reports that the Department of Justice has made inquiries about how the acquisition would impact competition in the AI industry. In [00:10:00] particular, the DOJ has asked whether ownership of Hugging Face would allow NVIDIA to control a key distribution point for AI models. When they made the $13 billion acquisition in September, NVIDIA maintained that Hugging Face would remain free to offer a platform and partner with a wide range of model developers, chip manufacturers, and cloud providers. They said the deal was expected to close in the first half of 2027, pending regulatory approval these inquiries form part of what is now revealed to be a multi-year antitrust investigation into NVIDIA In September, The New York Times reported that the DOJ had opened an investigation in NVIDIA's licensing and talent acquisition deal with chipmaking startup Groq. That deal was structured as an acqui-hire deal, which has been a common practice in the AI industry dating back to Google's 2024 deal with Character AI. Because these deals are not technically acquisitions, they don't trigger automatic antitrust reviews and can therefore theoretically close much faster. The assumption had been that the DOJ was using an investigation into NVIDIA as a way to crack down on this practice more broadly. However, sources told Reuters this week that the investigation is much broader and seems to predate the Groq deal. [00:11:00] Acquisitions of data center optimization companies SketchMD and Run:ai are also within scope for their impact on the competitive landscape and AI infrastructure. And both of those deals were traditional acquisitions, not acqui-hire deals. Furthermore, sources said that recent questioning from the DOJ, covered NVIDIA's venture investments, sales practices, and whether the chip designer has exerted pressure on its customers In other words, rather than just a narrow investigation about circumventing antitrust rules, it now seems the DOJ is embarking on a full top-to-bottom antitrust investigation of NVIDIA Obviously, no charges have been filed at this stage, and the investigation could conclude with no action. So there's no implication that Nvidia has done anything improper Still, compared to what most people thought this was, it appears to be a much more concerning investigation for the dominant chip maker. Something to keep an eye on as we go forward for sure, but for now, that is gonna do it for today's AI Daily Brief. Next up, the main episode A new study from KPMG and the University of Texas at Austin found that when people work with AI, similar [00:12:00] skills don't guarantee similar outcomes. Researchers studied more than five hundred early career professionals and found that the best performers consistently amplified the value of AI by guiding, evaluating, and refining its outputs. These top performers, called AI amplifiers, weren't defined by what they knew alone, but by how they worked with AI. Learn more about what separates AI amplifiers from everyone else at kpmg.com/us/aiamplifiers. This episode is brought to you by Cdata. Token spend is climbing, security is holding agents back, and companies rolling out AI initiatives might not know how it's truly being used The root cause: AI is cut off from the systems and context your business runs on. That's why CData built Connect AI Gateway, one control point between your AI and your systems Connect AI Gateway is not just another static model router. prompts get intelligently routed to the right model with exactly the tools and data it needs governed down to the record The result? Accurate answers at a fraction of the [00:13:00] token cost with a context engine that learns your business with every interaction. Built on the connectivity layer powering Palantir, Google, Salesforce, and hundreds more. Give it a try and start your free trial at cdata.ai. That's C as in connect data.ai You're in back-to-back meetings all day. You're trying to stay present, but you're also worried you'll forget the decision, the action item, the important next step. This is a scenario that we absolutely live here at the AI Daily Brief. Given how fast things move and change and how quickly we're evolving everything that we do That's where Granola comes in. Granola is an AI-powered notepad for You jot down rough notes like you always do, and in the background, Granola transcribes and turns them into clear, useful notes when the meeting ends. No bots joining your calls, no distractions, just a clean notepad that helps you focus. During or after the call, you can chat with your notes, ask Granola to pull out action items, help you negotiate, write a follow-up email, or even coach you using recipes, which are pre-made prompts Once you try it on a first meeting, it is very hard to go without head to granola.ai/aidaily And [00:14:00] use code AIDaily To get 100% off for three months for new users That's granola.ai/aidaily and code aidaily to get 100% off for new users for three months Blitzy's deep code base understanding unlocks the thing every roadmap owner cares about: shipping new features. Here's the truth about building inside a massive enterprise code base. Writing code was never the bottleneck. Context is Which system does this touch? Which contracts can't break? Which standards apply? Blitzy already knows because it reverse-engineered your entire code base into a dynamic knowledge graph before feature work began. With that complete picture, Blitzy builds features end to end. Architecture, APIs, UI, and tests all validated against your existing systems One Blitzy customer built an AI native application from scratch with 100% autonomous completion, saving over 2,700 engineering hours. Features that respect your code base instead of fighting it Stop letting your backlog grow faster than your team. Accelerate your roadmap at blitzy.com. That's B-L-I-T-Z-Y.com [00:15:00] welcome back to the AI Daily Brief. Brief. Today we are talking about benchmarks. Specifically a new set of benchmarks and how they're shaking up our sense of the AI field Now, the eagle-eared among you may wonder exactly what I think about benchmarks, given that on the one hand, with every new model release I implore you to only give them so much attention and yet at the same time still rattle off a whole bunch of them as though they matter in short, I think that benchmarks are inherently limited, but also important. without them, we don't have good tools other than strictly vibes to understand the difference between different models and yet in almost no circumstances do I think we should take the performance of one model on any one benchmark or even in the aggregate as the gospel. done well, they are a useful guide to helping us understand where different models fit into our overall stack But as we will see today, there is more and more effort to addressing some of the inherent limitations of benchmarks and trying to make them better Whenever there is a new model, you'll hear me talk about the scores that it gets on the independent test [00:16:00] administered by artificial analysis. Well, that company has just overhauled the way that they test model giving us a brand new yardstick for Frontier AI with version five of their intelligence index. The index takes around a dozen key benchmarks and distills them into a single score and while the index is now entrenched in how we understand the AI race, leaning so hard on that single score is a relatively new phenomenon. Prior to the era of agentic coding When LLMs were in many cases still glorified Google or writing assistants, vibes reigned supreme Ranking on the Chatbot Arena leaderboard was the pinnacle test of a new model because it told developers what users wanted But as research and coding became more important use cases, we got more objective benchmarks giving a clearer indication of howeach model would perform on real work These evaluations have evolved over time. First, by adding harder questions as benchmarks became saturated in other words, as all the models crowded the very top score of a benchmark, the creators of said benchmark would go add in a bunch of new or harder questions creating an iterative version that would hopefully provide [00:17:00] more granularity on the difference between different models until that model got saturated as well As As agentic computer use became a fundamental part of model performance, the evaluations got an even more complete rethink through through each of these changes, artificial analysis also evolved, switching out benchmarks and rebalancing their weightings as different model abilities became comparatively more important This hasn't always been a smooth process. There have been moments where the intelligence index looked extremely saturated and stopped giving us a good indication of which models were best. There were moments when benchmark maxing allowed developers to game the system and rank much higher than real-world testing would support. In fact, this has been a fairly consistent story when it comes to Chinese open weight models. In In early September, artificial analysis pushed an early update with version 4.2, pulling forward many of the changes they had planned for V5. that decision was triggered by the release of GPT-6 Astra, which focused heavily on improving computer use. The problem was that V4.1 of the intelligence index still had relatively light weightings for agentic benchmarks, leading GPT-6 [00:18:00] Astra to have a slightly disappointing And more than just disappointing, one that didn't really comport with people's lived experience of the model After the update, Astra was still ranked slightly behind Fable-5-1, but the index had become a closer reflection on how people were actually using the new crop of models And this has been the story of the Intelligence Index, a constant cat and mouse game of updating the mix of benchmarks to match the rapidly evolving real world use cases for AI Introducing version five of the index, artificial analysis highlighted that for the first time, the majority of the benchmarks are built on private datasets. The idea is to make benchmark maxing more difficult as model developers theoretically can't train their models to the test. And that, by the way, is what we mean by benchmark maxing When developers include those sort of evaluation tests and their answers in the training data set leading to an artificially inflated score as the benchmark ceased to measure what the model can do and instead reflects simply a memory of the correct answers. For version five, AA added two new benchmarks Frontier Code 2.0, which is the brand new version of Cognition's private [00:19:00] benchmark of real world coding tasks This test features scoring based on whether a repository maintainer would merge the code. They also added Terminal Bench Science, which is an agentic scientific research benchmark Humanity's Last Exam, once pitched as a near impossible knowledge search task, has been upgraded to HLE Diamond, the even more difficult version. Similar saturation updates have been made to AA Omniscience, which measures knowledge and hallucinations, as well as Crit PT, a physics test GDPVal AA, the measure of agentic work in real-world settings is out, With the index now relying on AA Briefcase, AutomationBench AA, and GDP.pdf to measure white-collar work along more granular lines. Also removed are PsyCode, a scientific research test, and AALCR, which tests long context reasoning Now, Now, rather than just a list of names and acronyms, these adjustments say a lot about how AI models have changed since V4 of the index was released in January at that time, long context reasoning was one of the most critical features used for working on large code bases Many coding models still had [00:20:00] limited token context windows and suffered from fatal issues with context rot. Since then, improvements to compaction and memory have allowed us to take a lot of that long context performance for granted. Agentic work is the other huge change that's evident from the redesign of the index The coding benchmarks included in the index now only measure agentic performance, as that has been cemented as the way developers use modern AI and the science benchmarks are now testing real-world agentic research work rather than pre-existing scientific knowledge For the time, Artificial Analysis has chosen to equally weight the index across four categories: knowledge work, coding, science in general Which encompasses both general knowledge as well as hallucination rate In some ways it feels like that 25% waiting for science is a forward-looking bet based on how Anthropic and OpenAI have been trading science and math headlines over recent weeks So So what did this change in the scores? In version five, Opus 5.5 was still in the lead with a score of 58, at least until Google dropped Gemini 4 Carbon on Wednesday. GPT-6 Astra jumped into then second place with a score of 57, and Astra had previously [00:21:00] been in sixth place, showing there's a pretty big jump based on the index giving more weight to computer use in scientific tasks GPT-5 1 Soul takes third place up from ninth, followed by Sonnet 55 which slipped from second to fourth. GPT-6 1 Terra was in fifth up from 10th, suggesting the model is a little better than many assumed earlier in the week In In just a moment, we'll get into what I think that reflects Gemini 4 Argon gained a rank to tie Meta Wonder full in sixth place, which is a pretty notable drop for Wonder, which was previously in third, although many felt that that ranking didn't line up with real-world experience. Fable 5.1 is looking a little long in the tooth in eighth place, sliding from fifth. And in ninth place, we have one of the most notable drops in the new index with KimiK 3.1 Pro falling all the way from fourth In fact, artificial analysis noted that the gap between open weight models and US proprietary models had opened up massively after the index rebalancing. They wrote, " The open weights gap widens from three points to 12. KimiK 3.1 Pro is the top open weights model at 46. Among models with [00:22:00] released weights, DeepSeek V4-1-Flash and GLM 5.3 lead at 32, 26 points behind the leader and below Claude Haiku 5.5 with 36 points." Now, in addition to the new gulf between US and Chinese models, there was also a clear drop-off between the handful of leading labs and everyone else. Anthropic, OpenAI, Google, and Meta all have models with scores of 50 and above Fable 5.1 and KimiK 3.1 Pro are just outside that cutoff. However, there's then a 10-point gap between KimiK 3.1 and Claude Haiku 5.5, which leads the rest of the pack Basically, by adding more agentic tasks to the index, there's now a clear separation between frontier models and the rest Artificial analysis noted that costs vary wildly among the top 10 models that arguably make up the frontier That's largely because the top 10 is a mix of large models, token-intensive models, and a few smaller, more efficient models At each end of the spectrum, we have GPT 61-Tera as the cheapest at 96 cents per task, and Fable 5.1 is the most expensive at $13.04 per task [00:23:00] For a more apples to apples comparison based on model size Opus and Sonnet both cost around $10 per task, compared to $6.24 for GPT-6 Astra and only a buck 27 for GPT-6-1 Muse, Wonder, Grok 4.7, and Gemini 4 Carbon are all slightly more expensive than GPT-6 Astra in the high $7 range Now, most people found this to be a welcome update OpenAI Head of Strategic Futures, Dean Ball wrote, " This is the first time an artificial analysis ranking has even roughly approximated my own subjective ranking of models." Haseeb Qureshi wrote, " Artificial analysis re-ranking their intelligence score Seems to match the vibes a lot better and clear out a lot of benchmarking. Shows a deeper gap between Anthropic, OpenAI, and everyone else. Also shows a bigger gap between closed and open weights Ethan Ethan Mollick wrote that the analysis, quote, "Matches the general vibes that we haven't seen a fable class open model." " And AI content creator Theo drove that point home, writing, " Terrible day to be an open weights models are just as good as Opus 55 guy." Now, there were a few people who spoke up for the open weights [00:24:00] models Brookings Institute fellow Kyle Chan pointed out that even before this artificial analysis had some private benchmarks, it's just that the share went from 45% to 60%. using Kimi 3.1 Pro score as an example he suggested that while the headline news is the gap with the top US model widening from three points to 12, that it's more important to look at the specifics science he points out is a major weakness, with Kimi scoring just 34% on Terminal Bench science versus 63% for GPT-6 Astra. But he writes, Kimi is still extremely strong at coding agents, pointing out that it actually leads all models on Terminal Bench 4.0, scoring 67%. Kyle's conclusion Is this evidence of benchmark gaming? Not necessarily. Some of Kimi's biggest deficits are on non-private benchmarks, while it actually beats Claude on the private Automation Bench AA evaluation. The bigger story, he says, Chinese models can match or beat US frontier models on certain agentic tasks, but a significant gap remains in advanced scientific reasoning and research workflows gen-- and ultimately, even if in general people think that this is a positive [00:25:00] update no one is going to agree on everything. In particular, I think one of the biggest debates will become what the relevant weighting should actually be. yes, OpenAI and Anthropic are clearly emphasizing scientific discovery. But relative to the average individual use case, does that matter as much as agentic coding as it does in the weighting of the new V5 index? Now, that might be an argument for a different weighting, or it might be an argument that instead of paying attention to the headliner number, we'd be better suited to focus on the individual benchmarks for the categories of use cases that we care about one one critique that I do think is worthy of consideration is the fact that the leaderboards are run based on max settings Sridot World writes, " No one runs max in real life. Where's the medium/high/ex- extra high leaderboard?" And to be clear, it's not like artificial analysis ignores that, especially in their analysis that involves cost The effort setting does become part of the calculus but I do think that on a practical level People might be better served By the ability to organize the index, not just by model, but also by effort now now one last note. alongside the new intelligence [00:26:00] index, Artificial Analysis introduced a new progress index. The index places all models released since November 2022 on a capability scale to see how each lab's leading model improves over time. it's designed less to be about the current batch of models and more to give a long-term view of model progression Still, the new measure gave people easy access to a bunch of data to play with Lousan Al-Gaib, for example, looked at how the switch to reasoning models impacted the major labs Commenting OpenAI really does have the best reinforcement learning, or their models just suck without reasoning And by charting the data in a different way They estimated that the gap between the US and China is now 3.3 months and closing closing Summing up, like I said at the beginning, I think benchmarks are important and inherently limited, but I love that more effort is going into improving them and I think even the debates that follow are also extremely value accretive when it comes to how we understand how different models compare to one another. For For now, however, that is gonna do it for today's AI Daily Brief. Appreciate you listening or watching as always, and until next time, peace ​ [00:27:00]