Dear friends,
The gulf between LLM performance on verifiable tasks vs. unverifiable tasks that require judgement is larger than most people realize. Verifiable tasks, like coding or solving math problems, have a right answer that can be checked automatically. Judgement tasks — like how to respond to a customer complaint, how to organize data collection for your AI project, what product feature to add, or how to form business strategies — don’t have unambiguous, ground-truth, correct answers. But they do have better and worse answers.
We have much better techniques for training AI systems to do well on verifiable tasks. As a result, progress on performing such tasks has been much faster than for judgement tasks. In the case of LLMs, RL (reinforcement learning) has been a huge driver of progress. We can set up RL gyms, or simulation environments, in which we let an LLM practice different ways to approach a task; then we automatically check which approaches are more successful (as measured by a reward function) and tune the model’s parameters to take the successful approaches more often.
For supervised learning tasks, too, verifiable tasks are easier to improve performance on. We can obtain unambiguous labels for verifiable tasks, either via automated labeling or by asking human labelers. Defining the right label for a given output in judgement tasks is much harder. Furthermore, learning theory shows that machine learning algorithms can learn faster for tasks where it is possible to get to 0% error (called realizable tasks in the literature) than ones where the optimal error (called Bayes error) is non-zero.
It’s true that applying supervised fine-tuning to LLMs can help them learn to perform unverifiable tasks, and progress is being made here too, but data collection and eking out incremental progress is harder and slower. Intelligence is jagged, and we do have a recipe for improving AI’s capability for judgement tasks: We can collect training data and train on a specific domain, laboriously craft reward functions that capture our goals, and so on. For the subset of applications where this is valuable, businesses will invest to get this work done faster.
Society also has a long history of looking at AI’s performance on a subset of tasks and overgeneralizing its capabilities. We saw this when AI beat the human world chess champion (Deep Blue), won Jeopardy (Watson), and excelled at Go (AlphaGo). Each of these moments led to hopes about AI’s immediate applicability to other tasks that turned out to be over-optimistic. Further, we can more easily evaluate performance on verifiable tasks (since test sets are easier to create) than unverifiable ones, which creates a more optimistic impression of progress than reality can justify.
To be clear, there is a lot to be optimistic about in AI — including progress on verifiable tasks and judgement tasks! But the tremendous progress on verifiable tasks, coupled with hype around AGI and RSI, is leading people to overgeneralize to AI’s ability to do judgement tasks well. In the long term, I remain bullish about AI’s ability to do this well too, but a lot of work remains ahead.
One interesting challenge for AI Engineers is that, as verifiable tasks get automated, the need for humans to do judgement tasks — which often requires drawing context from a bewilderingly large array of raw data and identifying the pieces that critically influence a decision — remains important. As I’ve written before, I expect people to continue to have a context advantage over LLMs on many tasks for a long time.
Keep building! Andrew
A MESSAGE FROM DEEPLEARNING.AI
In a fast-moving field, give your team skills you can trust. We analyzed 10,000+ job roles to find out which AI engineering skills matter most. DeepLearning.AI for Business gives your whole team a structured path to those skills. See how it works
News
Claude Fills Out Its Model Family
Claude Haiku, Anthropic’s fastest and least expensive model gets a long-awaited update. Meanwhile, the new version of Claude Sonnet nearly matches Claude Opus 5.5 on an index of independent benchmarks, but only at a reasoning level that uses more tokens per task than any other model tested.
What’s new: Claude Haiku 5.5 replaces the year-old Claude Haiku 4.5. Independent tests show marked improvement, even with a steep price. Two weeks earlier, Anthropic launched Claude Sonnet 5.5, which the company says is best suited to routine coding and office work, including fixing bugs and producing documents, slides, and spreadsheets. It replaces Claude Sonnet 5 at the same per-token prices.
How it works: Anthropic disclosed few details about how it built Claude Haiku 5.5 or Sonnet 5.5 beyond their reasoning controls and safeguards.
Performance: Claude Haiku 5.5 at max effort outperforms DeepSeek V4.1 Flash and all other “fast” models, but behind the top models. The latest Haiku version is also the fastest model Artificial Analysis has tested by tokens per second. Claude Sonnet 5.5 set to max effort ranks first or second on three leaderboards from two independent testers. But at every effort setting, another model matches or outperforms Sonnet’s Artificial Analysis score at a lower cost per task.
Behind the news: Anthropic launched Claude Haiku 5.5 on October 7, Claude Sonnet 5.5 on September 28, and Claude Opus 5.5 on September 22, At launch, Opus 5.5 led Artificial Analysis and Vals AI’s overall indices. Even with the later releases of Gemini 4 Argon and Muse Wonder and Muse Wonder Full, Claude Opus 5.5 and Sonnet 5.5 remain the top models on independent benchmark indices measuring overall intelligence.
Why it matters: Per-token prices alone don’t tell developers what a task costs. On Artificial Analysis’ Intelligence Index tests, Claude Sonnet 5.5’s cost per task varies more than 18X across reasoning levels, and at max, it costs more per task than Claude Opus 5.5 set to max reasoning. On Vals AI’s tests, the cost is about two-thirds as much. Anthropic concedes Sonnet’s cost advantage over Opus is strongest at its lower reasoning levels. Claude Haiku 5.5, too, can be used as a reasoning model, but its cost advantages stand out for simpler tasks like summaries and classifications. Developers should measure each model’s cost at each relevant reasoning level on their own tasks, alongside accuracy and completion time, since each varies significantly.
We’re thinking: A higher reasoning level can change what an agent does, not just how long it reasons, and the change doesn’t always help. On FrontierCode, a test of whether an agent’s code changes are ready to merge without human edits, Anthropic says that Claude Sonnet 5.5 scores lower at max reasoning than at xhigh (46.2 percent versus 52.1 percent). Test your tasks on different effort levels to see what works best.
Mistral’s Sovereign Appeal
Mistral’s latest model is its largest and most capable, and the most powerful yet released by a European company. Its performance on independent benchmarks is comparable to GPT-6 Luna. It trails less-expensive China-based open weights models but performs better than all U.S.-based open models. It leads all models in finding and repairing a class of software vulnerabilities.
What’s new: Mistral announced API preview access to Mistral Large 4, nicknamed “le Chonk.” Mistral says it will release the model’s weights by the end of this month. Meanwhile, selected companies, partners, and government agencies have access to a version with limited safeguards and extra cybersecurity capabilities.
How it works: Mistral has said little about Mistral Large 4’s architecture but says it will share more when the model’s weights become available. The company says it pretrained the model from scratch and then fine-tuned it using supervised learning followed by reinforcement learning (RL).
Performance: The preview of Mistral Large 4 outscores every other model built outside China and the United States on Artificial Analysis’ popular Intelligence Index. It trails a handful of other open weights models, all from companies in China, but is ahead of all U.S.-based open models. The model preview’s strongest results come in cybersecurity tasks.
Behind the news: Mistral’s flagship models have alternated between closed and open weights and have been sometimes more and sometimes less competitive with top proprietary models. The company introduced Mistral Large, a proprietary model, in early 2024. Mistral Large 2 followed in July 2024 with weights licensed for research and non-commercial use; at the time, Mistral Large 2 was competitive with OpenAI’s GPT-4 and Anthropic’s Claude Sonnet 3. In December 2025, Mistral released Mistral Large 3 under the permissive Apache 2.0 license, along with smaller Ministral 3 models, which the company built by repeatedly pruning and distilling the larger model.
Why it matters: The most capable open weights models on popular benchmarks are from companies based in China. Developers in the United States and Europe are trying to compete: In the U.S. Nvidia, Reflection AI, and Thinking Machines Lab, among others, have made progress, but Mistral remains the leading European lab. All of these companies pitch their models to organizations that want AI they can better control, whether via further post-training or greater trust they will be able to access the models regardless of geopolitics. Companies and governments all over the world, from South Korea to Saudi Arabia to Mexico, would prefer not to depend on either the United States or China for their access to intelligence. As Mistral CEO Arthur Mensch said earlier this year: “If you treat intelligence as electricity, then you just want to make sure that your access to intelligence cannot be throttled.”
We’re thinking: If you need AI sovereignty, and some industries and governments do, the best kind is running an open weights model on your own hardware. It doesn’t matter where a model was built (the U.S., Europe, China, or elsewhere) if your data and the model’s weights are entirely under your control.
Learn More About AI With Data Points!
AI is moving faster than ever. Data Points helps you make sense of it just as fast. Data Points arrives in your inbox twice a week with six brief news stories. This week, we covered Anthropic’s major upgrade to Claude Haiku and Meta’s new Muse Wonder model challenging leading AI systems. Subscribe today!
Computers Built for Local AI
Developers have warmed to the economics and security of local models, either as an alternative or a complement to the cloud. Specialized computers from Apple, Microsoft, and Nvidia aim to take a bite out of the AI market with machines capable of running larger and more capable open models on-device.
What’s new: Apple’s upgraded Mac mini and Mac Studio desktops enable developers and AI researchers to run and fine-tune local models faster than before, offering an alternative to cloud computing for certain tasks. Nvidia’s DGX Spark series and Microsoft’s Surface Ultra laptops target a similar market: customers who want enough power to build with AI on their local machines, but in a user-friendly package.
How it works: By avoiding per-token cloud or monthly LLM subscription costs, a local setup could potentially save developers money over time, particularly those running always-on agents. But genuinely capable hardware is expensive: A maxed-out Mac Studio with 256GB of memory, an 80-core GPU, and 16TB of storage costs $18,299. For those with more modest local agent needs, a Mac mini starts at $899.
Performance: Apple Silicon and Nvidia’s RTX Spark chips were recently tested by an independent website, but most scores are self-reported.
Behind the news: Professionals using local AI inference are still a small market compared to the cloud, but now large enough to have genuine competition for computers designed for them at a wide range of price points. Mac desktops have sold well since the emergence of open-source personal agents, with news outlets describing weeks-long waits for Macs with more than 24GB of RAM as early as February. Nvidia’s DGX Spark laptops and desktops aim to similarly serve Windows users. For Nvidia, entering the desktop chip market ensures it can continue to sell its products to both consumer and enterprise markets, but also to increasingly blur that distinction. For Apple, it’s an alternative way to capitalize on the AI boom rather than building and hosting models, one that plays to its strengths in manufacturing high-end consumer hardware.
Why it matters: On-device AI models have several advantages over cloud-based models besides cost: more customizability, privacy, data sovereignty, potentially lower latency, and the ability to run without an internet connection. Others see cloud models and local models as complementary, designing workflows where a more powerful proprietary model solves high-level problems, then hands more mechanical work to a low-cost local model.
We’re thinking: We argued back in 2023 that strong commercial interests would increase AI usage at the edge. Not every application needs a frontier model, especially if a mid-sized model is fine-tuned to the task. But the main attraction today for local AI are privacy and data sovereignty, which is increasingly important as AI models handle more sensitive workloads. It’s useful for all of us to have a wide range of options.
Data Mixing, Minus the Guesswork
When you’re building a fine-tuning dataset, finding a mix of subject domains that yields a high-performing model can require expensive trial and error, because fine-tuned performance also depends on the quality of the individual examples. A cause-and-effect approach aims to separate the two effects.
What’s new: Researchers at Tsinghua University, Ant Group, and Renmin University of China, with Zinan Tang and Yukun Zhang as lead authors, built CausalMix, a method for choosing the share of training examples drawn from various domains regardless of their quality.
Key insight: Datasets can be analyzed according to their data mixture (the proportions of examples that represent different domains) and data quality (variations among examples in difficulty, reasoning complexity, and/or composition). Data mixture and data quality correlate, and finding their correlation in any given dataset makes it possible to differentiate their separate impacts on the performance of a model trained on the dataset. Given a dataset and a consistent training budget, to find the marginal gain of its data mixture — that is, its effect on a fine-tuned model’s performance — we can subtract the effect of data quality from the model’s performance overall. The best data mixture is the one that yields the largest marginal gain.
How it works: The authors compiled 512 subsets of 100,000 examples each drawn from the Tulu 3 dataset. Each subset contained examples for instruction-following, coding, math, knowledge, and safety in different proportions. In addition, the authors measured data quality with respect to difficulty, reasoning complexity, and writing clarity as computed by OpenDataArena-Tool. They fine-tuned a separate Qwen2.5-0.5B model on each subset and recorded each fine-tuned model’s performance on 10 benchmarks. Then they trained three models, all of them ensembles of decision trees: one to estimate the portion of each Qwen2.5-0.5B’s performance attributable to data quality, one to estimate the portion of a data mixture that correlates with the corresponding data quality, and one to represent the effect of the data mixture on fine-tuned performance.
Results: The authors compared their method to various baselines, including distributing data across domains in equal proportions and Data Mixing Optimization (DMO), a method that automatically chooses fine-tuning mixtures based on minimizing loss. Fine-tuning on 800,000 examples, CausalMix outperformed every competing method tested on the 10 benchmarks used to select the mixtures.
Yes, but: It’s not clear how well the authors’ method extends across datasets and to other datasets. On a held-out subset of Tulu 3, CausalMix often underperformed competing methods. For example, fine-tuning Qwen2.5-0.5B on 800,000 examples from the held-out set, DMO (26.49 percent on average) outperformed CausalMix (25.04 percent on average).
Why it matters: The authors’ method provides a rational approach to optimizing data mixtures rather than trial and error, and it guarantees that a good mixture will continue to apply as data or model size scales up. This makes it possible to find the best data mixture using smaller datasets and models and maintain the same proportions when scaling up without jeopardizing performance.
We’re thinking: The impact of any given training example on trained performance deepens on other examples. For instance, one example might be too hard until the model learns from another. The impact of data mixture reflects this principle. How well a model follows an instruction may depend on reasoning it learned from contemplating a math problem, and vice versa.
Work With Andrew Ng
The Batch is looking for freelance writers to summarize recent AI and ML research. If you'd like to help us keep the AI community informed, and if you actively follow AI research and have a knack for writing, we invite you to apply.
Subscribe and view previous issues here.
Thoughts, suggestions, feedback? Please send to thebatch@deeplearning.ai. Avoid our newsletter ending up in your spam folder by adding our email address to your contacts list.
|
-Oct-15-2026-06-01-57-1804-PM.png?width=1200&upscale=true&name=The%20Batch%20top%20banners%20(1)-Oct-15-2026-06-01-57-1804-PM.png)
.png?width=1200&upscale=true&name=LECHONK%20(1).png)
.png?width=1200&upscale=true&name=AIMACS%20(1).png)
