RL on math and code trains search, not knowledge. Search is worth exactly what the scorer is worth. Outside math and code the scorer lives in the world: scattered, partly unwritable, and delivered as verdicts to whoever was in the deal.
Last week François Chollet, who wrote Keras and runs the ARC Prize, asked whether the jagged frontier is mostly math and code, which you can push arbitrarily far with RL on verifiable rewards, while everything else plateaus because it still runs on human-generated data. Luis Garicano agreed and gave the reason: the labs do almost no pretraining now, the data is running out, so they moved to RL, and RL only has clean rewards in math and code.
I think the mechanism is right and the boundary is wrong. The boundary isn't “math and code.” It's “wherever something cheap can score the answer.” The question that matters is where that scorer lives for everything else. Short answer: it's what Hayek called local knowledge, it only shows up as verdicts, and the verdicts go to the parties in the deal, not to the labs.
Even math made the point that week. OpenAI posted 722 manuscripts from an unreleased model: 372 result families, roughly 4,000 problems attempted, about three hours of ChatGPT Pro per kept result. A day later three were pulled for a sign error, and by OpenAI's own count 300 of the remaining 719 have a Lean-checked main result. The rest wait on human referees. In the most verifiable field there is, the free verifier covered less than half the output.
A year of papers opened up what RL on verifiable rewards does to a model. My read: it changes where the model turns, not what it knows.
Over 93% of the gain from RL arrives in the first third of training
Cumulative share of total change over 2,400-step RL runs, averaged across 11 models
Performance gainedEntropy consumed
.\n\nOkay and Olmo-3-7B goes from 42% to 78% on MATH-500; its RL version gets 75%. RL raised the probability of that cue from 0.14 to 0.65. A year earlier, random rewards improved Qwen2.5-Math by amplifying a habit already in pretraining. Karpathy's version, on Dwarkesh: RL is sucking supervision through a straw.If RL moves search and search is what's been scaling, capability without search should grow more slowly. It does. Think Fast measured what frontier models can do with no visible reasoning across 43 benchmarks: doubling every 373 days, against METR's 182 days for agentic tasks with reasoning. The two tracked each other until GPT-4 and have diverged since. Even without reasoning, math and science double every 246 days versus 387 for language. The prior grows slowly. Search grows fast, wherever something can score it.
With reasoning, task horizons double every 182 days; without it, every 373
50% task-completion time horizon in human minutes, log scale, by model release date
With reasoning (METR, agentic software tasks)No reasoning (Think Fast, 43 benchmarks)
The grey line is the prior. The blue line is the prior plus search, scored by a test suite. The gap between them is what a free verifier is worth.
“Verifiable” isn't a property a domain has or lacks. Everything meets the world eventually. What varies is the terms:
The first three are what people mean by verifiability. The last two are the ones I care about, because compute can't buy them down.
Jason Wei's verifier's law: how easy a task is to train on is proportional to how verifiable it is. Underneath it is an asymmetry. Checking is usually much cheaper than doing (a Sudoku takes an hour to solve and a minute to check), and that gap is what makes RL pay. Where it inverts, like fact-checking an essay, RL stalls.
Even code's verifier is a stand-in. METR had four maintainers review 296 agent PRs that had passed SWE-bench's tests. About half wouldn't have been merged, and merge rates ran about 24 points below the automated score. OpenAI had already stopped reporting SWE-bench Verified after auditing 138 tasks its own model couldn't reliably solve and finding flawed tests or descriptions in 59.4% of them. Tests are a cheap proxy for a maintainer. Very cheap, and still a proxy.
Sutton's line on Dwarkesh that LLMs have no ground truth is, in these terms, that the model has no verifier of its own. The world does. The question is the terms on which it shares one.
Only math and code get a verdict that is instant, clean, and public
Illustrative placement: delays are orders of magnitude, noise is a judgment of how often two judges disagree
verdict from a public programverdict from a bought graderverdict goes only to the parties
Formalization moves a written-rule domain toward the left edge. Nothing on this map moves a verdict's reach.
The labs' answer is to buy the scorer: pay a lawyer to write a rubric, have a model grade against it. The ceiling on that was measured in 2022. Gao, Schulman and Hilton optimized a policy against a learned stand-in for the true reward. The true reward rose, peaked, and fell while the proxy kept climbing. A bigger grader moves the peak out. Nothing removes it.
Against a bought scorer, the true score peaks while the proxy keeps climbing
Shape of the law, not data: true score follows d(α − β log d); the proxy keeps rising with d
what is actually true (gold)what the grader reports (proxy)
Everything past the peak is the policy learning the grader instead of the task.
It's showing up everywhere RL goes past math:
None of that makes bought scorers useless. They're the reason models write and advise at all. But the spend tells you how scarce the real thing is. Anthropic discussed over $1B a year on RL environments; the environments in highest demand are coding, to the point that defunct startups reportedly get bought for their private repos; Mercor passed a $2B run rate selling expert judgment. A billion dollars for scorers, mostly in the one domain the labs already own.
Why can't the scorer for most of the economy be written down? Polanyi: we know more than we can tell. The customs broker who knows which entries draw an inspection can't write you the rulebook. Nisbett and Wilson showed in 1977 that people can't even report why they judge the way they do. That's why RLHF works at all. You can grade what you can't tell.
Hayek named the limit that doesn't go away. In The Use of Knowledge in Society, the knowledge that matters is of the particular circumstances of time and place: scattered across everyone, changing daily, never showing up in a statistic. A rubric is a statistic. One expert's judgment, one day, sent to the center and already going stale. That's the drift dial, and a central buyer of scorers can't turn it.
The labs act like they know this. Two spends at once: buying scorers, above, and sending people to the edge. About $10B in a year went into forward-deployed engineering, Palantir's model of putting engineers inside the customer: OpenAI $4B into a standalone deployment company, Anthropic $1.5B into a JV, FDE postings on Indeed from 643 to 5,330. An FDE's value is knowledge inside one customer's context: their exceptions, the process everyone works around. (See this post by Nikhil Davar and Byrne Hobart.) Hayek's local knowledge with a salary. The center buys statistics and sends agents to where statistics fail, in the same year.
Hypercubic has the right foil: don't compress tacit knowledge, build the environment it's expressed in and let agents explore. Right for legacy code, because code runs. You can't replicate a payer or a court. They aren't executable anywhere. They're observable, and only by the parties.
The human version, which Chollet raised: learning to program doesn't make you generally smarter. Graduate training in psychology and medicine improved everyday reasoning; chemistry improved nothing measured. And Luria's peasants in 1931, asked what color the bears are in Novaya Zemlya, refused to answer: they'd never been there. The peasant won't score a premise he hasn't seen. Math is the limit case, where the text is the whole world and there's nothing else to check against.
Why do some domains get a cheap scorer and others don't? The simplest way I can put it:
Some facts are true no matter what anyone thinks. A tumor shrank or it didn't. Other facts are true only because an institution says so: this claim is payable, this shipment owes 7.5% duty, this contract was breached. (Searle called these brute facts and institutional facts.) The second kind come with a rulebook, and a rulebook is text. Text can be turned into a program. That's why tax, customs, contracts, and fund waterfalls can be compiled: a model reads the rules and computes the answer the rules give.
But the rules aren't the whole story. Rules get applied by someone: a customs officer, a claims examiner, a judge. They have backlogs, habits, discretion, quirks. None of that is in the rulebook. The only way to see it is to watch what they actually decide, case by case. So every institutional domain has two layers. The written rules, which you can compile. And the way the rules actually get enforced, which you can only observe.
This is what I mean when I say Lean-style formalization is useful but orthogonal. Autoformalizing a rulebook gets you the first layer and only the first layer. It's still progress. Gwern argued in 2018 that the best learners carry an explicit model of their environment (his “white box”), and a compiled rulebook is exactly that. Astra on ARC-AGI-3 is a model building one on its own: it turns each game into symbolic rules and writes its own parsers and planners. That works because a game answers every move instantly. A game has no second layer.
My own two environments showed the first layer is basically done. One was US import-duty classification, where the tariff schedule fully determines the answer. One was private-fund administration, where the fund documents do. A small model went 10/10 on the first. Three frontier models made one real error in about 38 runs on the second. If a document determines the answer, models already get it. What they can't get is anything the documents don't determine, because that only exists as verdicts.
So the recipe for any institutional domain is: compile the rules, collect the verdicts, and treat every gap between what the rules say should have happened and what actually happened as a piece of the enforcer's behavior you just learned. The compiled layer is a head start that better models erode. The verdicts are the part no model can generate.
| Channel | What it does | What it buys | Where it stops |
|---|---|---|---|
| Compile | Written rules become a program | Tax, customs, contracts, fund waterfalls: anything the text determines | At the edge of the text; erodes as models improve |
| Buy | Expert judgment becomes a rubric or a reward model | Tacit knowledge extracted as grades, at scale | Gao's peak; the rubric drifts off the world it describes |
| Deploy | An agent acts in the world and collects verdicts | Live, honest, unfrozen feedback | Only where the agent is a party; the world's clock |
| Be the party | A transaction's verdict is addressed to you | Reach and drift resolved at once | Requires presence, liability, and capital |
Four ways to move a verifier, and where each stops
Three of the four arrows run into the lab. The verdicts that matter most run past it.
Compile is where the labs are fastest and where advantage erodes fastest: the same text plus a better model is the same program. Buy works until the peak, and its real limit is Hayek's: the center can't keep up with the edge. Deploy is Silver and Sutton's era of experience, the only channel whose verdicts don't drift, but the world reports on its own clock and only to those in the deal. A lab gets verdicts on code (merged or not) and chat (thumbs up or not) because it's a party to those. Be the party has no technical component; the price is presence, liability, and capital.
This is the answer to Chollet, and he wrote half of it in 2017: intelligence is bounded by the friction of the environment it acts in. Amdahl's law for learning. The slowest step is the world's verdict, and compute can't shorten it. So progress outside math and code isn't set by G. It's set by the reach of the verifier. “Plateau” is wrong because three channels are moving at once, and GDPval, graded blind by professionals with 14 years' experience, is improving roughly linearly. But each channel has a ceiling, and the one without a ceiling isn't available to a lab.
I expect none of the three.