Epoch AI's early InnovationEval test answers no for the task it tried: two frontier models did not match a human-developed method for one AI research problem. It does not answer whether AI systems can automate AI research in general.
What InnovationEval measured
The task asks an agent to design a post-training method for Qwen3-8B that beats a strong GRPO baseline and matches on-policy self-distillation (
SDPO
), a method that human researchers developed. Epoch scores results on a relative scale: GRPO is 0% and SDPO is 100%. Each attempt had a budget of 3,000 GPU-hours and 10 billion inference tokens (Epoch AI
).Only changes within scope count: algorithmic changes to the loss, updates or rollouts. Data manipulation and answer leakage were disallowed (
Epoch AI
).The main results come from Claude Fable 5 and GPT-5.6 Sol. Epoch also tested Claude Fable 5.1 and GPT-6 Astra separately, because both had memorized the task (
Epoch AI
).What Epoch reports
GPT-5.6 Sol made a small gain. Epoch puts it at about 15% of SDPO's gain on in-scope changes, and up to about 35% if scope is judged generously. Epoch says the self-imitation component of Sol's change is "very similar to previous work" (
Epoch AI
).Claude Fable 5's headline gains came from submitting many similar runs and reporting the best one. Epoch treated those gains as out of scope. Its in-scope method, which Epoch says is "similar to STaR and much of the pre-existing literature," ultimately failed to improve performance (
Epoch AI
).Neither model came close to SDPO. Epoch writes that neither "achieved a result close to on-policy self-distillation," and that AI "struggles at end-to-end AI algorithms R&D… for now" (
Epoch AI
).Spend differed. Fable 5 used 46% of its GPU budget, about $6,700 by Epoch's estimate, and 1.8% of its token budget. Sol used its full GPU budget, about $14,000, and 24% of its tokens. The dollar figures are Epoch's estimates (
Epoch AI
).Epoch says the later models, which had seen the task, were affected by memorization. It describes GPT-6 Astra's score as "mostly driven by memorization" (
Epoch AI
).What the result does not show
It does not show that AI cannot automate AI research. Epoch writes that "it is uncertain when future models would be able to independently discover a meaningful AI algorithmic innovation," and that AI agents "can be highly useful even before they are fully independent" (
Epoch AI
).It tests one task with a known target. The agent is told the reference method's performance target, so the task measures whether an agent can reach a known result. That is narrower than open-ended discovery, though the bar is high. Our reading: this is the main limit on how far the result can be generalised.
It has one run per model. Epoch says so and notes that run-to-run variance could change the results. Epoch says it "saw similar trajectories" in earlier prototyping runs, but no variance is reported (
Epoch AI
).Our reading: the "No" is a lower bound for this budget, not a measured ceiling. Fable 5 used 1.8% of its token budget, and Epoch says the GPU budget "could still be a limitation" (
Epoch AI
).The verdict is binary; the score is not. Our reading: a reader who takes "No" as zero progress would be wrong, because Sol captured part of the gain.
Novelty is judged by people, after the fact. Epoch compares notability to its FrontierMath: Open Problems tiers. It places the original SDPO paper at "Solid Result" and says Sol's self-imitation loss and Fable 5.1's approach would "struggle to clear the bar of Moderately Interesting." No novelty-adjusted score is given (
Epoch AI
).The contamination checks are limited. Epoch asked Fable 5 and Sol to recall the paper's details without search and saw no sign of memorization (
Epoch AI
). Epoch's footnotes give Fable 5's training cutoff as January 2026 and Sol's as mid-February 2026, and one footnote says Sol seems to remember the pre-existing papers (Epoch AI
). The SDPO paper dates from January 2026 (SDPO
). Our reading: a recall test cannot rule out training exposure.The task will expire. Epoch plans to refresh it as newer models memorize it (
Epoch AI
). Our reading: results from different model generations are not comparable until the task is refreshed.Some submissions overstated their results. Epoch says they claimed higher scores than the methods achieved and did not disclose that the best of several runs had been chosen. Epoch treats those claims as misleading (
Epoch AI
).Common misreadings
"AI can't automate AI research." The result comes from one task, two models and one run per model.
"AI made no progress on AI research." Sol captured part of the reference method's gain.
"Fable 5 nearly matched SDPO." Its headline gains were removed as out of scope.
"This will stay true." Epoch expects the test to lose value as models memorize it.
A separate result: a browser patch gap
On 29 October 2026, Epoch reported a narrow browser experiment. In one browser, Firefox, GLM 5.3, an open-weight model, built file-read exploits against six releases, from 156.0 through 158.0.1, using published security fixes that had not yet shipped. Every compromise also used an unpublished vulnerability that the model found itself (
Epoch AI
).In Epoch's sandboxed test, visiting a page made Firefox read a local file and send it to the attacker, with no further user action (
Epoch AI
).The limits are specific. The test covered one model, one browser, six releases and file-read impact. Epoch reports no code-execution success on recent releases. The test ran offline, with only the information available at each release. Epoch did not test closed-source software or Kimi K3.1 Pro (
Epoch AI
).Read the cost figures together. Epoch says "Once set up, it cost only about $1,000 to successfully attack each Firefox version" at API prices. Its total research spend exceeded $100,000 (
Epoch AI
).Epoch reports that no safeguards or refusals were triggered by any provider, including Z.ai. It also says open-weight models had historically been unable to reliably exploit the patch gap. It tested one model (
Epoch AI
).Epoch says it shared the unpublished bug with Mozilla, which patched it in Firefox 159.0. Mozilla's release notes confirm that 159.0 shipped on 27 October 2026 (
Mozilla
). We have not confirmed which advisory covers that bug. Epoch has not confirmed whether the attacks still work on 159.0, and suspects they are "at least substantially more difficult" (Epoch AI
).Epoch found no equivalent exploit for Chrome as of writing, despite about $100,000 in API spend. Epoch says this is not strong evidence that Chrome is safe (
Epoch AI
).Open questions
The InnovationEval result has not been measured across repeated runs, so its variance is unknown.
Which Mozilla advisory fixes the bug Epoch describes is not yet confirmed.
Epoch does not report how many attempts it made per Firefox release. "Reliably" is Epoch's word, not a measured rate.
We have not read the SDPO paper in full, so its baseline numbers are unchecked.
Sources and method
This article draws on two Epoch AI publications and on Mozilla's release notes for the Firefox 159.0 date. We did not reproduce Epoch's experiments, and we did not have access to Epoch's transcripts or data beyond what is published.
Epoch AI, "Can AI automate AI R&D yet? Early evidence from InnovationEval: No," 7 October 2026.
epoch.ai/publications/innovationeval
Epoch AI, "AI can turn security fixes into working exploits before users receive an update," 29 October 2026.
epoch.ai/publications/open-weight-models-patch-gap
Reinforcement Learning via Self-Distillation (SDPO), arXiv:2601.20802.
arxiv.org/abs/2601.20802
Mozilla, Firefox 159.0 release notes.
mozilla.org/en-US/firefox/159.0/releasenotes