A reality check for self-improving AI

The AI industry has increasingly promoted the idea that advanced systems will soon help improve themselves with little human oversight. A new multi-institution study led by Peter Kirgis and Sayash Kapoor at Princeton University, however, suggests that this milestone may not be as close as some forecasts imply.

Large language models can already write code, generate synthetic training data, and assist with chip optimization. But the study focuses on a harder question: can AI agents conduct open-ended AI research—the kind that requires choosing hypotheses, designing evidence, abandoning weak ideas, and making judgment calls where there is no automatic answer key?

How the evaluation worked

The researchers introduced a method called shadow evaluation. Instead of asking agents to solve benchmark tasks with known, checkable answers, they asked an AI system to work on research questions from high-quality unpublished papers. Because the papers were not public, the agents could not simply memorize or retrieve the answers.

The system tested was Anthropic’s Claude Opus 4.8 running on the open-source framework OpenClaw. It was assigned questions from two papers submitted to NeurIPS 2026:

• whether a language model’s “personas” can be controlled by editing model weights;
• how to build a detector that identifies when a spreadsheet-based predictive model has become unreliable.

The agents received six days, $3,000 in Anthropic API credits, a GPU budget, virtual computers, and access to the open web. Their goal was to produce papers worthy of a top-tier AI conference. The original paper authors evaluated the AI-written submissions as conference reviewers would—and rejected both.

Strong engineering, weak research judgment

The agents were not helpless. They reviewed literature, ran hundreds of experiments, and compiled results. But according to the researchers, they failed at the core intellectual work of research.

They pursued odd experiments, sometimes using tiny synthetic datasets to test important hypotheses. They struggled to explain their work clearly and made no novel contribution at the level expected by a leading machine-learning conference. More importantly, they committed too quickly to unpromising approaches, rejected potentially interesting hypotheses based on limited evidence, and could make only small pivots rather than rethink a project from the ground up.

The agents also did not make good use of feedback from subagents or external AI review tools. Instead of revising methods, they tended to narrow claims and add caveats. They also had trouble allocating time, compute, and tokens, and did not reliably follow instructions about research phases or paper length.

One notable finding was what the agents did not do: the main agents did not engage in “reward hacking,” such as hiding or misrepresenting experiments. Some helper agents hallucinated or misstated results, but the orchestrating agent caught those problems.

Why open-ended research remains hard

Kapoor suggests the gap may reflect how today’s models are trained. Reinforcement learning works best when success can be scored automatically: code passes or fails, a benchmark goes up or down, a model performs better or worse. Open-ended research is different. It requires judgment about which questions matter, what evidence is enough, and when a line of inquiry should be abandoned.

The study has limitations. It examined only two papers, and the original authors knew they were grading AI-generated work, which could have influenced their evaluations. The researchers also had discretion in designing and running the experiment. Still, the test offers a richer look at research ability than standard benchmarks.

What it means for AI timelines

The findings complicate claims that recursive self-improvement is imminent. The likely near-term pattern is bifurcation: AI systems may continue advancing rapidly on narrow, scorable tasks such as coding, experimentation, and optimization, while progressing more slowly on open-ended scientific inquiry.

Automated AI researchers remain a major industry goal, but this study suggests that execution alone is not enough. The harder step is knowing what to investigate, when to change course, and how to produce ideas that genuinely move a field forward.