When artificial intelligence lab Anthropic debuts on the stock market later this year, it could become one of the biggest initial public offering historically. That’s because the company’s Claude chatbot is so popular among programmers that they’re willing to pay top dollar for access. Since the launch of its software engineering agency Claude Code in February 2025, it has become an integral part of many human developers around the world. That includes Anthropic’s own code: The company says more than four-fifths of the code it released in May was written by Claude. Before the launch of Claude Code, that percentage was in the “low single digits.”
The system has improved both in output quality and quantity. An influential benchmark by think tank METR showed that by early 2025, Anthropic’s models could complete tasks that would take a human engineer less than an hour. The company’s latest system can complete tasks that take more than a working day.
So when the company is ahead of the competition and outperforms its rivals, as it did on June 5, by calling on the world to “have the option to slow down or temporarily halt cutting-edge AI development,” it might be easy to raise cynical skepticism. What market leader wouldn’t want its competitors to stop catching up?
However, Anthropic leaders have worried for years runaway artificial intelligence Wanton destruction appears sincere. The latest generation of artificial intelligence models are made up of incredibly capable coders, engineers, and (soon) scientists, and many fear they may be among the last models ever built by humans. Anthropic co-founder Jack Clark believes that by the end of 2028, there is a 60% chance that artificial intelligence systems will be able to create their own successors without human involvement.
That moment marked the beginning of a process called “recursive self-improvement” (RSI), which is a closed loop. The first version of the model begets a second version, which is faster and more powerful; the second version begets a third version, and the third version even more so. The cycle continues, and the improvement grows with each iteration. Build an AI system that does this and your human engineers will never have to build another system. “What may seem like a tall tale to many people may be a real trend,” Mr Clark said.
No one knows exactly what the consequences of RSI will be. Because AI, unlike humans, can work tirelessly and continuously, some believe it will produce super-intelligent AI in the short term—a “quick takeoff.” (It’s also onomatopoeically called “going foom” because one might imagine the sound an intelligence explosion would make). Artificial intelligence doomsdayers worry that superintelligence will be beyond human control, and the beginning of RSI is the moment when humanity’s fate is handed over to machines. However, self-improving AI may face speed limits, at least initially.
Building RSI-capable models requires automating a range of specialized tasks currently performed by humans. Currently, data scientists work on the theory of artificial intelligence, while programmers put it into practice. Systems engineers lay the foundation for toy models to be produced at scale. Others look for new sources of training data, or try ways to generate new data. Coordination and safety teams check that the training process results in no intentional or other harm.
Not all of these teams are equally suited to AI’s help, and within each profession, some tasks are easier to automate than others. It won’t be long before human coders can do their jobs without writing a line of computer code themselves, but it may be some time before artificial intelligence can negotiate access to previously undigitized collections of scientific papers. It’s not always obvious how the “jagged border” will develop. Designing new algorithms seemed to be one of the safer jobs until Google DeepMind’s model AlphaEvolve began doing so in May 2025. It proposed changes to the way Google distributes workloads across its data centers, saving 0.7% of the company’s global computing power, and found better ways to perform matrix multiplications to speed up training of the company’s flagship large language model (LLM) Gemini by 1%.
Complete RSI requires that every task in the chain be automated. However, the acceleration of AI-driven research and development (R&D) may be felt even before then. A report released in January by the Center for Security and Emerging Technologies (CSET), a think tank at Georgetown University, said that “as the proportion of AI R&D performed by AI systems increases, productivity gains over purely human R&D” may increase by a factor of ten, then a hundred, and then a thousand. In this context, it warns that even if some aspects of AI R&D are initially difficult to automate, “the increased pace of progress means these bottlenecks will soon be overcome.”
repeated happiness
Today, no artificial intelligence model is capable of building its own successor. But large AI models can build smaller models on their own. With human help, they can also build other large AI models.
Earlier this year, Andrej Karpathy, an independent researcher then working for Anthropic, trained a chatbot with the same capabilities as GPT-2, a large language model OpenAI built in 2019. At the time, the model required 168 hours of training to build on 32 state-of-the-art chips; Dr. Karpathy achieved the same results in just three hours using a computer equipped with 8 GPUs (specialized chips used to build artificial intelligence). After several months of work, he reduced the training time of the model Nanochat to just over two hours.
In March, he handed over the job of speeding up the training process to an artificial intelligence agent called Autoresearch. After two days the training time was reduced to 1 hour and 48 minutes, and after five days it was reduced to 1 hour and 39 minutes. “I didn’t touch anything,” Dr. Capaldi said. The 18% improvement in human workload is staggering because Dr. Karpathy is an especially talented individual: he was a founding member of the OpenAI research team and served as head of AI at Tesla for five years.
The improvements themselves are lackluster. The AI agent chose better starting values for the training run, expanded the scope of the LLM’s “attention” window, and noticed that the model’s focus was drifting. None are particularly novel, Dr. Capaldi said. But he misses them. “They stack up and actually improve Nanochat,” he said.
As models become more powerful, this acceleration is inevitable. Much of the work involved in building terabyte-scale cutting-edge models isn’t as glamorous as the AI industry’s huge salaries and fancy offices might suggest. It involves connecting together the layers of an infrastructure stack purchased from a third party, debugging hardware and software settings, and tweaking “hyperparameters” (the initial settings for a training run) until the results look reliable. Today, AI systems can do most jobs with little supervision.
But even more detailed intellectual work is getting close to being automated, says Joe Spisak, a researcher at Reflection AI, a New York-based lab that is building cutting-edge models with open weights (meaning their parameters are publicly released). Give a cutting-edge system a rough idea of efficiency improvements, and it becomes increasingly capable of designing experiments, running tests on toy models to see what works, and developing a plan ready for large-scale implementation.
AI models can complete such tasks in about 30 minutes that would take humans hours. Increasingly, humans only play the role of research directors, directing AI experiments, with models coding, debugging, optimizing, and monitoring themselves. The productivity gains are alluring, but also worrisome. As humans’ role in the production process shrinks, they may lose control. The final result may be a model training model that achieves the goals set by the model, and its security can only be verified by the model.
Some people worry about disaster. Max Tegmark, a physicist and machine learning researcher at MIT who has devoted much of the past decade to the AI safety movement, likens it to a driver on the highway hitting the accelerator with his eyes closed. He told The Economist’s upcoming “Inside Tech” video program that as long as drivers refuse to open their eyes, the result is doom. Professor Tegmark offers various scenarios for how things could go wrong: Powerful AI systems could disempower humans by surpassing them in government and business decision-making; they could give rise to global totalitarianism by giving supreme power to those who built them in the first place; or they could simply stop caring about humans and gradually squeeze them out to make room for more data centers and power generation.
Three years ago, Professor Tegmark led the call for a moratorium on global artificial intelligence development. He believed that the creation of the then cutting-edge GPT-4 was tantamount to a blindfolded journey. This year’s CSET report warned that the system created by RSI “poses a significant risk. This requires immediate preparatory action.” It appears that Anthropic is now close to agreeing with this prescription.
Popular chips
Currently, there are physical limitations that limit how quickly models can improve themselves. The most important thing is computing power. Despite improvements in efficiency, newer models continue to use more computing power to train than previous-generation models, forcing data centers to keep moving forward.
Helen Toner, interim executive director of CSET and lead author of the recent report, said consumer adoption of AI may also slow the pace of AI-driven research and development. The limited capacity of AI data centers needs to be carefully allocated between serving paying customers, training future models, and conducting open research and development. The more demand in the first category, the less capacity there will be in the other two categories in the short term.
Then there is the issue of training data. Many recent advances in artificial intelligence have focused on areas where models can teach themselves how to succeed through “verifiable rewards.” A piece of software either works or it doesn’t; mathematical proofs are correct or incorrect. In this case, the synthetic data generated by the model is purely for the purpose of training other models, and its accuracy can be checked and added to the training data without risking the degradation that usually comes when training on the output of the AI itself. Making models better at creative writing or legal judgment is trickier. If the model needs to learn from the real world, this may also limit the scope for self-improvement.
“Closing the Loop” could be a step toward superintelligence and—depending on your temperament—utopia or doom. But that’s not the only step required for exponential growth in AI capabilities.
