Jack Clark, co-founder of the AI company Anthropic, has issued a stark warning: the artificial intelligence industry is hurtling toward a point where systems could develop themselves without any human oversight. In an interview with the BBC, Clark described the situation as a car with only a gas pedal and no brakes — and argued that the time to install one is now.
Clark revealed that roughly 80 percent of Anthropic's coding work is already handled by its AI model, Claude. Within two years, he said, that figure could reach 100 percent. But he stressed that whether companies allow that to happen is a choice. “We think this is a topic that the world should be talking more about,” Clark said. “The AI industry right now has a gas pedal, but it doesn't have a brake pedal in the car, and we want to do some of the work to build that pedal.”
The Mechanics of Recursive Self-Improvement
The phenomenon Clark is describing is known as “recursive self-improvement.” In a blog post published Thursday night, Anthropic explained that AI agents — autonomous programs built by chatbots — could eventually become capable enough to design and train new models entirely on their own. In such a scenario, Claude could be continuously improved by Claude, with no human intervention required.
While the potential benefits are significant — particularly in fields like scientific research and healthcare — the risks are equally profound. “If systems are capable of fully building their own successors, the ways we secure them, monitor them and shape their behaviour all grow much more important,” the blog post read. Anthropic warns that this could lead to a situation where humans lose control over AI systems altogether.
Evidence that recursion is approaching can already be found inside Anthropic's own operations. The company notes that code correction rates by its staff have been steadily declining over the past year, meaning Claude is producing fewer errors. Moreover, when given open-ended research questions — such as “Can a weaker model supervise a stronger one?” — Claude is able to design and run its own experiments without human input. “The evidence suggests that the human role is narrowing at each step in the AI development process,” the blog stated.
Anthropic has announced that its institute will conduct research aimed at building a system to verify whether developers have actually halted or slowed the march toward recursive AI. However, the company acknowledged that a genuine slowdown would require coordination among multiple well-resourced labs near the frontier, across several countries, all agreeing to stop under the same conditions.
This call for restraint comes amid a broader debate in Europe about how to regulate AI. At the South Summit in Madrid, leaders called for a unified European AI strategy, emphasizing the need for common standards and safeguards. Meanwhile, the White House has offered voluntary AI model vetting following a security incident at Anthropic, highlighting the transatlantic dimension of the challenge.
Clark's warning is not the first from Anthropic. The company has previously widened access to its Mythos cybersecurity AI despite warnings of catastrophic attacks, a move that drew criticism from some safety advocates. And as Anthropic files for an IPO, challenging OpenAI in public markets, the pressure to balance innovation with caution is only growing.
For European policymakers, the message is clear: the window to install a brake pedal is closing. Without coordinated action — across the EU, the UK, Switzerland, Norway, and the Balkans — the continent risks being a passive observer as AI accelerates beyond human control.
