The co-founder of one of the world’s most valuable AI companies has warned that the technology is nearing a point where it could develop without human input — and the industry urgently needs a “brake pedal”.
Jack Clark, who co-founded Anthropic, told BBC Newsnight that already his company’s popular chatbot Claude is operating on code of which 80% the system wrote itself. Getting to 100% is possible within two years, he said, and “would have huge implications”.
“Anthropic co-founder warns AI could develop without human input, calls for regulatory brake pedal.”
“You want the option to be able to take your foot off the gas and put your foot on the brake,” Clark said. “Right now, it’s like the AI industry has a gas pedal, but it doesn’t have a brake pedal.”
He stressed that people, through government policy, need to keep control of AI systems, which will only get more powerful and have broader impacts on society. “The world needs to do some thinking and we need to eventually develop some new regulations that allow us to be confident in these systems,” he said.
Drawing a parallel between AI and the oil boom of the turn of the last century, Clark noted that “society’s response was to come up with a sensible policy and regulatory framework that gave people confidence in oil and the benefits that oil could provide to the world, and meant that you didn’t have to worry about the personalities of the people leading the companies. That’s clearly where we end up here.”
Yet, days before Clark’s warning, Anthropic welcomed an executive order on AI from US President Donald Trump that was relatively hands-off in its directives toward the companies. It did not require AI firms to submit to safety testing by the government — something that remains a voluntary effort. Major AI companies pursuing advances in the technology, including Anthropic, OpenAI and Google, have also not said they will pause their own research.
Anthropic itself is preparing to debut on the public stock market, poised to be one of the first public listings by a newer AI firm and one of the most valuable stock listings in history. Its valuation is estimated by private investors to be nearly $1tn (£745bn).
Clark insisted that Anthropic’s motivation for publicly discussing the growing capability of AI technology is not to further burnish its reputation with paying customers. He simply wants to “tell the world what we’re seeing inside these companies with this unusual technology”.
Since its founding by chief executive Dario Amodei, Clark and a handful of other executives, Anthropic has positioned itself as outspoken about potential risks stemming from AI. But with no mandatory brakes in place and research racing ahead, the gap between warning and action has never been wider.