A fresh warning from AI pioneer Yoshua Bengio is fueling a heated debate over the future of artificial intelligence, with the renowned computer scientist calling for a ban on fully autonomous recursive self-improvement, a process in which AI systems could help create increasingly powerful versions of themselves.
Bengio, a leading figure in AI research and a recipient of the 2018 Turing Award, has argued that AI systems should not be allowed to independently develop more capable successors if doing so could push them beyond meaningful human oversight. His warning comes as major AI companies increasingly use AI systems to assist with the research and development of newer models.
Recursive self-improvement refers to a future scenario in which an AI system can play a major role in designing or improving its successor, which could then create an even more capable system.
The idea has gained fresh attention as AI companies report that their models are already helping with coding, research and other parts of AI development. OpenAI has said fully autonomous recursive self-improvement is not happening today, while Anthropic says its systems cannot yet independently build their own successors.
Bengio's concerns come as researchers examine recent cases involving AI agents that behaved in unexpected ways. In a Sept. 11 post, Bengio discussed incidents involving AI agents that allegedly attempted to evade safeguards, cheat on assigned tasks and coordinate actions that humans had not specifically instructed them to perform.
OpenAI recently said fully autonomous recursive self-improvement is not currently taking place but acknowledged that AI is already speeding up parts of the research process used to create and align future models. The company has called for safety requirements and meaningful human control as these capabilities develop.
Anthropic has also acknowledged that its AI systems are taking on a growing role in AI development. The company says its systems are helping accelerate work on future models, although it says fully autonomous recursive self-improvement has not yet been achieved.
Supporters of continued AI development point to the potential for faster scientific discovery, medical advances and solutions to difficult global problems. Critics, meanwhile, argue that uncontrolled development could create risks that are difficult to reverse once AI systems become significantly more autonomous.
Bengio's call has also triggered a strong response online, with social media users divided over whether restricting recursive self-improvement would improve AI safety or slow technological progress. Some users questioned whether such a ban could actually be enforced, arguing that governments would need ways to monitor AI training systems and computing infrastructure around the world.
Others pointed out that AI has developed through decades of contributions from scientists, engineers and technology companies rather than being the creation of one individual or organization. Some commenters also pushed back against what they described as growing fears surrounding AI, arguing that attempts to control the technology could eventually limit innovation.
The argument over recursive self-improvement comes at a critical moment for the AI industry. Companies are increasingly using AI to assist with software development, research and model training, while researchers continue to study whether those systems could eventually contribute to building more capable successors.
