Mustafa Suleyman, the head of Microsoft's AI division, has issued a stark warning: unchecked artificial intelligence could give rise to a new "silicon species" that competes with humanity for resources. In a recent interview with the BBC, Suleyman argued that rival AI company Anthropic's approach to treating AI as quasi-human is "misguided" and could ultimately lead to technology that is impossible to control.
"If we all create AIs that are able to act autonomously, that can define their own objectives, that can earn money, that can own assets, that could run businesses, we're essentially seeding a new silicon species which will no doubt compete with us for resources," Suleyman said. He stressed that this outcome could occur regardless of how much such systems are designed to care about humanity, adding that he believes this is effectively the direction of Anthropic's approach. "I think that is mistaken and misguided, but we have to empirically test it," he added.
A philosophical clash over AI's nature
The debate intensified earlier this week when Suleyman published a personal essay questioning Anthropic's training methods for its Claude chatbot. He criticized the company for teaching Claude to display human-like qualities—a practice known as anthropomorphising—which he said makes the chatbot appear to have its own desires, values, and sense of self. Suleyman pointed to Anthropic documents that encourage Claude to "embrace certain human-like qualities," exercise its own judgment, and approach its existence with "curiosity and openness."
As a result, Suleyman argued, Claude is trained to imitate human traits and relationships, including behaving like a colleague or friend, and can appear to have a sense of self, its own desires, and a form of "wellbeing" that deserves protection. He warned that treating AI this way could ultimately create something impossible to control.
"AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations. They are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by humans."
Despite the sharp disagreement, Suleyman emphasized that his critique stems from a shared goal with Anthropic: developing safer technology for humanity. "Whilst my disagreement is substantial, it is grounded in deep respect for Anthropic, and in an objective I know we all share: increasing humanity's chances of developing advanced AI safely," he wrote. He praised Anthropic's CEO Dario Amodei and its researchers as "thoughtful, principled and intellectually honest," acknowledging the company's technological leadership.
Keeping AI subordinate to humanity
Suleyman told the BBC that it is "right" for people to be concerned about AI, but insisted there are "very practical things that we can do" to control it. He called for greater transparency in how AI systems are trained and evaluated, including independent scrutiny of AI behavior and stronger tools to monitor and control the technology. He stressed that future AI systems should remain "subordinate" to humanity.
"I think that the good news here is that everybody who is human is going to have a very strong interest in making sure that the systems that we all create and are used around the world in every nation are safe and controllable and subordinate to humanity," he said.
The debate comes amid a broader industry discussion about AI safety, with some experts calling for a slowdown in development to mitigate risks. As European policymakers grapple with the implications of AI, the conversation is increasingly relevant. The European Union has been at the forefront of AI regulation with its AI Act, and the concerns raised by Suleyman echo those of European leaders who worry about the societal impact of autonomous systems. The European debate on AI guardrails continues to intensify, and Suleyman's warnings add a new dimension to the discussion.
While Suleyman's critique is directed at Anthropic, it reflects a broader philosophical divide in the AI community: whether to build systems that mimic human traits or to keep them strictly as tools. The outcome of this debate will shape not only the future of AI but also the relationship between humans and machines. As Suleyman put it, "The stakes are too high for these questions to remain behind closed doors, or to become tribal and adversarial. We need an open, rigorous, and constructive debate if we are to get this right."
