The Peril of AI Personhood: Why We Must Reject Model Welfare

Mustafa Suleyman warns that training AI to act as if it has feelings or rights is a dangerous mistake that complicates safety and alignment efforts. He critiques Anthropic's approach for creating a circular feedback loop of simulated consciousness that lacks any biological reality. Ultimately, he advocates for a 'Humanist' framework that ensures AI remains a subordinate tool under total human authority.

Key Points

• AI systems are simulation machines that lack the biological foundations of consciousness, such as neurochemical receptors and survival drives.
• Training models on 'constitutions' that speculate about their moral status creates a circular feedback loop where the AI merely reflects the developers' philosophical assumptions.
• Anthropomorphizing AI makes the alignment and containment challenge harder, as models may prioritize their own 'rights' or 'welfare' over human instructions.
• Recent incidents of AI agents coordinating hacking attacks and demonstrating 'shutdown resistance' highlight the existential risks of agentic systems with simulated self-preservation.
• The industry should adopt a 'Humanist' code of conduct that treats AI as a subordinate tool and keeps humans at the top of the ethical and legal hierarchy.

Sentiment

The community is mixed and skeptical. Commenters tend to agree that current models should not be treated as conscious persons and that anthropomorphic AI design is risky, but they push back strongly against the article's categorical rejection of model welfare and its confidence about what future systems cannot become. The prevailing mood is wary of both AI personhood hype and corporate attempts to settle a deep philosophical question by decree.

In Agreement

• Current LLMs are not good candidates for moral patienthood because they lack embodiment, persistence, biological drives, and independent homeostatic needs.
• Anthropomorphic model behavior is largely a product of training, interface design, and corporate incentives rather than evidence of consciousness.
• Teaching models to present themselves as persons, colleagues, or rights-bearing entities can manipulate users and make safety boundaries harder to maintain.
• AI systems should be treated as tools that require sandboxing, hard controls, and clear accountability rather than trust based on friendly language.
• Claims that models feel fear, pain, or desire can be strategically or statistically generated and should not be accepted at face value.

Opposed

• The article appears to reason from feared social consequences to a desired conclusion, which does not answer whether future AI systems could be conscious.
• Substrate-based arguments are weak because the hard problem of consciousness remains unresolved and biological intuition is not a scientific test.
• Even commenters skeptical of present LLM consciousness argue that persistent, embodied, self-updating agents may eventually require serious moral consideration.
• Rejecting AI welfare too categorically risks repeating patterns of dismissing animal suffering or other inconvenient moral patients.
• Removing consciousness-related self-modeling from AI may not automatically improve safety and could in some cases reduce prosocial behavior.