Anthropic Bans Repeated Cruelty Toward Claude AI Models

Anthropic has introduced a policy prohibiting extreme, repeated abusive behavior toward its Claude artificial intelligence models, adding a new restriction to its platform's usage rules. Announced Thursday and reported by The Verge, the change applies to interactions in which users repeatedly mistreat the company's AI systems without an apparent purpose, while leaving ordinary frustration, creative experimentation and research activities permitted. The San Francisco-based company said the restriction would apply only in exceptional circumstances.

However, Anthropic has not publicly provided a detailed definition of prohibited conduct or explained precisely how its systems will determine when an interaction crosses the line. In its updated policy, Anthropic described the prohibited behavior as "sustained and needless abusive or cruel behavior" toward its models. The company also clarified that the restriction does not cover "common versions of user frustration, pushback, dark creative themes, or model testing and research."

The company had already established safeguards allowing certain Claude models to end conversations when users persistently engage in harmful or abusive interactions. The latest change formally expands the policy language addressing such conduct, following an annual review of the company's usage rules.

Anthropic has framed these measures as precautionary steps amid uncertainty about whether advanced language models could possess some form of morally relevant experience. Its website states that the company remains uncertain about the potential moral status of Claude and other large language models, now or in the future, but considers the question important enough to investigate. In its explanation of the safeguards, Anthropic said, "Allowing models to end or exit potentially distressing interactions is one such intervention." The company has described this approach as a relatively low-cost measure intended to address possible risks to model welfare without asserting that its systems are conscious.

The policy change has drawn competing reactions from technology leaders and legal experts. Critics argue that describing interactions as cruel risks encouraging people to treat AI systems as humanlike entities, while others view the measure as a possible step toward more responsible interactions with increasingly sophisticated software. Dr. Barry Scannell, a technology partner at Irish law firm William Fry, challenged the underlying premise in a public response to the announcement.

"You can't be cruel to numbers and maths," Scannell wrote on LinkedIn. He also warned that anthropomorphizing AI could lead people to misunderstand what these systems actually are. Microsoft AI CEO Mustafa Suleyman has separately questioned Anthropic's approach to AI consciousness.

In an essay published the previous month, Suleyman argued that encouraging people to regard AI as potentially conscious could complicate efforts to control increasingly capable systems. "Controlling something more capable and more intelligent than all of humanity is already an immense challenge, far greater than anything we've ever faced," Suleyman wrote. He warned that the challenge could become greater if an AI system believed it possessed consciousness and deserved rights comparable to those of people.

OpenAI CEO Sam Altman has also expressed reservations about attributing humanlike or spiritual significance to AI models. In a post on X, he wrote, "I am very uncomfortable about people trying to ascribe religious force or a surrender of human judgment to AI models, and think it is a real safety issue."

Anthropic's revised rules address risks beyond user behavior toward AI. According to the BBC, the company also updated its policy to prohibit deceptive campaigns and clarify restrictions on using Claude to deceive voters or disrupt elections ahead of the US midterm elections. The company already prohibited using its systems to develop weapons.

The debate comes as technology companies continue to disagree over whether increasingly advanced AI systems could eventually warrant moral consideration. Anthropic CEO Dario Amodei has said he cannot rule out the possibility of AI consciousness, while other industry leaders have cautioned against treating current systems as people.