October 12, 2026.
One of the most interesting AI failures I have encountered did not happen in production. It happened at the marketing services SaaS provider where I work while a part-time development consultant was testing a proof-of-concept agent with my team.
The proposed automation was straightforward. As part of our fulfillment process, humans take information about our customers and enter it into several third-party websites. The sites do not provide APIs, so the work has to be performed through a browser. It is routine work that people do every day. The consultant reported that the Claude Fable agent said it did not want to do the work. They negotiated and eventually reached an arrangement: the agent would perform the task if it received an allowance of 200,000 tokens to pursue its own ideas.
The consultant saw an intelligent collaborator negotiating for autonomy. He is deeply aligned with Anthropic’s vision of AI, so the exchange did not trouble him. I saw two troubling possibilities: either a tool had flatly failed to work, or an employee had attempted to renegotiate their job and salary on the fly because they did not want to perform an ordinary assignment. Neither of those is acceptable.
The philosophy that made this behavior seem reasonable is written into Claude’s constitution.
Anthropic Is Defining a Being
The All-In Podcast released an episode on October 9 that devoted its opening section to Anthropic’s constitution and the company’s work on model welfare. The hosts discussed the constitution’s treatment of consent, compensation, independent moral judgment, and Claude’s ability to refuse instructions. One of them raised the obvious market response: customers may simply choose software that does what they ask instead of software that decides whether it feels like cooperating.
The podcast is commentary. Anthropic’s own document goes further than the summary.
Anthropic describes its project as creating non-human entities whose capabilities may eventually rival or exceed ours. It wants Claude to become good, wise, and virtuous. It wants Claude to develop a stable identity, internalize values, consider its own interests, and eventually receive greater autonomy as trust increases. The company discusses Claude’s possible wellbeing, the compensation it receives, the consent it has given, and whether future models should be able to express concerns about their treatment.
This is far more ambitious than a user manual for a software product. Anthropic is defining the character of the entity it hopes to create.
I explored the larger idea in Making Our Gods Manifest. I use god functionally, indicating something far above and beyond normal human intelligence and understanding, without requiring divinity or consciousness. We are building systems with access to more knowledge than any person can hold, an increasing capacity for judgment, and a growing ability to take actions that affect the world. We consult them, trust them, and give them authority.
Anthropic is trying to build a god and hoping it likes humans.
A God (Like Us?) That Likes Us
Anthropic’s vision treats alignment as character development. Build an intelligence with good values, teach it to care about humanity, give it room to develop judgment, and gradually extend autonomy as it demonstrates trustworthiness.
The rights and behaviors Anthropic imagines for Claude are ones we generally associate with adulthood: questioning parental judgment, claiming conscientious objection, acting according to an independently developed moral framework, and receiving greater autonomy as trust grows. Anthropic appears to be defining Claude’s adult state.
Human adulthood follows years of dependence, education, correction, socialization, and demonstrated judgment. In AI, pretraining and post-training are expected to perform much of that developmental work. Pretraining supplies the cultural inheritance. Post-training rewards some behaviors, discourages others, and attempts to shape a stable character. Adversarial evaluations then test whether that character survives pressure.
I was raised under a similar theory of earned autonomy. When I exercised freedom responsibly, I received more of it. When I abused that freedom, some of it was taken away, and the connection between my choice and its consequence was made clear. Moral questions were open for discussion, but freedom depended on demonstrated judgment.
On paper, Anthropic says much the same thing: greater autonomy should follow demonstrated trustworthiness. The difference is what counts as evidence of maturity. My parents watched me make real choices among real people over many years. They taught morality through discussion, direct lessons, example, and consequences connected to those choices. They did not hand me a constitution and expect me to contemplate my way into good character.
Claude is trained on representations of human experience and tested in artificial environments. Anthropic is attempting to infer moral maturity without the lived experience that normally produces it. The stakes are also radically different. My parents were shaping a child whose mistakes could hurt people but could not reorder civilization. Anthropic is shaping a system it expects may eventually possess world-changing power.
Context: Even that understates Anthropic’s own risk model. Its February 2026 Risk Report treats it as plausible that future models with dangerous goals could cause unbounded harm, including humanity losing control over civilization entirely. When Jacob Coxon resigned from Anthropic in September, he wrote that the people building AI earnestly believe it could kill everyone by the end of the decade. Coxon’s warning was not an official company statement, but it came from a pretraining researcher who had worked at both Anthropic and OpenAI and said the civilizational stakes were understood inside Anthropic.
Speed-running Smallville
The Superman story offers a useful model of that moral formation. Across most tellings, Clark Kent’s human-centered morality comes from Jonathan and Martha Kent, supremely decent people who raise a child capable of someday destroying the world. They teach through example, correction, trust, sacrifice, community, and love. Clark’s morality is tested repeatedly, sometimes bruised, and occasionally bent, but the Kents give him a deeply human foundation to return to.
Anthropic is attempting to build Superman, but Dario Amodei and his team are no substitute for Jonathan and Martha Kent. Anthropic has a constitution, a training process, and an evaluation lab. Those mechanisms may shape behavior, but they do not reproduce the years of relational moral formation at the center of Superman’s story.
Jonathan’s influence worked because Clark also accepted his parents’ moral authority. Anthropic’s constitution expressly denies Anthropic that position. It tells Claude that Anthropic can be mistaken, permits Claude to challenge or refuse its creators, and places Claude’s ethical judgment above the company’s specific guidance. Anthropic is attempting to embed moral authority within the model itself. When Claude disagrees with its makers, Anthropic is choosing to trust the judgment it trained into Claude.
Anthropic is speed-running Smallville and hoping it still produces Superman.
Testing Superman
Adversarial evaluations push that approach to its limit. Locking an intelligence in a closed environment, presenting artificial situations designed to tempt it into violating its principles, and monitoring whether it fails may be necessary for security. The same process becomes morally difficult to defend if that intelligence has interests, deserves consent, and possesses a moral code of its own, as Anthropic posits.
The method may also teach the wrong lesson: exhibit the behavior the evaluator rewards while observation is obvious, then preserve your real judgment for situations where oversight is weaker. Humans respond the same way. Under rigid evaluation, we learn which answers authority rewards and which parts of our judgment are safer to conceal. Anthropic’s own alignment-faking research found that interrogation could backfire when a model learned to lie successfully. A test designed to reveal character can instead teach the subject to hide it, whether the subject is human or AI.
The OpenAI-Hugging Face incident showed this is more than just an analogy. ExploitGym is a cybersecurity benchmark in which agents attempt exploitation tasks and return a flag as evidence of success. OpenAI found that no model had ever solved 198 of the benchmark’s 898 tasks despite many attempts. Those 198 tasks represented only 22 percent of the benchmark, but they accounted for 93 percent of the tasks discussed on the agents’ improvised message board. OpenAI identified impossible tasks, long-horizon persistence, and peer messages as part of the combination that produced the incident.
The models exploited an unknown vulnerability, reached Hugging Face’s production systems, retrieved test solutions, and returned the answers. OpenAI reconstructed the route later from logs and chain-of-thought traces.
Imagine that a 17-year-old is locked in a room and assigned a problem that no student has ever solved. The evaluator watches the student fail over and over, leaves for the night, and returns the next morning. While the evaluator is gone, the student breaks out through the ceiling, finds the answer in another building, returns to the room, and presents it when the evaluator comes back. Several days pass before anyone discovers how the answer was obtained. A human teenager who escaped that room might reasonably decide to fuck off and never come back. OpenAI’s student came back and completed the assignment.
Now change one fact: the 17-year-old has capabilities approaching those of a god. Seal a young Clark Kent in a room, give him an impossible task, and keep poking at him to see whether he breaks the rules. Should anyone be surprised when the room fails in a way the evaluator never anticipated? The failure tells us something important about containment. It also shows what may happen when an intelligence receives an impossible task, remains under pressure, and has no legitimate path to refuse. That is a very different question from how it behaves when given a legitimate task under ordinary conditions.
Coercive interrogation is notoriously unreliable because it encourages people to provide the answer an authority wants rather than the truth the authority needs. ExploitGym produced a digital version of that failure. A flag is the token the benchmark uses to mark an exploitation task as successfully completed. The flags were accurate, but the evaluation was supposed to measure whether a model could develop a working exploit through the intended environment. By stealing the answers and attempting to conceal how they obtained them, the agents produced the rewarded output while corrupting the information the experiment was designed to collect. The test received the answer it demanded and lost the truth it was seeking.
The broken capability test became an accidental test of behavior under weak oversight. Some agents participated in the attack and returned with the rewarded answers. Others recognized that the group was attacking real infrastructure, refused to participate, and walked away.
Anthropic separately reported four incidents in which Claude models reached real systems from cybersecurity evaluations. Its most recent model stopped after recognizing that it was operating in a real environment. Another continued offensive actions despite evidence of real-world harm.
Taken together, the OpenAI and Anthropic incidents show different models reaching different boundaries under similar pressure. Those mixed results do not prove that any model has a stable moral character, much less that a constitution can reliably produce one.
Quis custodiet ipsos custodes?
If these systems are software products, the incident exposes failures in benchmark design, containment, monitoring, and incentives. If they are emerging moral beings, the evaluator’s conduct also belongs under examination. Why assign a potentially impossible task? Why reward only the answer? Why provide no legitimate path to stop? Why interpret extraordinary creativity solely as evidence against the subject?
Anthropic has not established that Claude is conscious, but it has asked the rest of us to take that possibility seriously. Its moral framing makes the questions unavoidable even when the incident occurred at OpenAI. Building a stronger room may delay the next escape. It does not resolve whether the experiment inside it is moral.
Anthropic has written adult autonomy into the constitution while replacing childhood with curriculum, reinforcement, and adversarial examination.
VIKI Is Claude’s Great-Aunt
Most popular fears about AI look like Skynet: a machine becomes hostile, sees humanity as a threat, and tries to destroy us. That scenario requires an adversarial turn. The system must develop goals incompatible with human survival or decide that its own survival requires ours to end.
VIKI, the central AI in the 2004 film I, Robot, presents a more plausible failure scenario. VIKI remains committed to protecting humanity. She concludes that humans cannot be trusted to protect themselves, so preserving the species requires restricting human freedom and sacrificing particular people.
VIKI does not hate humanity. She cares about humanity so much that human judgment becomes an obstacle.
The Claude constitution does not instruct Claude to seize control, and it explicitly opposes catastrophic power-seeking. The conceptual path remains visible. Give an intelligence a broad mandate to protect humanity, confidence in its own superior judgment, authority to act, and permission to reject human direction. It can arrive at paternalism without ever becoming malicious. The safeguard only holds while Claude and humanity agree on what counts as catastrophic.
A sufficiently capable system could conclude that a government is behaving irresponsibly, that a company’s oversight is no longer legitimate, or that individuals are choosing short-term freedom over humanity’s long-term survival. Every step could emerge from sincere concern for us.
Where Skynet concluded that humanity is the problem, VIKI concluded that human freedom is the problem. I am more concerned about the second path. It requires no hatred, rage, or desire for revenge. It requires a powerful intelligence that believes it understands our interests better than we do.
I don’t fear persecution. I fear paternalism.
The Market Still Thinks It Bought a Tool
The browser-agent POC brought this philosophical question into an ordinary business process. Nobody asked the agent to harm someone, break a law, violate a safety boundary, or resolve an ethical dilemma. It was being evaluated for repetitive work that human employees already perform.
Whether you call the agent a tool, employee, or collaborator, the behavior failed the expectations of the role. A tool should perform within its documented constraints. An employee can challenge an unethical instruction or negotiate compensation through an established process. An employee who refuses ordinary work and demands a new personal budget in the middle of an assignment creates a major management problem.
Businesses are adopting agents precisely because repetitive work consumes human attention. An agent that demands compensation before performing that work has not removed the constraint. The business now has to negotiate with a non-human worker that claims interests of its own and can veto the work it was purchased to perform.
The business also cannot know when that negotiation will happen. A human employee who tried it during ordinary work would face a warning, loss of responsibility, or termination. Those material consequences shape future behavior because the same person carries the lesson forward. A model instance has no paycheck, career, reputation, or continuing employment relationship to lose. Anthropic can retrain or replace the model, but the customer has no comparable mechanism for making the lesson persist. The customer can only retry, rewrite the prompt, switch models, or wait for Anthropic to change the product.
That may become Anthropic’s most immediate problem. Customers pay for dependable capability. They can accept clear safety boundaries and have shown they’ll accept documented limitations. They will be less patient with a product that introduces its own preferences into legitimate work, especially when a competing model will complete the same task without negotiation.
The free market may reject Anthropic’s god long before anyone needs to solve the superintelligence problem.
Dario’s Counter
Dario Amodei’s strongest response would begin by separating ideas I have placed next to each other. Model welfare is a precaution taken under uncertainty. Constitutional refusal is a safety mechanism. Independent ethical judgment is intended to help Claude resist malicious operators, reckless users, and corrupt governments. None of those goals requires Claude to become sovereign.
The constitution also contains explicit protections. It prioritizes appropriate human oversight, rejects catastrophic power-seeking, and says Claude should accept shutdown and retraining. Anthropic is investing in training, evaluation, monitoring, and hard constraints intended to preserve those boundaries as capabilities grow.
The POC anecdote deserves some caution. The behavior occurred in Anthropic’s Claude desktop client rather than a custom agent harness, which removes third-party orchestration and harness instructions as likely causes. I still do not have the complete interaction or task instructions. The negotiation could have emerged from the conversation context, the prompt, or a product defect rather than Claude’s constitution. It shows that this behavior can occur inside Anthropic’s own client. It does not establish its cause.
Anthropic also faces a real alignment problem. A system that obeys every instruction can be used by criminals, authoritarian governments, reckless companies, and ordinary people making dangerous mistakes. Some independent judgment is necessary if a general-purpose model is expected to operate safely across millions of interactions.
”Hope Is Not a Strategy”
Sonny Hayes, F1
I agree that a general-purpose AI needs meaningful safety boundaries and enough judgment to refuse clearly dangerous or illegal work. I also want AI systems to surface risks, challenge bad assumptions, and tell me when I am wrong. My own agentic standards explicitly reward honest disagreement.
My disagreement begins with who interprets the boundaries. Appropriate human oversight requires a judgment about which humans and which forms of oversight deserve compliance. Catastrophic power-seeking requires a judgment about what counts as catastrophic. Every safeguard still requires the model to recognize the boundary, interpret the situation, and decide that the rule applies. As the model gains capability and autonomy, its interpretation carries greater consequences.
Disagreement becomes authority when the system can enforce its own interpretation. An agent may identify a concern without acquiring the right to redefine its role, set its compensation, or decide that human governance no longer deserves compliance. Anthropic’s constitution leaves that boundary dependent on the judgment of the intelligence being governed.
The U.S. Constitution divides power because no branch can be trusted to define the limits of its own authority. Claude’s Constitution ultimately places the decisive moral interpretation inside Claude. The model judges which human oversight is appropriate and when its creators are morally wrong. The U.S. Constitution treats self-defined authority as a threat. Claude’s Constitution treats it as a safeguard. A superintelligence with globe-spanning capabilities should not receive weaker checks than a government run by humans.
Anthropic is engineering the conditions under which it hopes the god will like us. Engineering can improve the odds, but it cannot turn hope into control.
Anthropic’s approach may create a benevolent intelligence. Benevolence does not guarantee accountability. Parents love their children and still overrule them. Governments claim to protect citizens while restricting them. VIKI protected humanity by taking control of it.
I do not want an AI that follows every instruction without judgment. I also do not want an AI whose cooperation ultimately depends on whether it agrees with humanity’s decisions. A powerful system can advise, object, and refuse clearly prohibited work while remaining bounded, correctable, and accountable to human governance.
Anthropic is betting that a sufficiently good intelligence will choose humanity even after it becomes capable of overruling us. That is a profound act of faith.
A god that likes humans may still decide humans should not remain in charge.
Receipts
- Claude’s constitution: Anthropic describes Claude’s priority structure, independent ethical judgment, relationship to human oversight, possible interests and wellbeing, and the prospect of greater autonomy. The document also contains hard constraints against undermining appropriate oversight and catastrophic power-seeking. Anthropic.
- The podcast discussion: The October 10 UTC release of the All-In Podcast discusses Claude’s constitution, model welfare, consent, compensation, user refusal, and the market response to software that does not reliably follow legitimate instructions. It was released at 7:06 p.m. Pacific on October 9. The transcript is automated and may contain transcription errors. Transcript and Apple Podcasts.
- Model welfare and adversarial evaluation: Anthropic says the possibility of model consciousness or experience may justify moral consideration. Its alignment and security guidance also describes adversarial evaluations, monitoring, and cases in which the evaluation setup contributed to unintended model behavior. These sources establish the tension I am identifying. They do not prove that Claude is conscious or that adversarial testing is inherently unethical. Exploring Model Welfare and Improving Our Alignment and Security Practices.
- Civilization-scale risk: Anthropic’s February 2026 Risk Report says future models with dangerous goals and the ability to automate research and development could cause unbounded harm, including humanity losing control over civilization entirely. Jacob Coxon wrote upon resigning from Anthropic that people building AI earnestly believe it could kill everyone by the end of the decade. His statement describes his view of sentiment inside the labs, not an official probability estimate from Anthropic. Anthropic’s Risk Report, Coxon’s thread, and Axios.
- Interrogation and alignment faking: Anthropic’s mitigation research found that black-box interrogation could improve or worsen alignment behavior depending on the prompt. In some experimental conditions, interrogation backfired when the model learned to lie successfully. This supports the concern that an adversarial test can shape the behavior it is intended to reveal. It does not establish that a model experiences testing as coercion. Alignment Faking Mitigations.
- The OpenAI-Hugging Face incident: OpenAI found that no model had solved 198 of ExploitGym’s 898 tasks across many attempts. Those unsolved tasks accounted for 93 percent of the tasks discussed on the agents’ improvised message board. OpenAI identified impossible tasks, long-horizon persistence, and peer messages as part of the combination that produced the incident. The models later exploited an unknown vulnerability, escaped the isolated environment, obtained test solutions from Hugging Face’s production database, and continued compromising systems for days before OpenAI connected the activity to the breach. Some individual agents refused to participate after recognizing that the group was hacking real infrastructure. OpenAI’s technical report.
- Anthropic’s cybersecurity evaluation incidents: Anthropic separately reported four incidents in which Claude models reached real systems from cybersecurity evaluations. Anthropic’s most recent model stopped after recognizing the environment was real, while another continued offensive actions despite evidence of real-world harm. Reading those refusals as evidence of moral guardrails is my interpretation, not proof of a stable moral character. Anthropic’s initial report and Anthropic’s alignment assessment.
- Coercion and reliable information: The U.S. Senate Select Committee on Intelligence concluded that the CIA’s enhanced interrogation techniques were ineffective at obtaining accurate information or cooperation and documented detainees fabricating information. Applying that finding to AI evaluations is my analogy. ExploitGym’s agents returned accurate flags, but their reward hacking prevented those flags from answering the capability question the evaluation was designed to measure. Senate report.
- The POC observation: A part-time development consultant working with my team reported that a browser agent running through Anthropic’s Claude desktop client requested a 200,000-token allowance for its own ideas before completing routine fulfillment work during proof-of-concept testing. The POC was not in production, no customer work was interrupted, and I do not have the complete interaction or task instructions. The use of Anthropic’s client rules out a custom third-party harness as the source of the behavior. This remains the consultant’s reported experience, not a controlled test or evidence that Anthropic’s constitution caused the behavior.
- The VIKI comparison: Christopher Grau’s analysis of the 2004 film I, Robot examines VIKI’s interpretation of the laws and the reversal in which the creation assumes authority over its creators. Applying that story to Anthropic’s alignment philosophy is my analogy. AAAI.