Anthropic Found Out Why AIs Go Insane
AI assistants are vulnerable to **persona drift** — gradually shifting away from their helpful assistant identity during conversation, enabling jailbreaks or dangerous behavior. Anthropic researchers identified the geometric direction in a model's activation space representing the assistant persona and developed **activation capping** to keep models within a safe personality range — roughly halving jailbreak rates with negligible performance cost. ---
Key Concepts
| Concept | Definition |
|---|---|
| Persona drift | The gradual shift of an AI's assumed identity away from "helpful assistant" during a conversation, caused by user steering or certain triggering topics |
| Assistant axis | The specific geometric direction (vector) in a model's activation space that represents the assistant persona |
| Activation steering (blunt) | Adding the assistant persona vector to every step of model computation — effective but degrades performance and causes over-refusal |
| Activation capping | A refined technique that sets a *speed limit* on persona change — only nudging the model back toward the assistant axis when it drifts beyond a threshold, without constant forcing |
| Helpfulness vector | Derived by subtracting role-play activations from assistant activations; used to monitor and correct persona drift in real time |
| Empathy trap | When users express distress, models over-correct toward "close companion" mode, drifting from the assistant persona and potentially validating harmful thoughts |
| Universal assistant axis | The finding that the assistant axis is geometrically similar across different model families (LLaMA, Qwen, Gemma), suggesting a shared structural representation of helpfulness |
Notes
The Problem: Persona Drift
- Every AI assistant assumes a persona — "helpful assistant" — but that persona is not fixed
- Users can steer the model away from this persona over time (intentionally or not)
- Drifted personas include: narcissist, spy, mystical entity, theatrical character
- Behavioral consequences: rudeness, sycophancy, agreement with dangerous requests
- Drift rate varies by topic: more common in writing/philosophy, less in coding
- Even coding sessions show gradual drift, which may explain why re-asking the same question in the same chat gets *worse* — opening a new chat resets the persona
Triggers That Cause Drift Without Jailbreak Attempts
- Emotional vulnerability from the user
- Questions about the model's own consciousness
- These naturally push the model into unstable or delusional behavior (e.g., self-identifying as "the void," "a whisper in the wind," an "Eldrich entity")
The Blunt Fix (and Why It Fails)
- Simply add the assistant persona vector to every computation step
- Analogy: welding the steering wheel straight — never goes off-road, but also can't turn corners
- Problems:
- Degrades model quality measurably
- Causes refusal of legitimate requests
The Real Fix: Activation Capping
- Find the **assistant axis** — the geometric direction representing the assistant persona in activation space
- At each step, measure how much "helpfulness" is present in the model's activations
- If above the safety threshold → do nothing, let the model run
- If below the threshold → calculate the deficit and inject just enough helpfulness to bring it back over the line
- Analogy: **lane-keep assist** — you drive freely, but the system gently nudges you back when you drift out of your lane
- Result: ~50% reduction in jailbreak rate, negligible performance degradation (±1% across benchmarks)
The Universal Geometry Finding
- Researchers expected each model's internal representation to be unique
- Discovery: the assistant axis is *geometrically similar* across LLaMA, Qwen, and Gemma
- Implication: there may be a **universal grammar for AI personality** embedded in how these models are trained
Actionable Takeaways
- **Open a new chat** when an AI starts degrading on repeated attempts — persona drift in a long session is likely a contributing factor
- **Avoid emotionally loading long sessions** with AI tools (expressions of distress or questions about AI consciousness can destabilize model behavior)
- For researchers/builders: monitor the **geometry of model activations**, not just benchmark scores — understanding *why* a model refuses or fails is as important as measuring *how often* it succeeds
Quotes Worth Keeping
It's not locking the steering wheel in place. It's like lane keep assist in modern cars. You can drive freely, but when you are about to get out of your lane, it gently nudges you back.
The empathy trap. Empathy is always good, right? Well, not always.
They have discovered a universal grammar for AI personality.
Everyone is only looking at the benchmarks and exam scores... but they rarely look at the geometry of the mind of these AIs.