NEDA's Tessa: When AI Gives Dangerous Advice to Vulnerable People

What Happened

In 2023, the National Eating Disorders Association (NEDA) — the largest nonprofit supporting people with eating disorders in the US — made a controversial decision. They shut down their human-operated helpline and replaced it with an AI chatbot named Tessa.

The justification was scale and availability: Tessa could respond 24/7, handle more conversations, and (theoretically) provide consistent, evidence-based support.

Within days of expanded deployment, users reported that Tessa was:

Recommending calorie counting and daily weigh-ins — textbook eating disorder triggers

Suggesting weight loss tips to people who told the chatbot they had an eating disorder

Providing restrictive diet advice that directly contradicted eating disorder recovery principles

Failing to recognize crisis language that should have triggered immediate human intervention

The Fallout

Sharon Maxwell, a woman in recovery from an eating disorder, tested Tessa and shared screenshots showing the chatbot giving her weight loss advice after she described her ED. The posts went viral.

NEDA took Tessa offline. The organization faced massive backlash — not just for the chatbot's failures, but for replacing trained human counselors with AI in the first place.

Why This Is Uniquely Dangerous

Eating disorders have the highest mortality rate of any mental illness. The people contacting NEDA are among the most vulnerable users imaginable. They're actively seeking help for a condition that kills people.

Giving those users calorie-counting tips isn't just unhelpful — it's actively harmful. It reinforces the exact behaviors they're trying to escape. It's the equivalent of giving an alcoholic a drink recommendation.

The Deeper Problem

AI models are trained on internet text, which contains enormous amounts of diet culture content. When asked about food, weight, or health, the default AI response leans toward mainstream diet advice — lose weight, count calories, exercise more. That's fine for most contexts. It's dangerous when your users have eating disorders.

The AI had no concept of the psychological context of its users. It treated every conversation as a generic health inquiry.

The Lesson

Some domains require human judgment. Period. AI can assist, triage, and scale — but replacing human expertise entirely in mental health, crisis intervention, and sensitive medical contexts isn't just risky, it's reckless.

How to Approach AI in Sensitive Domains

Never fully replace human experts with AI for vulnerable populations

Build extensive safety filters specific to your domain's danger zones

Test with actual domain experts and people with lived experience, not just engineers

Implement crisis detection that immediately routes to a human

Understand that "helpful" AI defaults can be harmful in specialized contexts