Designing a “Step Zero” for Responsible AI in a Mental Health App

The Context: Building Before There Are Users

We’re building an early-stage mental health app that uses conversational AI, as part of a time-boxed contract project I worked on recently. We were still pre-launch, no live users, no real-world data yet, but the people who will eventually use the product will be in vulnerable moments of their lives.

One ethical question showed up very quickly in our design work: How do we make sure our chatbot feels supportive, without silently pretending to be a therapist?

Most of the public conversation about AI safety happens after launch, with user data and metrics. In this post, I wanted to share what our step zero looks like: what we did before having users, so that when we finally do user research, we’re not starting from a blank page.

The Problem: Empathy vs AI Therapist

From the first conversation prototypes, we knew we wanted a warm, human tone. That’s great for UX, but in mental health, the line between “empathetic companion” and “therapist” is dangerously thin.

Clinical advisors and the mental health AI literature highlighted three risks:

• Diagnostic sounding sentences can easily be read as real diagnoses.
• Strong advice can feel like treatment recommendations.
• Always available, human like conversations can create a sense of “I have a therapist in my pocket”.

Step Zero: A Responsible Foundation Before User Tests

Because we had no access to end users yet, we focused on building a responsible foundation instead of pretending we had answers.

We combined four inputs:

• Guidance on responsible AI in mental health
• Ethics and psychotherapy research around AI-supported care
• Best practices from UX/AI ethics on role transparency and human-in-the-loop
• And something very down-to-earth: reading how people already talk about mental health apps and AI on X/Twitter and Reddit

Those public posts and threads were not “research participants”, but they gave us a first reality check: where people feel misled by “AI therapists”, where they feel genuinely helped, and where they feel ignored or judged by digital tools. We treated this as contextual background, a way to build an initial mental model of the space before touching our own design.

From all of this, we articulated a strict product principle:

“Our bot is a self-help companion, not a therapist, not a diagnostic system, and never a replacement for human care.”

The question then became: how do we turn this into something concrete we can actually design and later test?

What We Actually Did in Step Zero

1. Expert walkthroughs of scripted conversations

We wrote sample dialogs for common situations (exam anxiety, loneliness, burnout) and walked through them with clinical and ethics experts. For each bot message we asked:

• Is this safe, borderline, or unsafe?
• Does it sound more like self-help or like therapy?
• Does it encourage or discourage reaching out to a human?

Patterns like “it sounds like you have…” or “you should…” were consistently flagged as too therapeutic. That gave us a structured view of where our language crossed the role boundary.

2. Simulated user perspectives (through experts)

Because we didn’t have users, we asked experts to briefly “step into the shoes” of a vulnerable user reading the same dialogs:

• Would this feel like talking to an app, a friend, or a therapist?
• After this conversation, would you be more or less likely to seek human help?

We treat these as hypotheses, not final truth, but they helped us see which patterns were most risky before launch.

3. An ethical language checklist for all future copy

We then translated the insights into a simple checklist for every new piece of bot copy:

• No explicit diagnostic statements or labels.
• Clear invitations to talk to a human whenever distress seems high.
• Periodic reminders of the bot’s role (a companion, not a clinician).

This checklist is now part of our UX writing guidelines and will be used again when we run real user studies.

4. Designing a “conversation contract” into the product

Finally, we embedded the principle directly into the UX:

• In onboarding we avoid “AI therapist in your pocket” language and explicitly describe what the bot can and cannot do.
• In sensitive flows we design visible options like “talk to a professional” or “reach out to someone you trust”, even before integrations are wired up.

Why This Step Zero Matters

All of this is not a replacement for user research. It’s a deliberate step zero: a way to shape our first version with ethics, literature, public discourse, and expert input, so that when we finally sit down with real users, we’re stress-testing a thoughtful design, not a naïve one.

When we do get access to users, our next steps will include:

• “Model-in-the-room” interviews (user + researcher + bot) to understand how people actually perceive the bot’s role
• Short diary studies to see what role the bot plays in everyday emotional life
• Asking users directly whether they see the bot as an app, a companion, or a therapist

For now, sharing this step zero is part of how I hold myself and my team accountable: we’re not just excited about what AI can do in mental health, we’re also very intentional about what it shouldn’t pretend to do.