"Human-led, AI-assisted" is easy to say and hard to hold. The temptation, once a model performs well on a benchmark, is to let it do more — to let it triage, to let it decide, to let it close. But in mental health, the cost of a confident wrong answer is measured in people, not accuracy scores.
What AI does well
AI is genuinely useful for the parts of care that are repetitive, language-heavy, and low-stakes:
- Reflection prompts that help a young person name what they feel.
- Documentation that drafts a case note a clinician then reviews and signs.
- Routing that surfaces a case to the right human based on what was said.
In each of these, the human remains the author of the decision. The AI prepares, suggests, and organizes — it does not conclude.
Where the line sits
The line is not about capability. It is about accountability. A model can produce a plausible risk assessment. It cannot take responsibility for one. So the boundary is simple:
- AI may support reflection, documentation, and routing.
- AI may not diagnose, prescribe, or close a case.
- Every escalation to human review is logged, auditable, and never optional.
Why the line matters
When the line blurs, two failures follow. First, the human stops checking — because the machine "usually" gets it right. Second, when it doesn't, nobody knows who was responsible. Both failures are quiet until someone is hurt.
Safety before scale is not a slogan. It is a design constraint: the system must be unable to close a case on its own, even if it wanted to. That constraint is the whole point.
