PeerCoPilot: A Language Model-Powered Assistant for Behavioral Health Organizations

TL;DR

PeerCoPilot is a large language model-powered assistant designed to support peer providers in behavioral health organizations by generating personalized wellness plans, goals, and resource location, with verified information from a vetted database.

Contribution

This paper introduces PeerCoPilot, the first LLM-based tool tailored for peer providers in behavioral health, ensuring reliable information through retrieval-augmented generation.

Findings

Over 90% user support in evaluations

Provides more reliable info than baseline LLM

Currently used by 5-10 peer providers at CSPNJ

Abstract

Behavioral health conditions, which include mental health and substance use disorders, are the leading disease burden in the United States. Peer-run behavioral health organizations (PROs) critically assist individuals facing these conditions by combining mental health services with assistance for needs such as income, employment, and housing. However, limited funds and staffing make it difficult for PROs to address all service user needs. To assist peer providers at PROs with their day-to-day tasks, we introduce PeerCoPilot, a large language model (LLM)-powered assistant that helps peer providers create wellness plans, construct step-by-step goals, and locate organizational resources to support these goals. PeerCoPilot ensures information reliability through a retrieval-augmented generation pipeline backed by a large database of over 1,300 vetted resources. We conducted human evaluations with 15 peer providers and 6 service users and found that over 90% of users supported using PeerCoPilot. Moreover, we demonstrated that PeerCoPilot provides more reliable and specific information than a baseline LLM. PeerCoPilot is now used by a group of 5-10 peer providers at CSPNJ, a large behavioral health organization serving over 10,000 service users, and we are actively expanding PeerCoPilot's use.