Agentic AI for Substance Use Education: Integrating Regulatory and Scientific Knowledge Sources

TL;DR

This paper presents an agentic AI web application that combines regulatory and scientific sources to deliver real-time, accurate, and context-sensitive substance use education, evaluated positively by experts.

Contribution

It introduces a novel agentic AI system integrating DEA records and scientific literature for scalable, transparent substance use education with expert validation.

Findings

System achieved high ratings (4.18-4.35) across accuracy, citation quality, coherence, and appropriateness.

Expert evaluation showed substantial inter-rater agreement (Cohen's kappa = 0.78).

The approach demonstrates promise for scalable health education using authoritative sources.

Abstract

The delivery of traditional substance education has remained problematic due to challenges in scalability, personalization, and the currency of information in a rapidly evolving substance use landscape. While artificial intelligence (AI) offers a promising frontier for enhancing educational delivery, its application in providing real-time, authoritative substance use education remains largely underexplored. We built an agentic-based AI web application that combined Drug Enforcement Administration records with peer-reviewed literature in real-time to provide transparent context-sensitive substance use education. The system uses retrieval-augmented generation with a carefully filtered corpus of 102 documents and dynamic PubMed queries. Document storage was semantically chunked and placed in a vector representation in order to be easily retrieved. We conducted an expert evaluation study in which a panel of five subject matter experts generated 30 domain-specific questions, and two independent raters assessed 90 system interactions (30 primary questions plus two contextual follow-ups each) using a five-point Likert scale across four criteria: factual accuracy, citation quality, contextual coherence, and regulatory appropriateness. Mean ratings ranged from 4.18 to 4.35 across the four criteria (overall category range: 4.05-4.52), with substantial inter-rater agreement (Cohen's kappa = 0.78). These findings suggest that agentic AI architectures integrating authoritative regulatory sources with real-time scientific literature represent a promising direction for scalable, accurate, and verifiable health education delivery, warranting further evaluation through longitudinal user studies.