CLiVR: Conversational Learning System in Virtual Reality with AI-Powered Patients

TL;DR

CLiVR is an innovative virtual reality system that uses AI and speech technology to simulate doctor-patient interactions, providing a scalable and immersive training tool for medical education.

Contribution

This paper introduces CLiVR, a novel VR-based conversational learning system integrating LLMs, speech processing, and 3D avatars for realistic medical training simulations.

Findings

High user acceptance and perceived educational value

Effective simulation of doctor-patient interactions in VR

Positive feedback for system improvements

Abstract

Simulations constitute a fundamental component of medical and nursing education and traditionally employ standardized patients (SP) and high-fidelity manikins to develop clinical reasoning and communication skills. However, these methods require substantial resources, limiting accessibility and scalability. In this study, we introduce CLiVR, a Conversational Learning system in Virtual Reality that integrates large language models (LLMs), speech processing, and 3D avatars to simulate realistic doctor-patient interactions. Developed in Unity and deployed on the Meta Quest 3 platform, CLiVR enables trainees to engage in natural dialogue with virtual patients. Each simulation is dynamically generated from a syndrome-symptom database and enhanced with sentiment analysis to provide feedback on communication tone. Through an expert user study involving medical school faculty (n=13), we assessed usability, realism, and perceived educational impact. Results demonstrated strong user acceptance, high confidence in educational potential, and valuable feedback for improvement. CLiVR offers a scalable, immersive supplement to SP-based training.