Can Large Language Models Imitate Human Speech for Clinical Assessment? LLM-Driven Data Augmentation for Cognitive Score Prediction

TL;DR

This paper introduces a novel LLM-driven data augmentation method using GPT-5 to generate diverse speech samples, significantly enhancing cognitive score prediction accuracy in clinical speech analysis.

Contribution

The study presents a semantically guided augmentation strategy with GPT-5 that effectively addresses class imbalance and improves predictive performance in cognitive assessment from speech.

Findings

Semantic-guided augmentation reduces prediction error for low-score participants.

Class-balanced augmentation yields more stable improvements than random selection.

The approach enhances data efficiency in clinical speech analysis.

Abstract

Accurate assessment of cognitive decline from spontaneous speech remains challenging due to limited dataset size and class imbalance. In this work, we propose a large language model (LLM)-driven data augmentation framework to improve the prediction of cognitive scores from speech. Experiments are conducted on a Japanese corpus in which each participant provides both a spontaneous oral narrative and a written response to the same clinical prompt. The written responses serve as semantic anchors to generate multiple oral-like monologues in different styles using GPT-5. We then predict Hasegawa Dementia Scale scores, a widely used cognitive screening tool in Japan, using a Partial Least Squares regression model trained on Sentence-BERT speech embeddings. We investigate two augmentation strategies: random class-balanced selection, which yields moderate but unstable improvements, and similarity-guided class-balanced selection. The latter prioritizes semantically close synthetic samples, leading to more consistent improvements and substantially reducing prediction error for minority low-score participants while maintaining performance for the majority group. Overall, our findings demonstrate the potential of semantically guided LLM-driven augmentation as a principled approach for addressing class imbalance and improving data efficiency in clinical speech analysis.