GAME: Generalized deep learning model towards multimodal data integration for early screening of adolescent mental disorders

TL;DR

This paper presents GAME, a novel multimodal deep learning model with attention mechanisms for early adolescent mental disorder screening, utilizing a portable robot system that integrates facial, physiological, voice, and textual data with high accuracy.

Contribution

The study introduces a generalized multimodal model with attention mechanisms and a portable robot system for early adolescent mental disorder screening, which is novel in its integration and explainability.

Findings

High screening accuracy (73.34%-92.77%) achieved.

Dynamic contribution of modalities to screening results.

Feasibility of explainable multimodal mental health assessment.

Abstract

The timely identification of mental disorders in adolescents is a global public health challenge.Single factor is difficult to detect the abnormality due to its complex and subtle nature. Additionally, the generalized multimodal Computer-Aided Screening (CAS) systems with interactive robots for adolescent mental disorders are not available. Here, we design an android application with mini-games and chat recording deployed in a portable robot to screen 3,783 middle school students and construct the multimodal screening dataset, including facial images, physiological signs, voice recordings, and textual transcripts.We develop a model called GAME (Generalized Model with Attention and Multimodal EmbraceNet) with novel attention mechanism that integrates cross-modal features into the model. GAME evaluates adolescent mental conditions with high accuracy (73.34%-92.77%) and F1-Score (71.32%-91.06%).We find each modality contributes dynamically to the mental disorders screening and comorbidities among various mental disorders, indicating the feasibility of explainable model. This study provides a system capable of acquiring multimodal information and constructs a generalized multimodal integration algorithm with novel attention mechanisms for the early screening of adolescent mental disorders.