Neural Architecture Search with Multimodal Fusion Methods for Diagnosing Dementia

TL;DR

This paper introduces a novel approach combining neural architecture search with advanced multimodal fusion techniques to improve early dementia detection from speech and text, outperforming existing methods.

Contribution

It is the first to integrate NAS with multimodal fusion methods like Bilinear Pooling and Tucker Decomposition for dementia diagnosis from spontaneous speech.

Findings

Outperforms state-of-the-art methods on ADReSS dataset

Demonstrates the effectiveness of NAS in optimizing CNN architectures for this task

Shows that advanced fusion methods improve multimodal integration accuracy

Abstract

Alzheimer's dementia (AD) affects memory, thinking, and language, deteriorating person's life. An early diagnosis is very important as it enables the person to receive medical help and ensure quality of life. Therefore, leveraging spontaneous speech in conjunction with machine learning methods for recognizing AD patients has emerged into a hot topic. Most of the previous works employ Convolutional Neural Networks (CNNs), to process the input signal. However, finding a CNN architecture is a time-consuming process and requires domain expertise. Moreover, the researchers introduce early and late fusion approaches for fusing different modalities or concatenate the representations of the different modalities during training, thus the inter-modal interactions are not captured. To tackle these limitations, first we exploit a Neural Architecture Search (NAS) method to automatically find a high performing CNN architecture. Next, we exploit several fusion methods, including Multimodal Factorized Bilinear Pooling and Tucker Decomposition, to combine both speech and text modalities. To the best of our knowledge, there is no prior work exploiting a NAS approach and these fusion methods in the task of dementia detection from spontaneous speech. We perform extensive experiments on the ADReSS Challenge dataset and show the effectiveness of our approach over state-of-the-art methods.