Hybrid Transformer for Early Alzheimer's Detection: Integration of Handwriting-Based 2D Images and 1D Signal Features

TL;DR

This paper introduces a novel multimodal hybrid Transformer model that combines 2D handwriting images and 1D signals for early Alzheimer's detection, achieving state-of-the-art results on the DARWIN dataset.

Contribution

It proposes a learnable multimodal hybrid attention model with a gated mechanism to effectively integrate handwriting modalities for improved AD detection.

Findings

Achieved an F1-score of 90.32% and accuracy of 90.91% on DARWIN dataset.

Surpassed previous best performance by 4.61% in F1-score and 6.06% in accuracy.

Demonstrated the effectiveness of multimodal Transformer architecture in capturing handwriting features.

Abstract

Alzheimer's Disease (AD) is a prevalent neurodegenerative condition where early detection is vital. Handwriting, often affected early in AD, offers a non-invasive and cost-effective way to capture subtle motor changes. State-of-the-art research on handwriting, mostly online, based AD detection has predominantly relied on manually extracted features, fed as input to shallow machine learning models. Some recent works have proposed deep learning (DL)-based models, either 1D-CNN or 2D-CNN architectures, with performance comparing favorably to handcrafted schemes. These approaches, however, overlook the intrinsic relationship between the 2D spatial patterns of handwriting strokes and their 1D dynamic characteristics, thus limiting their capacity to capture the multimodal nature of handwriting data. Moreover, the application of Transformer models remains basically unexplored. To address these limitations, we propose a novel approach for AD detection, consisting of a learnable multimodal hybrid attention model that integrates simultaneously 2D handwriting images with 1D dynamic handwriting signals. Our model leverages a gated mechanism to combine similarity and difference attention, blending the two modalities and learning robust features by incorporating information at different scales. Our model achieved state-of-the-art performance on the DARWIN dataset, with an F1-score of 90.32\% and accuracy of 90.91\% in Task 8 ('L' writing), surpassing the previous best by 4.61% and 6.06% respectively.