Understanding Cross Task Generalization in Handwriting-Based Alzheimer's Screening via Vision Language Adaptation

TL;DR

This paper introduces a novel framework that adapts vision-language models for handwriting-based Alzheimer's screening, enabling zero-shot cross-task generalization and revealing key writing patterns for early diagnosis.

Contribution

It proposes a lightweight Cross-Layer Fusion Adapter framework that repurposes CLIP for handwriting-based AD detection, enhancing cross-task generalization and diagnostic insights.

Findings

Effective zero-shot inference for handwriting-based AD screening.

Identification of key stroke patterns linked to early AD.

Benchmark for handwriting-based cognitive assessment.

Abstract

Alzheimer's disease is a prevalent neurodegenerative disorder for which early detection is critical. Handwriting-often disrupted in prodromal AD-provides a non-invasive and cost-effective window into subtle motor and cognitive decline. Existing handwriting-based AD studies, mostly relying on online trajectories and hand-crafted features, have not systematically examined how task type influences diagnostic performance and cross-task generalization. Meanwhile, large-scale vision language models have demonstrated remarkable zero or few-shot anomaly detection in natural images and strong adaptability across medical modalities such as chest X-ray and brain MRI. However, handwriting-based disease detection remains largely unexplored within this paradigm. To close this gap, we introduce a lightweight Cross-Layer Fusion Adapter framework that repurposes CLIP for handwriting-based AD screening. CLFA implants multi-level fusion adapters within the visual encoder to progressively align representations toward handwriting-specific medical cues, enabling prompt-free and efficient zero-shot inference. Using this framework, we systematically investigate cross-task generalization-training on a specific handwriting task and evaluating on unseen ones-to reveal which task types and writing patterns most effectively discriminate AD. Extensive analyses further highlight characteristic stroke patterns and task-level factors that contribute to early AD identification, offering both diagnostic insights and a benchmark for handwriting-based cognitive assessment.