CogPic: A Multimodal Dataset for Early Cognitive Impairment Assessment via Picture Description Tasks

TL;DR

CogPic introduces a large, multimodal dataset with synchronized audio, visual, and linguistic data from 574 participants for early cognitive impairment detection, enabling more robust machine learning models.

Contribution

This paper presents the CogPic dataset, the largest and most detailed multimodal benchmark for cognitive impairment assessment, with expert-validated ground truth and extensive data modalities.

Findings

Benchmark experiments demonstrate the dataset's potential for robust cognitive impairment detection.

CogPic provides a clinically validated, synchronized multimodal dataset for future research.

The dataset supports development of generalizable machine learning models for early dementia diagnosis.

Abstract

The automated evaluation of cognitive status utilizing multimedia technologies presents a promising frontier in early dementia diagnosis. However, the development of robust machine learning models for cognitive impairment detection is frequently hindered by the scarcity of large-scale, strictly synchronized, and clinically validated multimodal datasets. To bridge this critical gap, we introduce the CogPic database, a comprehensive multimodal benchmark meticulously designed for fine-grained cognitive impairment detection. The dataset comprises strictly synchronized audio, visual, and linguistic data continuously collected from 574 participants during a naturalistic picture description task. To establish highly reliable diagnostic ground truth, expert clinical neuropsychologists conducted exhaustive evaluations, stratifying participants into distinct cognitive groups through a comprehensive clinical consensus. Consequently, CogPic stands as the largest, most modality-rich, and most meticulously evaluated dataset of its kind to date. By conducting extensive benchmark experiments on the CogPic dataset, we establish an exceptionally robust, unbiased, and clinically generalizable foundation to propel future multimedia research in automated cognitive health assessment. Detailed information and access application procedures for our CogPic database are available at https://cogpic.github.io/.