ArtCognition: A Multimodal AI Framework for Affective State Sensing from Visual and Kinematic Drawing Cues

TL;DR

ArtCognition introduces a multimodal AI framework that combines visual and kinematic data from drawings to improve the assessment of psychological and affective states, leveraging psychological knowledge for explainability.

Contribution

The paper presents a novel multimodal framework that fuses visual and behavioral drawing cues with RAG architecture for affective sensing, enhancing interpretability and accuracy.

Findings

Fusion of visual and kinematic cues improves assessment accuracy.

Significant correlations with psychological metrics validate the approach.

Framework supports scalable, non-intrusive mental health assessment.

Abstract

The objective assessment of human affective and psychological states presents a significant challenge, particularly through non-verbal channels. This paper introduces digital drawing as a rich and underexplored modality for affective sensing. We present a novel multimodal framework, named ArtCognition, for the automated analysis of the House-Tree-Person (HTP) test, a widely used psychological instrument. ArtCognition uniquely fuses two distinct data streams: static visual features from the final artwork, captured by computer vision models, and dynamic behavioral kinematic cues derived from the drawing process itself, such as stroke speed, pauses, and smoothness. To bridge the gap between low-level features and high-level psychological interpretation, we employ a Retrieval-Augmented Generation (RAG) architecture. This grounds the analysis in established psychological knowledge, enhancing explainability and reducing the potential for model hallucination. Our results demonstrate that the fusion of visual and behavioral kinematic cues provides a more nuanced assessment than either modality alone. We show significant correlations between the extracted multimodal features and standardized psychological metrics, validating the framework's potential as a scalable tool to support clinicians. This work contributes a new methodology for non-intrusive affective state assessment and opens new avenues for technology-assisted mental healthcare.