Reasoning Like Experts: Leveraging Multimodal Large Language Models for Drawing-based Psychoanalysis

TL;DR

This paper presents PICK, a hierarchical framework leveraging multimodal large language models for psychoanalytical interpretation of drawings, especially the House-Tree-Person test, integrating knowledge injection and multi-level analysis to assess psychological states.

Contribution

The paper introduces a novel multi-step hierarchical framework, PICK, that combines MLLMs with knowledge injection and structured analysis for drawing-based psychological assessment.

Findings

PICK improves MLLMs' performance in psychoanalytical tasks.

Hierarchical analysis captures spatial and content-based psychological cues.

Framework extends to emotion understanding tasks.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated exceptional performance across various objective multimodal perception tasks, yet their application to subjective, emotionally nuanced domains, such as psychological analysis, remains largely unexplored. In this paper, we introduce PICK, a multi-step framework designed for Psychoanalytical Image Comprehension through hierarchical analysis and Knowledge injection with MLLMs, specifically focusing on the House-Tree-Person (HTP) Test, a widely used psychological assessment in clinical practice. First, we decompose drawings containing multiple instances into semantically meaningful sub-drawings, constructing a hierarchical representation that captures spatial structure and content across three levels: single-object level, multi-object level, and whole level. Next, we analyze these sub-drawings at each level with a targeted focus, extracting psychological or emotional insights from their visual cues. We also introduce an HTP knowledge base and design a feature extraction module, trained with reinforcement learning, to generate a psychological profile for single-object level analysis. This profile captures both holistic stylistic features and dynamic object-specific features (such as those of the house, tree, or person), correlating them with psychological states. Finally, we integrate these multi-faceted information to produce a well-informed assessment that aligns with expert-level reasoning. Our approach bridges the gap between MLLMs and specialized expert domains, offering a structured and interpretable framework for understanding human mental states through visual expression. Experimental results demonstrate that the proposed PICK significantly enhances the capability of MLLMs in psychological analysis. It is further validated as a general framework through extensions to emotion understanding tasks.