Can Multimodal Large Language Models Understand Pathologic Movements? A Pilot Study on Seizure Semiology

TL;DR

This study explores the use of multimodal large language models for recognizing pathological seizure movements in videos, showing promising zero-shot performance and potential for clinical application with targeted preprocessing.

Contribution

It demonstrates that general-purpose multimodal large language models can effectively recognize seizure semiology features without task-specific training, especially with feature-targeted signal enhancement.

Findings

MLLMs outperform CNN and ViT baselines on 13 of 18 features.

Preprocessing improves performance on 10 of 20 features.

94.3% of explanations for correct predictions are highly faithful.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated robust capabilities in recognizing everyday human activities, yet their potential for analyzing clinically significant involuntary movements in neurological disorders remains largely unexplored. This pilot study evaluates the capability of MLLMs for automated recognition of pathological movements in seizure videos. We assessed the zero-shot performance of state-of-the-art MLLMs on 20 ILAE-defined semiological features across 90 clinical seizure recordings. MLLMs outperformed fine-tuned Convolutional Neural Network (CNN) and Vision Transformer (ViT) baseline models on 13 of 18 features without task-specific training, demonstrating particular strength in recognizing salient postural and contextual features while struggling with subtle, high-frequency movements. Feature-targeted signal enhancement (facial cropping, pose estimation, audio denoising) improved performance on 10 of 20 features. Expert evaluation showed that 94.3 percent of MLLM-generated explanations for correctly predicted cases achieved at least 60 percent faithfulness scores, aligning with epileptologist reasoning. These findings demonstrate the potential of adapting general-purpose MLLMs for specialized clinical video analysis through targeted preprocessing strategies, offering a path toward interpretable, efficient diagnostic assistance. Our code is publicly available at https://github.com/LinaZhangUCLA/PathMotionMLLM.