Text-Guided Multi-Instance Learning for Scoliosis Screening via Gait Video Analysis

TL;DR

This paper introduces TG-MILNet, a novel non-invasive gait video analysis method for scoliosis screening that leverages text guidance, temporal attention, and dynamic clustering to improve detection accuracy, especially for borderline cases.

Contribution

The paper presents a new multi-instance learning framework incorporating textual guidance, temporal attention, and dynamic clustering for improved scoliosis detection from gait videos.

Findings

Achieves state-of-the-art performance on Scoliosis1K dataset.

Effectively handles class imbalance and borderline cases.

Enhances interpretability with domain-guided textual features.

Abstract

Early-stage scoliosis is often difficult to detect, particularly in adolescents, where delayed diagnosis can lead to serious health issues. Traditional X-ray-based methods carry radiation risks and rely heavily on clinical expertise, limiting their use in large-scale screenings. To overcome these challenges, we propose a Text-Guided Multi-Instance Learning Network (TG-MILNet) for non-invasive scoliosis detection using gait videos. To handle temporal misalignment in gait sequences, we employ Dynamic Time Warping (DTW) clustering to segment videos into key gait phases. To focus on the most relevant diagnostic features, we introduce an Inter-Bag Temporal Attention (IBTA) mechanism that highlights critical gait phases. Recognizing the difficulty in identifying borderline cases, we design a Boundary-Aware Model (BAM) to improve sensitivity to subtle spinal deviations. Additionally, we incorporate textual guidance from domain experts and large language models (LLM) to enhance feature representation and improve model interpretability. Experiments on the large-scale Scoliosis1K gait dataset show that TG-MILNet achieves state-of-the-art performance, particularly excelling in handling class imbalance and accurately detecting challenging borderline cases. The code is available at https://github.com/lhqqq/TG-MILNet