SilLang: Improving Gait Recognition with Silhouette Language Encoding

TL;DR

SilLang introduces a novel approach that encodes gait silhouettes into a language-like space using a specialized tokenizer and a dual-branch model, leveraging LLMs to improve gait recognition accuracy across multiple datasets.

Contribution

The paper proposes the Contour-Velocity Tokenizer and Silhouette Language Model to align gait silhouette encoding with natural language, enhancing gait recognition performance.

Findings

Consistently improves state-of-the-art on SUSTech1K, GREW, and Gait3D datasets.

Effectively captures temporal motion patterns using language-inspired discrete encoding.

Enhances gait recognition by integrating linguistic embeddings into visual backbones.

Abstract

Gait silhouettes, which can be encoded into binary gait codes, are widely adopted to representing motion patterns of pedestrian. Recent approaches commonly leverage visual backbones to encode gait silhouettes, achieving successful performance. However, they primarily focus on continuous visual features, overlooking the discrete nature of binary silhouettes that inherently share a discrete encoding space with natural language. Large Language Models (LLMs) have demonstrated exceptional capability in extracting discriminative features from discrete sequences and modeling long-range dependencies, highlighting their potential to capture temporal motion patterns by identifying subtle variations. Motivated by these observations, we explore bridging binary gait silhouettes and natural language within a binary encoding space. However, the encoding spaces of text tokens and binary gait silhouettes remain misaligned, primarily due to differences in token frequency and density. To address this issue, we propose the Contour-Velocity Tokenizer, which encodes binary gait silhouettes while reshaping their distribution to better align with the text token space. We then establish a dual-branch framework termed Silhouette Language Model, which enhances visual silhouettes by integrating discrete linguistic embeddings derived from LLMs. Implemented on mainstream gait backbones, SilLang consistently improves state-of-the-art methods across SUSTech1K, GREW, and Gait3D.