Improving Continuous Sign Language Recognition with Adapted Image Models

TL;DR

This paper introduces AdaptSign, a lightweight adaptation strategy for large vision-language models like CLIP, enabling efficient and effective continuous sign language recognition while preserving pretraining knowledge.

Contribution

AdaptSign employs fixed CLIP features with learnable modules for spatial and temporal modeling, achieving high efficiency and superior performance in CSLR tasks.

Findings

AdaptSign outperforms existing CSLR methods on multiple benchmarks.

The additional modules only add 3.2% extra computations.

Visualizations show effective focus on informative regions and trajectories.

Abstract

The increase of web-scale weakly labelled image-text pairs have greatly facilitated the development of large-scale vision-language models (e.g., CLIP), which have shown impressive generalization performance over a series of downstream tasks. However, the massive model size and scarcity of available data limit their applications to fine-tune the whole model in downstream tasks. Besides, fully fine-tuning the model easily forgets the generic essential knowledge acquired in the pretraining stage and overfits the downstream data. To enable high efficiency when adapting these large vision-language models (e.g., CLIP) to performing continuous sign language recognition (CSLR) while preserving their generalizability, we propose a novel strategy (AdaptSign). Especially, CLIP is adopted as the visual backbone to extract frame-wise features whose parameters are fixed, and a set of learnable modules are introduced to model spatial sign variations or capture temporal sign movements. The introduced additional modules are quite lightweight, only owning 3.2% extra computations with high efficiency. The generic knowledge acquired in the pretraining stage is well-preserved in the frozen CLIP backbone in this process. Extensive experiments show that despite being efficient, AdaptSign is able to demonstrate superior performance across a series of CSLR benchmarks including PHOENIX14, PHOENIX14-T, CSL-Daily and CSL compared to existing methods. Visualizations show that AdaptSign could learn to dynamically pay major attention to the informative spatial regions and cross-frame trajectories in sign videos.