Recognising BSL Fingerspelling in Continuous Signing Sequences

TL;DR

This paper introduces a large-scale BSL fingerspelling dataset and a recognition model that leverages bi-manual and mouthing cues, significantly improving recognition accuracy and supporting future sign language research.

Contribution

The work presents a new extensive BSL fingerspelling dataset and a novel recognition model that explicitly models bi-manual and mouthing cues, reducing character error rate.

Findings

Halves the character error rate compared to previous methods

Demonstrates the effectiveness of explicit bi-manual and mouthing cue modeling

Provides a scalable dataset for future sign language research

Abstract

Fingerspelling is a critical component of British Sign Language (BSL), used to spell proper names, technical terms, and words that lack established lexical signs. Fingerspelling recognition is challenging due to the rapid pace of signing and common letter omissions by native signers, while existing BSL fingerspelling datasets are either small in scale or temporally and letter-wise inaccurate. In this work, we introduce a new large-scale BSL fingerspelling dataset, FS23K, constructed using an iterative annotation framework. In addition, we propose a fingerspelling recognition model that explicitly accounts for bi-manual interactions and mouthing cues. As a result, with refined annotations, our approach halves the character error rate (CER) compared to the prior state of the art on fingerspelling recognition. These findings demonstrate the effectiveness of our method and highlight its potential to support future research in sign language understanding and scalable, automated annotation pipelines. The project page can be found at https://taeinkwon.com/projects/fs23k/.