LLMs are Good Sign Language Translators

TL;DR

This paper introduces SignLLM, a novel framework that converts sign videos into language-like representations, enabling off-the-shelf LLMs to perform sign language translation with state-of-the-art results.

Contribution

The paper presents a new method combining visual sign encoding and optimal transport to improve sign language translation using large language models.

Findings

Achieves state-of-the-art gloss-free translation results

Introduces a novel sign-to-language representation framework

Demonstrates effective sign language translation with off-the-shelf LLMs

Abstract

Sign Language Translation (SLT) is a challenging task that aims to translate sign videos into spoken language. Inspired by the strong translation capabilities of large language models (LLMs) that are trained on extensive multilingual text corpora, we aim to harness off-the-shelf LLMs to handle SLT. In this paper, we regularize the sign videos to embody linguistic characteristics of spoken language, and propose a novel SignLLM framework to transform sign videos into a language-like representation for improved readability by off-the-shelf LLMs. SignLLM comprises two key modules: (1) The Vector-Quantized Visual Sign module converts sign videos into a sequence of discrete character-level sign tokens, and (2) the Codebook Reconstruction and Alignment module converts these character-level tokens into word-level sign representations using an optimal transport formulation. A sign-text alignment loss further bridges the gap between sign and text tokens, enhancing semantic compatibility. We achieve state-of-the-art gloss-free results on two widely-used SLT benchmarks.