SignAgent: Agentic LLMs for Linguistically-Grounded Sign Language Annotation and Dataset Curation

TL;DR

SignAgent leverages large language models to automate and improve the linguistic annotation and dataset curation process for Sign Language, addressing limitations of traditional methods and manual efforts.

Contribution

The paper introduces SignAgent, a novel agentic framework combining reasoning and knowledge-grounded LLMs for scalable, linguistically-aware Sign Language annotation and dataset curation.

Findings

Effective in pseudo-gloss annotation with multi-modal evidence

Accurately detects and refines visual clusters for ID glossing

Achieves strong performance in large-scale linguistic annotation

Abstract

This paper introduces SignAgent, a novel agentic framework that utilises Large Language Models (LLMs) for scalable, linguistically-grounded Sign Language (SL) annotation and dataset curation. Traditional computational methods for SLs often operate at the gloss level, overlooking crucial linguistic nuances, while manual linguistic annotation remains a significant bottleneck, proving too slow and expensive for the creation of large-scale, phonologically-aware datasets. SignAgent addresses these challenges through SignAgent Orchestrator, a reasoning LLM that coordinates a suite of linguistic tools, and SignGraph, a knowledge-grounded LLM that provides lexical and linguistic grounding. We evaluate our framework on two downstream annotation tasks. First, on Pseudo-gloss Annotation, where the agent performs constrained assignment, using multi-modal evidence to extract and order suitable gloss labels for signed sequences. Second, on ID Glossing, where the agent detects and refines visual clusters by reasoning over both visual similarity and phonological overlap to correctly identify and group lexical sign variants. Our results demonstrate that our agentic approach achieves strong performance for large-scale, linguistically-aware data annotation and curation.