Prompting with Sign Parameters for Low-resource Sign Language Instruction Generation

TL;DR

This paper introduces BdSLIG, a Bengali sign language dataset, and proposes Sign Parameter-Infused prompting to improve zero-shot sign language instruction generation, enhancing inclusivity for under-resourced communities.

Contribution

The paper presents the first Bengali SLIG dataset and a novel SPI prompting method that incorporates sign parameters into prompts for better zero-shot performance.

Findings

BdSLIG enables evaluation of VLMs on under-resourced SL tasks.

SPI prompting improves zero-shot instruction generation accuracy.

Structured prompts outperform free-form natural language prompts.

Abstract

Sign Language (SL) enables two-way communication for the deaf and hard-of-hearing community, yet many sign languages remain under-resourced in the AI space. Sign Language Instruction Generation (SLIG) produces step-by-step textual instructions that enable non-SL users to imitate and learn SL gestures, promoting two-way interaction. We introduce BdSLIG, the first Bengali SLIG dataset, used to evaluate Vision Language Models (VLMs) (i) on under-resourced SLIG tasks, and (ii) on long-tail visual concepts, as Bengali SL is unlikely to appear in the VLM pre-training data. To enhance zero-shot performance, we introduce Sign Parameter-Infused (SPI) prompting, which integrates standard SL parameters, like hand shape, motion, and orientation, directly into the textual prompts. Subsuming standard sign parameters into the prompt makes the instructions more structured and reproducible than free-form natural text from vanilla prompting. We envision that our work would promote inclusivity and advancement in SL learning systems for the under-resourced communities.