How Do Lexical Senses Correspond Between Spoken German and German Sign Language?

TL;DR

This study analyzes how lexical senses in spoken German correspond to signs in German Sign Language, creating a new annotated dataset and evaluating computational methods to identify sense-to-sign mappings, revealing significant insights into cross-modal lexicography.

Contribution

It introduces the first annotated dataset for cross-modal sense correspondence between spoken German and DGS, and evaluates computational methods for identifying sense-to-sign mappings.

Findings

Semantic Similarity significantly outperforms Exact Match (88.52% vs. 71.31%)

Semantic Similarity greatly improves detection of one-to-many mappings (+52.1 percentage points)

The dataset and code are publicly available for further research.

Abstract

Sign language lexicographers construct bilingual dictionaries by establishing word-to-sign mappings, where polysemous and homonymous words corresponding to different signs across contexts are often underrepresented. A usage-based approach examining how word senses map to signs can identify such novel mappings absent from current dictionaries, enriching lexicographic resources. We address this by analyzing German and German Sign Language (Deutsche Geb\"ardensprache, DGS), manually annotating 1,404 word use-to-sign ID mappings derived from 32 words from the German Word Usage Graph (D-WUG) and 49 signs from the Digital Dictionary of German Sign Language (DW-DGS). We identify three correspondence types: Type 1 (one-to-many), Type 2 (many-to-one), and Type 3 (one-to-one), plus No Match cases. We evaluate computational methods: Exact Match (EM) and Semantic Similarity (SS) using SBERT embeddings. SS substantially outperforms EM overall 88.52% vs. 71.31%), with dramatic gains for Type 1 (+52.1 pp). Our work establishes the first annotated dataset for cross-modal sense correspondence and reveals which correspondence patterns are computationally identifiable. Our code and dataset are made publicly available.