Generating bilingual example sentences with large language models as lexicography assistants

TL;DR

This study evaluates large language models' ability to generate and rate bilingual dictionary example sentences across languages with different resource levels, highlighting their strengths, limitations, and potential for lexicography.

Contribution

It introduces a comprehensive evaluation of LLMs for bilingual example generation, including a new dataset and methods for aligning model outputs with human preferences.

Findings

LLMs generate reasonably good examples, but performance drops for low-resource languages.

In-context learning can align LLMs with individual annotator preferences.

Sentence perplexity correlates with example quality in high-resource languages.

Abstract

We present a study of LLMs' performance in generating and rating example sentences for bilingual dictionaries across languages with varying resource levels: French (high-resource), Indonesian (mid-resource), and Tetun (low-resource), with English as the target language. We evaluate the quality of LLM-generated examples against the GDEX (Good Dictionary EXample) criteria: typicality, informativeness, and intelligibility. Our findings reveal that while LLMs can generate reasonably good dictionary examples, their performance degrades significantly for lower-resourced languages. We also observe high variability in human preferences for example quality, reflected in low inter-annotator agreement rates. To address this, we demonstrate that in-context learning can successfully align LLMs with individual annotator preferences. Additionally, we explore the use of pre-trained language models for automated rating of examples, finding that sentence perplexity serves as a good proxy for typicality and intelligibility in higher-resourced languages. Our study also contributes a novel dataset of 600 ratings for LLM-generated sentence pairs, and provides insights into the potential of LLMs in reducing the cost of lexicographic work, particularly for low-resource languages.