Low-Cost Generation and Evaluation of Dictionary Example Sentences
Bill Cai, Clarence Boon Liang Ng, Daniel Tan, Shelvia Hotama

TL;DR
This paper presents a low-cost, zero-shot approach for generating and evaluating dictionary example sentences using foundational language models and introduces OxfordEval, a new metric that aligns well with human judgments.
Contribution
It introduces OxfordEval for automatic quality assessment and a novel FM-MLM model that significantly outperforms prior models in generating representative dictionary sentences.
Findings
OxfordEval correlates highly with human judgments.
FM-MLM achieves over 85% win rate against Oxford sentences.
Prior models had less than 40% win rate.
Abstract
Dictionary example sentences play an important role in illustrating word definitions and usage, but manually creating quality sentences is challenging. Prior works have demonstrated that language models can be trained to generate example sentences. However, they relied on costly customized models and word sense datasets for generation and evaluation of their work. Rapid advancements in foundational models present the opportunity to create low-cost, zero-shot methods for the generation and evaluation of dictionary example sentences. We introduce a new automatic evaluation metric called OxfordEval that measures the win-rate of generated sentences against existing Oxford Dictionary sentences. OxfordEval shows high alignment with human judgments, enabling large-scale automated quality evaluation. We experiment with various LLMs and configurations to generate dictionary sentences across word…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Taxonomy
TopicsNatural Language Processing Techniques · Topic Modeling · Advanced Text Analysis Techniques
