Prompt Engineering: How Prompt Vocabulary affects Domain Knowledge
Dimitri Schreiter

TL;DR
This study investigates how varying the specificity of vocabulary in prompts affects large language models' performance in domain-specific tasks, revealing an optimal range of prompt specificity for improved results.
Contribution
It introduces a systematic synonymization framework to analyze the impact of prompt vocabulary specificity on LLM performance across multiple domains and models.
Findings
Increasing prompt specificity generally has limited impact on performance.
An optimal specificity range enhances LLM question-answering and reasoning.
Identifying this range can improve prompt design for domain-specific applications.
Abstract
Prompt engineering has emerged as a critical component in optimizing large language models (LLMs) for domain-specific tasks. However, the role of prompt specificity, especially in domains like STEM (physics, chemistry, biology, computer science and mathematics), medicine, and law, remains underexplored. This thesis addresses the problem of whether increasing the specificity of vocabulary in prompts improves LLM performance in domain-specific question-answering and reasoning tasks. We developed a synonymization framework to systematically substitute nouns, verbs, and adjectives with varying specificity levels, measuring the impact on four LLMs: Llama-3.1-70B-Instruct, Granite-13B-Instruct-V2, Flan-T5-XL, and Mistral-Large 2, across datasets in STEM, law, and medicine. Our results reveal that while generally increasing the specificity of prompts does not have a significant impact, there…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
