Prompt Engineering: How Prompt Vocabulary affects Domain Knowledge

TL;DR

This study investigates how varying the specificity of vocabulary in prompts affects large language models' performance in domain-specific tasks, revealing an optimal range of prompt specificity for improved results.

Contribution

It introduces a systematic synonymization framework to analyze the impact of prompt vocabulary specificity on LLM performance across multiple domains and models.

Findings

Increasing prompt specificity generally has limited impact on performance.

An optimal specificity range enhances LLM question-answering and reasoning.

Identifying this range can improve prompt design for domain-specific applications.

Abstract

Prompt engineering has emerged as a critical component in optimizing large language models (LLMs) for domain-specific tasks. However, the role of prompt specificity, especially in domains like STEM (physics, chemistry, biology, computer science and mathematics), medicine, and law, remains underexplored. This thesis addresses the problem of whether increasing the specificity of vocabulary in prompts improves LLM performance in domain-specific question-answering and reasoning tasks. We developed a synonymization framework to systematically substitute nouns, verbs, and adjectives with varying specificity levels, measuring the impact on four LLMs: Llama-3.1-70B-Instruct, Granite-13B-Instruct-V2, Flan-T5-XL, and Mistral-Large 2, across datasets in STEM, law, and medicine. Our results reveal that while generally increasing the specificity of prompts does not have a significant impact, there appears to be a specificity range, across all considered models, where the LLM performs the best. Identifying this optimal specificity range offers a key insight for prompt design, suggesting that manipulating prompts within this range could maximize LLM performance and lead to more efficient applications in specialized domains.