Rethinking Semantic Parsing for Large Language Models: Enhancing LLM Performance with Semantic Hints

TL;DR

This paper investigates how semantic parsing impacts large language models and introduces SENSE, a prompting method that embeds semantic hints to enhance LLM performance across multiple tasks.

Contribution

It reveals that direct semantic parsing results can hinder LLMs and proposes SENSE, a novel prompting technique that effectively incorporates semantic hints to improve performance.

Findings

Directly adding semantic parsing results reduces LLM performance.

SENSE consistently improves LLM performance across various tasks.

Semantic hints embedded in prompts enhance LLM capabilities.

Abstract

Semantic Parsing aims to capture the meaning of a sentence and convert it into a logical, structured form. Previous studies show that semantic parsing enhances the performance of smaller models (e.g., BERT) on downstream tasks. However, it remains unclear whether the improvements extend similarly to LLMs. In this paper, our empirical findings reveal that, unlike smaller models, directly adding semantic parsing results into LLMs reduces their performance. To overcome this, we propose SENSE, a novel prompting approach that embeds semantic hints within the prompt. Experiments show that SENSE consistently improves LLMs' performance across various tasks, highlighting the potential of integrating semantic information to improve LLM capabilities.