Improving LLM's Attachment to External Knowledge In Dialogue Generation Tasks Through Entity Anonymization

TL;DR

This paper introduces a new evaluation method and an entity anonymization technique to enhance large language models' ability to incorporate external knowledge in dialogue generation, addressing their tendency to rely on internal knowledge.

Contribution

The paper proposes LLM-KAT for measuring knowledge attachment and a simple anonymization method to improve LLMs' external knowledge utilization in KG-DG tasks.

Findings

Improved knowledge attachment in LLM responses.

Entity anonymization enhances external knowledge usage.

Evaluation shows significant gains on OpenDialKG dataset.

Abstract

Knowledge graph-based dialogue generation (KG-DG) is a challenging task requiring models to effectively incorporate external knowledge into conversational responses. While large language models (LLMs) have achieved impressive results across various NLP tasks, their ability to utilize external knowledge in KG-DG remains under-explored. We observe that LLMs often rely on internal knowledge, leading to detachment from provided knowledge graphs, even when they are given a flawlessly retrieved knowledge graph. First, we introduce LLM-KAT, an evaluation procedure for measuring knowledge attachment in generated responses. Second, we propose a simple yet effective entity anonymization technique to encourage LLMs to better leverage external knowledge. Experiments on the OpenDialKG dataset demonstrate that our approach improves LLMs' attachment on external knowledge.