Injecting Domain Knowledge in Language Models for Task-Oriented Dialogue Systems

TL;DR

This paper demonstrates that injecting small, domain-specific knowledge bases into pre-trained language models via adapters improves performance on task-oriented dialogue tasks, addressing the limitations of static, large-scale knowledge bases.

Contribution

It introduces a method for injecting domain-specific knowledge into PLMs using lightweight adapters and proposes a new probing task, KPRS, to evaluate knowledge integration in TOD systems.

Findings

Knowledge injection with adapters improves response accuracy.

The proposed KPRS probe effectively measures knowledge integration.

Adapters facilitate easy incorporation of domain knowledge.

Abstract

Pre-trained language models (PLM) have advanced the state-of-the-art across NLP applications, but lack domain-specific knowledge that does not naturally occur in pre-training data. Previous studies augmented PLMs with symbolic knowledge for different downstream NLP tasks. However, knowledge bases (KBs) utilized in these studies are usually large-scale and static, in contrast to small, domain-specific, and modifiable knowledge bases that are prominent in real-world task-oriented dialogue (TOD) systems. In this paper, we showcase the advantages of injecting domain-specific knowledge prior to fine-tuning on TOD tasks. To this end, we utilize light-weight adapters that can be easily integrated with PLMs and serve as a repository for facts learned from different KBs. To measure the efficacy of proposed knowledge injection methods, we introduce Knowledge Probing using Response Selection (KPRS) -- a probe designed specifically for TOD models. Experiments on KPRS and the response generation task show improvements of knowledge injection with adapters over strong baselines.