Lexical Knowledge Internalization for Neural Dialog Generation

TL;DR

This paper introduces knowledge internalization (KI), a method for embedding lexical knowledge directly into neural dialog models' parameters, enhancing knowledge integration without external retrieval.

Contribution

It presents a novel contrastive learning approach for internalizing lexical knowledge into neural dialog models, reducing reliance on external knowledge retrieval.

Findings

Effective token-level lexical knowledge retriever developed

Improved dialog generation across multiple datasets

Applicable to various model architectures

Abstract

We propose knowledge internalization (KI), which aims to complement the lexical knowledge into neural dialog models. Instead of further conditioning the knowledge-grounded dialog (KGD) models on externally retrieved knowledge, we seek to integrate knowledge about each input token internally into the model's parameters. To tackle the challenge due to the large scale of lexical knowledge, we adopt the contrastive learning approach and create an effective token-level lexical knowledge retriever that requires only weak supervision mined from Wikipedia. We demonstrate the effectiveness and general applicability of our approach on various datasets and diversified model structures.