Distilling Implicit Multimodal Knowledge into Large Language Models for Zero-Resource Dialogue Generation

TL;DR

This paper introduces VIKDF, a novel framework that distills and integrates implicit multimodal knowledge into large language models, significantly improving zero-resource dialogue generation by leveraging visual cues.

Contribution

The paper presents a new knowledge distillation and integration method for enhancing LLMs with implicit multimodal knowledge in zero-resource scenarios.

Findings

VIKDF outperforms existing models in dialogue quality

Effective encoding of visual implicit knowledge improves coherence

Seamless integration enhances contextual understanding

Abstract

Integrating multimodal knowledge into large language models (LLMs) represents a significant advancement in dialogue generation capabilities. However, the effective incorporation of such knowledge in zero-resource scenarios remains a substantial challenge due to the scarcity of diverse, high-quality dialogue datasets. To address this, we propose the Visual Implicit Knowledge Distillation Framework (VIKDF), an innovative approach aimed at enhancing LLMs for enriched dialogue generation in zero-resource contexts by leveraging implicit multimodal knowledge. VIKDF comprises two main stages: knowledge distillation, using an Implicit Query Transformer to extract and encode visual implicit knowledge from image-text pairs into knowledge vectors; and knowledge integration, employing a novel Bidirectional Variational Information Fusion technique to seamlessly integrate these distilled vectors into LLMs. This enables the LLMs to generate dialogues that are not only coherent and engaging but also exhibit a deep understanding of the context through implicit multimodal cues, effectively overcoming the limitations of zero-resource scenarios. Our extensive experimentation across two dialogue datasets shows that VIKDF outperforms existing state-of-the-art models in generating high-quality dialogues. The code is available at https://github.com/zhangbo-nlp/VIKDF.