CoDi: Conversational Distillation for Grounded Question Answering

TL;DR

CoDi is a novel data distillation framework that synthesizes large-scale conversational datasets, enabling small language models to perform well in grounded question answering without extensive world knowledge memorization.

Contribution

This paper introduces CoDi, a task-agnostic data distillation method that generates large, diverse datasets to improve small language models' conversational grounded reasoning capabilities.

Findings

SLMs trained with CoDi data perform comparably to models trained on human data.

Generated datasets enable SLMs to surpass larger instruction-tuned models in zero-shot tasks.

CoDi effectively synthesizes datasets from web data for improved model training.

Abstract

Distilling conversational skills into Small Language Models (SLMs) with approximately 1 billion parameters presents significant challenges. Firstly, SLMs have limited capacity in their model parameters to learn extensive knowledge compared to larger models. Secondly, high-quality conversational datasets are often scarce, small, and domain-specific. Addressing these challenges, we introduce a novel data distillation framework named CoDi (short for Conversational Distillation, pronounced "Cody"), allowing us to synthesize large-scale, assistant-style datasets in a steerable and diverse manner. Specifically, while our framework is task agnostic at its core, we explore and evaluate the potential of CoDi on the task of conversational grounded reasoning for question answering. This is a typical on-device scenario for specialist SLMs, allowing for open-domain model responses, without requiring the model to "memorize" world knowledge in its limited weights. Our evaluations show that SLMs trained with CoDi-synthesized data achieve performance comparable to models trained on human-annotated data in standard metrics. Additionally, when using our framework to generate larger datasets from web data, our models surpass larger, instruction-tuned models in zero-shot conversational grounded reasoning tasks.