# Small pre-trained model for background understanding in multi-round question answering

## TL;DR

This paper introduces a method to train a small, efficient model for multi-round question answering that performs as well as larger, resource-heavy models.

## Contribution

The novel contribution is a knowledge transfer approach combining distillation, co-learning, and fine-tuning to optimize small models.

## Key findings

- Small models can match or exceed large models in performance with multi-knowledge cooperative training.
- Combining co-learning across datasets and tasks improves efficiency and effectiveness.
- The proposed method reduces resource consumption while maintaining high accuracy.

## Abstract

Multi-round Q&A based on background text needs to infer the answer to the question through the current question, historical Q&A pairs, and background text. The pre-trained model has proved its effectiveness in this task; however, the existing model has many problems such as too many parameters and high resource consumption. We propose a knowledge transfer method that combines knowledge distillation, co-learning of similar datasets, and fine-tuning of similar tasks. Through multi-knowledge cooperative training from large model to small model, between different data sets, and between different tasks, the performance of the small model with low resource consumption can match or surpass that of the large model.