Knowledge Transfer from Answer Ranking to Answer Generation

TL;DR

This paper introduces a method to improve answer generation in question answering systems by transferring knowledge from an answer ranking model, reducing the need for large supervised datasets and enhancing answer synthesis.

Contribution

The authors propose a novel knowledge transfer approach from answer ranking to answer generation, utilizing prediction scores for better training without large supervised datasets.

Findings

Outperforms baseline answer ranking models

Achieves superior results over supervised GenQA models

Effective on multiple public and industrial datasets

Abstract

Recent studies show that Question Answering (QA) based on Answer Sentence Selection (AS2) can be improved by generating an improved answer from the top-k ranked answer sentences (termed GenQA). This allows for synthesizing the information from multiple candidates into a concise, natural-sounding answer. However, creating large-scale supervised training data for GenQA models is very challenging. In this paper, we propose to train a GenQA model by transferring knowledge from a trained AS2 model, to overcome the aforementioned issue. First, we use an AS2 model to produce a ranking over answer candidates for a set of questions. Then, we use the top ranked candidate as the generation target, and the next k top ranked candidates as context for training a GenQA model. We also propose to use the AS2 model prediction scores for loss weighting and score-conditioned input/output shaping, to aid the knowledge transfer. Our evaluation on three public and one large industrial datasets demonstrates the superiority of our approach over the AS2 baseline, and GenQA trained using supervised data.