TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection

TL;DR

TANDA is a two-step fine-tuning approach for pre-trained Transformer models that improves answer sentence selection accuracy and robustness, outperforming previous methods on benchmark datasets and demonstrating industrial applicability.

Contribution

The paper introduces TANDA, a novel transfer and adaptation fine-tuning method for Transformer models that enhances performance and robustness in answer sentence selection tasks.

Findings

Achieved state-of-the-art MAP scores of 92% on WikiQA and 94.3% on TREC-QA.

TANDA produces more stable and robust models with less hyper-parameter tuning effort.

Effective in noisy datasets and industrial domain applications.

Abstract

We propose TANDA, an effective technique for fine-tuning pre-trained Transformer models for natural language tasks. Specifically, we first transfer a pre-trained model into a model for a general task by fine-tuning it with a large and high-quality dataset. We then perform a second fine-tuning step to adapt the transferred model to the target domain. We demonstrate the benefits of our approach for answer sentence selection, which is a well-known inference task in Question Answering. We built a large scale dataset to enable the transfer step, exploiting the Natural Questions dataset. Our approach establishes the state of the art on two well-known benchmarks, WikiQA and TREC-QA, achieving MAP scores of 92% and 94.3%, respectively, which largely outperform the previous highest scores of 83.4% and 87.5%, obtained in very recent work. We empirically show that TANDA generates more stable and robust models reducing the effort required for selecting optimal hyper-parameters. Additionally, we show that the transfer step of TANDA makes the adaptation step more robust to noise. This enables a more effective use of noisy datasets for fine-tuning. Finally, we also confirm the positive impact of TANDA in an industrial setting, using domain specific datasets subject to different types of noise.