TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents

TL;DR

This paper presents TransferTransfo, a transfer learning approach using Transformer models and multi-task fine-tuning that significantly improves neural conversational agents' performance on dialogue benchmarks.

Contribution

It introduces a novel transfer learning method with multi-task fine-tuning for Transformer-based dialogue systems, achieving state-of-the-art results.

Findings

Achieved a perplexity of 16.28 on PERSONA-CHAT

Attained 80.7% Hits@1 accuracy

Reached 19.5 F1 score, outperforming previous models

Abstract

We introduce a new approach to generative data-driven dialogue systems (e.g. chatbots) called TransferTransfo which is a combination of a Transfer learning based training scheme and a high-capacity Transformer model. Fine-tuning is performed by using a multi-task objective which combines several unsupervised prediction tasks. The resulting fine-tuned model shows strong improvements over the current state-of-the-art end-to-end conversational models like memory augmented seq2seq and information-retrieval models. On the privately held PERSONA-CHAT dataset of the Conversational Intelligence Challenge 2, this approach obtains a new state-of-the-art, with respective perplexity, Hits@1 and F1 metrics of 16.28 (45 % absolute improvement), 80.7 (46 % absolute improvement) and 19.5 (20 % absolute improvement).