Probing Task-Oriented Dialogue Representation from Language Models

TL;DR

This paper evaluates pre-trained language models to identify which best encodes information for task-oriented dialogue, using supervised and unsupervised probing methods to guide model selection and understand pre-training impacts.

Contribution

It introduces an unsupervised mutual information probing method and offers insights into model selection and pre-training factors for dialogue tasks.

Findings

Supervised classifier probe assesses model performance on dialogue tasks.

Unsupervised mutual information probe evaluates the dependence between representations and true clusters.

Guidelines for selecting pre-trained models for dialogue applications.

Abstract

This paper investigates pre-trained language models to find out which model intrinsically carries the most informative representation for task-oriented dialogue tasks. We approach the problem from two aspects: supervised classifier probe and unsupervised mutual information probe. We fine-tune a feed-forward layer as the classifier probe on top of a fixed pre-trained language model with annotated labels in a supervised way. Meanwhile, we propose an unsupervised mutual information probe to evaluate the mutual dependence between a real clustering and a representation clustering. The goals of this empirical paper are to 1) investigate probing techniques, especially from the unsupervised mutual information aspect, 2) provide guidelines of pre-trained language model selection for the dialogue research community, 3) find insights of pre-training factors for dialogue application that may be the key to success.