Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations

TL;DR

This paper introduces DiscoEval, a comprehensive benchmark suite for evaluating discourse-aware sentence representations, and proposes new training objectives using Wikipedia annotations to improve encoding of broader context.

Contribution

It presents DiscoEval as a new evaluation benchmark and introduces training objectives leveraging Wikipedia annotations to enhance discourse-aware sentence encoding.

Findings

Training objectives improve encoding of document structure.

BERT and ELMo show strong performance with layer-specific characteristics.

DiscoEval effectively evaluates discourse-aware representations.

Abstract

Prior work on pretrained sentence embeddings and benchmarks focus on the capabilities of stand-alone sentences. We propose DiscoEval, a test suite of tasks to evaluate whether sentence representations include broader context information. We also propose a variety of training objectives that makes use of natural annotations from Wikipedia to build sentence encoders capable of modeling discourse. We benchmark sentence encoders pretrained with our proposed training objectives, as well as other popular pretrained sentence encoders on DiscoEval and other sentence evaluation tasks. Empirically, we show that these training objectives help to encode different aspects of information in document structures. Moreover, BERT and ELMo demonstrate strong performances over DiscoEval with individual hidden layers showing different characteristics.