DEPTH: Discourse Education through Pre-Training Hierarchically

TL;DR

DEPTH is a hierarchical pre-training approach for language models that enhances discourse understanding by learning sentence-level representations with novel objectives, leading to improved performance on discourse and NLU tasks.

Contribution

It introduces DEPTH, a discourse-oriented pre-training method that combines hierarchical sentence representations with novel objectives, improving discourse capabilities of language models.

Findings

DEPTH outperforms T5 in span-corruption loss.

DEPTH learns faster and better on discourse tasks.

Minimal impact on other NLU capabilities.

Abstract

Language Models (LMs) struggle with linguistic understanding at the discourse level, even though discourse patterns such as coherence, cohesion, and narrative flow are prevalent in their pre-training data. To improve the discourse capabilities of LMs already at the pre-training stage, we introduce DEPTH, an encoder-decoder model that learns latent representations for sentences using a discourse-oriented pre-training objective. DEPTH combines hierarchical sentence representations with two objectives: (1) Sentence Un-Shuffling, and (2) Span-Corruption. Our approach trains the model to represent both sub-word-level and sentence-level dependencies over a pre-training corpora. When trained either from scratch or continuing from a pre-trained T5 checkpoint, DEPTH learns semantic and discourse-level representations faster than T5, outperforming it in span-corruption loss despite the additional sentence-un-shuffling objective. Evaluations on the GLUE, DiscoEval, and NI benchmarks demonstrate DEPTH's ability to quickly learn diverse downstream tasks, which require syntactic, semantic, and discourse capabilities. Our approach extends the discourse capabilities of T5, while minimally impacting other natural language understanding (NLU) capabilities in the resulting LM. We share our codebase for reproducibility: https://github.com/zbambergerNLP/depth.git.