Transformer Models for Text Coherence Assessment

TL;DR

This paper introduces four Transformer-based models for automated text coherence assessment, significantly improving accuracy over previous methods across multiple datasets and tasks.

Contribution

It proposes novel Transformer architectures tailored for coherence evaluation, addressing limitations of prior approaches and achieving state-of-the-art results.

Findings

Transformer models outperform existing methods

Hierarchical and multi-task models enhance performance

Models generalize well across domains and tasks

Abstract

Coherence is an important aspect of text quality and is crucial for ensuring its readability. It is essential desirable for outputs from text generation systems like summarization, question answering, machine translation, question generation, table-to-text, etc. An automated coherence scoring model is also helpful in essay scoring or providing writing feedback. A large body of previous work has leveraged entity-based methods, syntactic patterns, discourse relations, and more recently traditional deep learning architectures for text coherence assessment. Previous work suffers from drawbacks like the inability to handle long-range dependencies, out-of-vocabulary words, or model sequence information. We hypothesize that coherence assessment is a cognitively complex task that requires deeper models and can benefit from other related tasks. Accordingly, in this paper, we propose four different Transformer-based architectures for the task: vanilla Transformer, hierarchical Transformer, multi-task learning-based model, and a model with fact-based input representation. Our experiments with popular benchmark datasets across multiple domains on four different coherence assessment tasks demonstrate that our models achieve state-of-the-art results outperforming existing models by a good margin.