A Generative Approach to Titling and Clustering Wikipedia Sections

TL;DR

This paper explores transformer-based models for generating Wikipedia section titles and embeddings, demonstrating how different decoder architectures impact extractive heading generation and semantic encoding, with a new loss function enhancing embedding quality.

Contribution

It introduces a novel task of section heading generation, compares various transformer decoder architectures, and proposes a new loss function to improve section embedding quality.

Findings

Attention-based decoders excel at extractive heading generation.

Decoders without attention produce better semantic embeddings.

The new loss function enhances embedding quality.

Abstract

We evaluate the performance of transformer encoders with various decoders for information organization through a new task: generation of section headings for Wikipedia articles. Our analysis shows that decoders containing attention mechanisms over the encoder output achieve high-scoring results by generating extractive text. In contrast, a decoder without attention better facilitates semantic encoding and can be used to generate section embeddings. We additionally introduce a new loss function, which further encourages the decoder to generate high-quality embeddings.