A Unifying Scheme for Extractive Content Selection Tasks

TL;DR

This paper introduces a unified instruction-guided framework for diverse extractive content selection tasks in NLP, along with a new benchmark and synthetic dataset to improve transfer learning and evaluation.

Contribution

It proposes a novel unified framework, introduces the first comprehensive benchmark, and creates a synthetic dataset to enhance transfer learning across content selection tasks.

Findings

Transfer learning improves performance across tasks.

Unified benchmark facilitates comparison of models.

Synthetic dataset boosts model generalization.

Abstract

A broad range of NLP tasks involve selecting relevant text spans from given source texts. Despite this shared objective, such \textit{content selection} tasks have traditionally been studied in isolation, each with its own modeling approaches, datasets, and evaluation metrics. In this work, we propose \textit{instruction-guided content selection (IGCS)} as a beneficial unified framework for such settings, where the task definition and any instance-specific request are encapsulated as instructions to a language model. To promote this framework, we introduce \igcsbench{}, the first unified benchmark covering diverse content selection tasks. Further, we create a large generic synthetic dataset that can be leveraged for diverse content selection tasks, and show that transfer learning with these datasets often boosts performance, whether dedicated training for the targeted task is available or not. Finally, we address generic inference time issues that arise in LLM-based modeling of content selection, assess a generic evaluation metric, and overall propose the utility of our resources and methods for future content selection models. Models and datasets available at https://github.com/shmuelamar/igcs.