SpanDrop: Simple and Effective Counterfactual Learning for Long Sequences

TL;DR

SpanDrop is a data augmentation method that improves model performance on long sequence tasks by identifying true supervision signals through sequence ablation, especially effective with limited data.

Contribution

The paper introduces SpanDrop, a novel simple augmentation technique for counterfactual learning in long sequences, with a beta-Bernoulli variant for diverse augmentation.

Findings

Improves performance on long sequence reasoning tasks.

Effective with both scarce and abundant data.

Enhances model understanding of true supervision signals.

Abstract

Distilling supervision signal from a long sequence to make predictions is a challenging task in machine learning, especially when not all elements in the input sequence contribute equally to the desired output. In this paper, we propose SpanDrop, a simple and effective data augmentation technique that helps models identify the true supervision signal in a long sequence with very few examples. By directly manipulating the input sequence, SpanDrop randomly ablates parts of the sequence at a time and ask the model to perform the same task to emulate counterfactual learning and achieve input attribution. Based on theoretical analysis of its properties, we also propose a variant of SpanDrop based on the beta-Bernoulli distribution, which yields diverse augmented sequences while providing a learning objective that is more consistent with the original dataset. We demonstrate the effectiveness of SpanDrop on a set of carefully designed toy tasks, as well as various natural language processing tasks that require reasoning over long sequences to arrive at the correct answer, and show that it helps models improve performance both when data is scarce and abundant.