How Optimal is Greedy Decoding for Extractive Question Answering?

TL;DR

This paper evaluates the effectiveness of greedy decoding in extractive question answering, showing that with minimal training, it closely approximates the optimal span-finding algorithm, especially when models are biased towards extractiveness.

Contribution

The study introduces the exact-extract algorithm for optimal span decoding and compares it with greedy decoding, revealing how training influences their relative performance.

Findings

Exact-extract outperforms greedy decoding in zero-shot settings.

Greedy decoding improves rapidly with few training examples.

Self-supervised training biases models towards extractive answers.

Abstract

Fine-tuned language models use greedy decoding to answer reading comprehension questions with relative success. However, this approach does not ensure that the answer is a span in the given passage, nor does it guarantee that it is the most probable one. Does greedy decoding actually perform worse than an algorithm that does adhere to these properties? To study the performance and optimality of greedy decoding, we present exact-extract, a decoding algorithm that efficiently finds the most probable answer span in the context. We compare the performance of T5 with both decoding algorithms on zero-shot and few-shot extractive question answering. When no training examples are available, exact-extract significantly outperforms greedy decoding. However, greedy decoding quickly converges towards the performance of exact-extract with the introduction of a few training examples, becoming more extractive and increasingly likelier to generate the most probable span as the training set grows. We also show that self-supervised training can bias the model towards extractive behavior, increasing performance in the zero-shot setting without resorting to annotated examples. Overall, our results suggest that pretrained language models are so good at adapting to extractive question answering, that it is often enough to fine-tune on a small training set for the greedy algorithm to emulate the optimal decoding strategy.