Passage-Mask: A Learnable Regularization Strategy for Retriever-Reader Models

TL;DR

This paper introduces Passage-Mask, a learnable regularization method for retriever-reader models that reduces overfitting to top retrieval passages and encourages reasoning over entire passages, improving performance across NLP tasks.

Contribution

The paper proposes a novel learnable passage masking strategy that enhances retriever-reader models by preventing overfitting and promoting comprehensive passage reasoning.

Findings

Consistently outperforms baseline models on multiple NLP tasks.

Effectively prevents overfitting to top-ranked passages.

General and beneficial across various NLP applications.

Abstract

Retriever-reader models achieve competitive performance across many different NLP tasks such as open question answering and dialogue conversations. In this work, we notice these models easily overfit the top-rank retrieval passages and standard training fails to reason over the entire retrieval passages. We introduce a learnable passage mask mechanism which desensitizes the impact from the top-rank retrieval passages and prevents the model from overfitting. Controlling the gradient variance with fewer mask candidates and selecting the mask candidates with one-shot bi-level optimization, our learnable regularization strategy enforces the answer generation to focus on the entire retrieval passages. Experiments on different tasks across open question answering, dialogue conversation, and fact verification show that our method consistently outperforms its baselines. Extensive experiments and ablation studies demonstrate that our method can be general, effective, and beneficial for many NLP tasks.