Reference-based Weak Supervision for Answer Sentence Selection using Web Data

TL;DR

This paper introduces a fully automatic weak supervision method called RWS that leverages web data to improve answer sentence selection models, achieving state-of-the-art results on WikiQA.

Contribution

The paper presents RWS, a novel large-scale data collection pipeline that enhances AS2 models by using reference-based weak supervision from web data.

Findings

RWS improves TANDA's performance on WikiQA.

Achieved state-of-the-art P@1 and MAP scores.

Demonstrated robustness of weak supervision approach.

Abstract

Answer sentence selection (AS2) modeling requires annotated data, i.e., hand-labeled question-answer pairs. We present a strategy to collect weakly supervised answers for a question based on its reference to improve AS2 modeling. Specifically, we introduce Reference-based Weak Supervision (RWS), a fully automatic large-scale data pipeline that harvests high-quality weakly-supervised answers from abundant Web data requiring only a question-reference pair as input. We study the efficacy and robustness of RWS in the setting of TANDA, a recent state-of-the-art fine-tuning approach specialized for AS2. Our experiments indicate that the produced data consistently bolsters TANDA. We achieve the state of the art in terms of P@1, 90.1%, and MAP, 92.9%, on WikiQA.