Differentiating Choices via Commonality for Multiple-Choice Question Answering

TL;DR

This paper introduces DCQA, a novel model for multiple-choice question answering that differentiates choices by identifying and removing their commonalities, thereby improving accuracy in challenging, semantically similar options.

Contribution

The paper proposes a new approach that leverages token-level attention to distinguish subtle differences among choices by eliminating their shared commonalities.

Findings

DCQA outperforms baseline models across five benchmarks.

The model effectively captures nuanced differences among choices.

Case studies show improved focus on differentiating features.

Abstract

Multiple-choice question answering (MCQA) becomes particularly challenging when all choices are relevant to the question and are semantically similar. Yet this setting of MCQA can potentially provide valuable clues for choosing the right answer. Existing models often rank each choice separately, overlooking the context provided by other choices. Specifically, they fail to leverage the semantic commonalities and nuances among the choices for reasoning. In this paper, we propose a novel MCQA model by differentiating choices through identifying and eliminating their commonality, called DCQA. Our model captures token-level attention of each choice to the question, and separates tokens of the question attended to by all the choices (i.e., commonalities) from those by individual choices (i.e., nuances). Using the nuances as refined contexts for the choices, our model can effectively differentiate choices with subtle differences and provide justifications for choosing the correct answer. We conduct comprehensive experiments across five commonly used MCQA benchmarks, demonstrating that DCQA consistently outperforms baseline models. Furthermore, our case study illustrates the effectiveness of the approach in directing the attention of the model to more differentiating features.