Leveraging Large Language Models for Multiple Choice Question Answering

TL;DR

This paper explores how large language models can better answer multiple choice questions by explicitly comparing options, introducing the concept of multiple choice symbol binding (MCSB), and demonstrating improved performance across diverse datasets.

Contribution

The paper introduces the MCSB ability as crucial for effective MCQA and shows that models with high MCSB significantly improve accuracy using a natural prompting approach.

Findings

Models with high MCSB outperform traditional methods.

Natural prompting reduces computational costs.

Performance gap with SOTA is largely closed.

Abstract

While large language models (LLMs) like GPT-3 have achieved impressive results on multiple choice question answering (MCQA) tasks in the zero, one, and few-shot settings, they generally lag behind the MCQA state of the art (SOTA). MCQA tasks have traditionally been presented to LLMs like cloze tasks. An LLM is conditioned on a question (without the associated answer options) and its chosen option is the one assigned the highest probability after normalization (for length, etc.). A more natural prompting approach is to present the question and answer options to the LLM jointly and have it output the symbol (e.g., "A") associated with its chosen answer option. This approach allows the model to explicitly compare answer options, reduces computational costs, and mitigates the effects of tokenization scheme and answer option representations on answer selection. For the natural approach to be effective, the LLM it is used with must be able to associate answer options with the symbols that represent them. The LLM needs what we term multiple choice symbol binding (MCSB) ability. This ability varies greatly by model. We show that a model with high MCSB ability performs much better with the natural approach than with the traditional approach across 20 diverse datasets and largely closes the gap with the SOTA, suggesting that the MCQA ability of LLMs has been previously underestimated.