General-Purpose Question-Answering with Macaw

TL;DR

Macaw is a versatile, open-source question-answering system built on T5 that demonstrates strong zero-shot performance across diverse topics, outperforming larger models like GPT-3 on specific benchmarks and offering flexible input-output configurations.

Contribution

The paper introduces Macaw, a novel, general-purpose QA system that leverages input-output permutations and achieves competitive zero-shot performance with a smaller model size.

Findings

Macaw outperforms GPT-3 by over 10% on Challenge300.

Macaw demonstrates strong zero-shot performance across various topics.

The system offers flexible input-output configurations for diverse question types.

Abstract

Despite the successes of pretrained language models, there are still few high-quality, general-purpose QA systems that are freely available. In response, we present Macaw, a versatile, generative question-answering (QA) system that we are making available to the community. Macaw is built on UnifiedQA, itself built on T5, and exhibits strong performance, zero-shot, on a wide variety of topics, including outperforming GPT-3 by over 10% (absolute) on Challenge300, a suite of 300 challenge questions, despite being an order of magnitude smaller (11 billion vs. 175 billion parameters). In addition, Macaw allows different permutations ("angles") of its inputs and outputs to be used, for example Macaw can take a question and produce an answer; or take an answer and produce a question; or take an answer and question, and produce multiple-choice options. We describe the system, and illustrate a variety of question types where it produces surprisingly good answers, well outside the training setup. We also identify question classes where it still appears to struggle, offering insights into the limitations of pretrained language models. Macaw is freely available, and we hope that it proves useful to the community. Macaw is available at https://github.com/allenai/macaw