TWEAC: Transformer with Extendable QA Agent Classifiers

TL;DR

TWEAC is a transformer-based system that efficiently identifies suitable QA agents for diverse questions, achieving high accuracy and scalability in selecting from many specialized agents.

Contribution

The paper introduces TWEAC, a novel transformer-based approach for extendable QA agent classification, improving selection accuracy and scalability over existing methods.

Findings

TWEAC achieves 94% accuracy in agent selection.

TWEAC scales effectively to over 100 agents with limited data per agent.

Supervised and unsupervised approaches are evaluated for agent identification.

Abstract

Question answering systems should help users to access knowledge on a broad range of topics and to answer a wide array of different questions. Most systems fall short of this expectation as they are only specialized in one particular setting, e.g., answering factual questions with Wikipedia data. To overcome this limitation, we propose composing multiple QA agents within a meta-QA system. We argue that there exist a wide range of specialized QA agents in literature. Thus, we address the central research question of how to effectively and efficiently identify suitable QA agents for any given question. We study both supervised and unsupervised approaches to address this challenge, showing that TWEAC -- Transformer with Extendable Agent Classifiers -- achieves the best performance overall with 94% accuracy. We provide extensive insights on the scalability of TWEAC, demonstrating that it scales robustly to over 100 QA agents with each providing just 1000 examples of questions they can answer. Our code and data is available: https://github.com/UKPLab/TWEAC-qa-agent-selection