Efficient Retrieval Optimized Multi-task Learning

TL;DR

This paper introduces a unified Retrieval Optimized Multi-task (ROM) framework that efficiently combines multiple tasks, knowledge retrieval, and question answering, reducing parameters while maintaining or improving performance.

Contribution

The novel ROM framework enables scalable, flexible multi-task learning with shared encoders, reducing memory usage and simplifying architecture compared to prior methods.

Findings

Achieves comparable or better QA performance

Reduces model parameters significantly

Supports flexible encoder changes

Abstract

Recently, there have been significant advances in neural methods for tackling knowledge-intensive tasks such as open domain question answering (QA). These advances are fueled by combining large pre-trained language models with learnable retrieval of documents. Majority of these models use separate encoders for learning query representation, passage representation for the retriever and an additional encoder for the downstream task. Using separate encoders for each stage/task occupies a lot of memory and makes it difficult to scale to a large number of tasks. In this paper, we propose a novel Retrieval Optimized Multi-task (ROM) framework for jointly training self-supervised tasks, knowledge retrieval, and extractive question answering. Our ROM approach presents a unified and generalizable framework that enables scaling efficiently to multiple tasks, varying levels of supervision, and optimization choices such as different learning schedules without changing the model architecture. It also provides the flexibility of changing the encoders without changing the architecture of the system. Using our framework, we achieve comparable or better performance than recent methods on QA, while drastically reducing the number of parameters.