MATTER: Memory-Augmented Transformer Using Heterogeneous Knowledge Sources

TL;DR

MATTER is a memory-augmented transformer that efficiently integrates diverse knowledge sources, improving accuracy and speed in knowledge-intensive tasks like question answering, with significantly higher throughput than traditional models.

Contribution

The paper introduces MATTER, a novel model that retrieves and reads from multiple heterogeneous knowledge sources using fixed-length neural memories, enhancing scalability and efficiency.

Findings

Outperforms existing models on QA benchmarks in accuracy and speed.

Achieves 100x inference throughput compared to traditional read-and-retrieve models.

Handles both unstructured and semi-structured knowledge sources effectively.

Abstract

Leveraging external knowledge is crucial for achieving high performance in knowledge-intensive tasks, such as question answering. The retrieve-and-read approach is widely adopted for integrating external knowledge into a language model. However, this approach suffers from increased computational cost and latency due to the long context length, which grows proportionally with the number of retrieved knowledge. Furthermore, existing retrieval-augmented models typically retrieve information from a single type of knowledge source, limiting their scalability to diverse knowledge sources with varying structures. In this work, we introduce an efficient memory-augmented transformer called MATTER, designed to retrieve relevant knowledge from multiple heterogeneous knowledge sources. Specifically, our model retrieves and reads from both unstructured sources (paragraphs) and semi-structured sources (QA pairs) in the form of fixed-length neural memories. We demonstrate that our model outperforms existing efficient retrieval-augmented models on popular QA benchmarks in terms of both accuracy and speed. Furthermore, MATTER achieves competitive results compared to conventional read-and-retrieve models while having 100x throughput during inference.