RESOLVE: Relational Reasoning with Symbolic and Object-Level Features Using Vector Symbolic Processing

TL;DR

RESOLVE is a neuro-vector symbolic architecture that enhances relational reasoning in transformer models by combining object and relational features in high-dimensional spaces, leading to improved accuracy and efficiency.

Contribution

It introduces a novel high-dimensional attention mechanism and a vector symbolic processing approach for better relational and object-level reasoning.

Findings

Achieves higher accuracy in relational reasoning tasks

Offers lower compute latency and memory efficiency

Demonstrates improved generalizability over state-of-the-art methods

Abstract

Modern transformer-based encoder-decoder architectures struggle with reasoning tasks due to their inability to effectively extract relational information between input objects (data/tokens). Recent work introduced the Abstractor module, embedded between transformer layers, to address this gap. However, the Abstractor layer while excelling at capturing relational information (pure relational reasoning), faces challenges in tasks that require both object and relational-level reasoning (partial relational reasoning). To address this, we propose RESOLVE, a neuro-vector symbolic architecture that combines object-level features with relational representations in high-dimensional spaces, using fast and efficient operations such as bundling (summation) and binding (Hadamard product) allowing both object-level features and relational representations to coexist within the same structure without interfering with one another. RESOLVE is driven by a novel attention mechanism that operates in a bipolar high dimensional space, allowing fast attention score computation compared to the state-of-the-art. By leveraging this design, the model achieves both low compute latency and memory efficiency. RESOLVE also offers better generalizability while achieving higher accuracy in purely relational reasoning tasks such as sorting as well as partial relational reasoning tasks such as math problem-solving compared to state-of-the-art methods.