Generalized Holographic Reduced Representations

TL;DR

This paper introduces Generalized Holographic Reduced Representations (GHRR), a novel extension of Fourier Holographic Reduced Representations that enhances encoding of complex structures in hyperdimensional computing, balancing symbolic expressiveness with connectionist flexibility.

Contribution

It proposes GHRR, a new HDC framework with a non-commutative binding operation, and demonstrates its theoretical properties and improved empirical performance over existing methods.

Findings

GHRR enables better encoding of complex data structures.

GHRR shows improved decoding accuracy for compositional data.

GHRR has higher memorization capacity compared to FHRR.

Abstract

Deep learning has achieved remarkable success in recent years. Central to its success is its ability to learn representations that preserve task-relevant structure. However, massive energy, compute, and data costs are required to learn general representations. This paper explores Hyperdimensional Computing (HDC), a computationally and data-efficient brain-inspired alternative. HDC acts as a bridge between connectionist and symbolic approaches to artificial intelligence (AI), allowing explicit specification of representational structure as in symbolic approaches while retaining the flexibility of connectionist approaches. However, HDC's simplicity poses challenges for encoding complex compositional structures, especially in its binding operation. To address this, we propose Generalized Holographic Reduced Representations (GHRR), an extension of Fourier Holographic Reduced Representations (FHRR), a specific HDC implementation. GHRR introduces a flexible, non-commutative binding operation, enabling improved encoding of complex data structures while preserving HDC's desirable properties of robustness and transparency. In this work, we introduce the GHRR framework, prove its theoretical properties and its adherence to HDC properties, explore its kernel and binding characteristics, and perform empirical experiments showcasing its flexible non-commutativity, enhanced decoding accuracy for compositional structures, and improved memorization capacity compared to FHRR.