Towards Vector Optimization on Low-Dimensional Vector Symbolic Architecture

TL;DR

This paper explores low-dimensional vector optimization in Vector Symbolic Architecture, employing gradient-based methods, batch normalization, and knowledge distillation to improve accuracy and efficiency in machine learning applications.

Contribution

It introduces a novel LDC optimization approach for VSA that reduces vector dimensions by ~100 times while maintaining accuracy, highlighting the roles of BN and KD.

Findings

LDC significantly reduces vector dimensions with maintained accuracy

Batch normalization improves training stability without extra inference cost

Knowledge distillation enhances inference confidence

Abstract

Vector Symbolic Architecture (VSA) is emerging in machine learning due to its efficiency, but they are hindered by issues of hyperdimensionality and accuracy. As a promising mitigation, the Low-Dimensional Computing (LDC) method significantly reduces the vector dimension by ~100 times while maintaining accuracy, by employing a gradient-based optimization. Despite its potential, LDC optimization for VSA is still underexplored. Our investigation into vector updates underscores the importance of stable, adaptive dynamics in LDC training. We also reveal the overlooked yet critical roles of batch normalization (BN) and knowledge distillation (KD) in standard approaches. Besides the accuracy boost, BN does not add computational overhead during inference, and KD significantly enhances inference confidence. Through extensive experiments and ablation studies across multiple benchmarks, we provide a thorough evaluation of our approach and extend the interpretability of binary neural network optimization similar to LDC, previously unaddressed in BNN literature.