Torus embeddings

TL;DR

This paper introduces torus embeddings as a new data representation method that aligns with the integer-based topology of digital computers, offering stable training and efficient quantisation for deep learning models.

Contribution

It demonstrates how to adapt deep learning frameworks to create torus-structured embeddings, providing an alternative to hyperspherical embeddings with comparable performance.

Findings

Torus embeddings can be integrated into existing deep learning frameworks.

Normalisation-based strategies yield stable and effective training.

Torus embeddings maintain quantisation properties suitable for efficient implementation.

Abstract

Many data representations are vectors of continuous values. In particular, deep learning embeddings are data-driven representations, typically either unconstrained in Euclidean space, or constrained to a hypersphere. These may also be translated into integer representations (quantised) for efficient large-scale use. However, the fundamental (and most efficient) numeric representation in the overwhelming majority of existing computers is integers with overflow -- and vectors of these integers do not correspond to either of these spaces, but instead to the topology of a (hyper)torus. This mismatch can lead to wasted representation capacity. Here we show that common deep learning frameworks can be adapted, quite simply, to create representations with inherent toroidal topology. We investigate two alternative strategies, demonstrating that a normalisation-based strategy leads to training with desirable stability and performance properties, comparable to a standard hyperspherical L2 normalisation. We also demonstrate that a torus embedding maintains desirable quantisation properties. The torus embedding does not outperform hypersphere embeddings in general, but is comparable, and opens the possibility to train deep embeddings which have an extremely simple pathway to efficient `TinyML' embedded implementation.