ResBit: Residual Bit Vector for Categorical Values

TL;DR

ResBit introduces a dense, low-dimensional representation for categorical data that accelerates training and maintains performance, addressing the limitations of one-hot vectors in high-cardinality tabular data generation.

Contribution

The paper proposes Residual Bit Vectors (ResBit), a novel method that overcomes the limitations of one-hot vectors and analog bits for efficient high-cardinality categorical data representation.

Findings

ResBit accelerates training in tabular data generation.

ResBit maintains performance comparable to traditional methods.

High-cardinality data challenges are mitigated by ResBit's low-dimensional encoding.

Abstract

One-hot vectors, a common method for representing discrete/categorical data, in machine learning are widely used because of their simplicity and intuitiveness. However, one-hot vectors suffer from a linear increase in dimensionality, posing computational and memory challenges, especially when dealing with datasets containing numerous categories. In this paper, we focus on tabular data generation, and reveal the multinomial diffusion faces the mode collapse phenomenon when the cardinality is high. Moreover, due to the limitations of one-hot vectors, the training phase takes time longer in such a situation. To address these issues, we propose Residual Bit Vectors (ResBit), a technique for densely representing categorical data. ResBit is an extension of analog bits and overcomes limitations of analog bits when applied to tabular data generation. Our experiments demonstrate that ResBit not only accelerates training but also maintains performance when compared with the situations before applying ResBit. Furthermore, our results indicate that many existing methods struggle with high-cardinality data, underscoring the need for lower-dimensional representations, such as ResBit and latent vectors.