Hyperbolic Binary Neural Network

TL;DR

This paper introduces Hyperbolic Binary Neural Networks (HBNN) that leverage hyperbolic geometry and a novel exponential parametrization to optimize binary neural networks more effectively, achieving superior performance on standard datasets.

Contribution

The paper proposes a novel hyperbolic geometry framework and the Exponential Parametrization Cluster method for optimizing binary neural networks, improving their performance.

Findings

HBNN outperforms state-of-the-art BNN methods on CIFAR and ImageNet datasets.

The EPC method increases weight flip probability, enhancing information gain.

Experimental results demonstrate the effectiveness of hyperbolic optimization in BNNs.

Abstract

Binary Neural Network (BNN) converts full-precision weights and activations into their extreme 1-bit counterparts, making it particularly suitable for deployment on lightweight mobile devices. While binary neural networks are typically formulated as a constrained optimization problem and optimized in the binarized space, general neural networks are formulated as an unconstrained optimization problem and optimized in the continuous space. This paper introduces the Hyperbolic Binary Neural Network (HBNN) by leveraging the framework of hyperbolic geometry to optimize the constrained problem. Specifically, we transform the constrained problem in hyperbolic space into an unconstrained one in Euclidean space using the Riemannian exponential map. On the other hand, we also propose the Exponential Parametrization Cluster (EPC) method, which, compared to the Riemannian exponential map, shrinks the segment domain based on a diffeomorphism. This approach increases the probability of weight flips, thereby maximizing the information gain in BNNs. Experimental results on CIFAR10, CIFAR100, and ImageNet classification datasets with VGGsmall, ResNet18, and ResNet34 models illustrate the superior performance of our HBNN over state-of-the-art methods.