HyBNN and FedHyBNN: (Federated) Hybrid Binary Neural Networks

TL;DR

This paper introduces HyBNN, a hybrid neural network architecture combining full-precision autoencoders with binary neural networks to improve accuracy while maintaining efficiency, and extends it to federated learning with FedHyBNN.

Contribution

The paper proposes a novel hybrid architecture, HyBNN, that reduces accuracy loss in binary neural networks using a full-precision autoencoder, and introduces FedHyBNN for efficient federated learning.

Findings

HyBNN significantly outperforms vanilla binary neural networks.

FedHyBNN achieves the same accuracy as non-federated HyBNN.

The approach combines deep neural network accuracy with binary neural network efficiency.

Abstract

Binary Neural Networks (BNNs), neural networks with weights and activations constrained to -1(0) and +1, are an alternative to deep neural networks which offer faster training, lower memory consumption and lightweight models, ideal for use in resource constrained devices while being able to utilize the architecture of their deep neural network counterpart. However, the input binarization step used in BNNs causes a severe accuracy loss. In this paper, we introduce a novel hybrid neural network architecture, Hybrid Binary Neural Network (HyBNN), consisting of a task-independent, general, full-precision variational autoencoder with a binary latent space and a task specific binary neural network that is able to greatly limit the accuracy loss due to input binarization by using the full precision variational autoencoder as a feature extractor. We use it to combine the state-of-the-art accuracy of deep neural networks with the much faster training time, quicker test-time inference and power efficiency of binary neural networks. We show that our proposed system is able to very significantly outperform a vanilla binary neural network with input binarization. We also introduce FedHyBNN, a highly communication efficient federated counterpart to HyBNN and demonstrate that it is able to reach the same accuracy as its non-federated equivalent. We make our source code, experimental parameters and models available at: https://anonymous.4open.science/r/HyBNN.