On Distributed Learning with Constant Communication Bits

TL;DR

This paper investigates distributed hypothesis testing under fixed communication constraints, developing optimal coding schemes and error exponent bounds for various bit-limited scenarios, with practical implications.

Contribution

It introduces a geometric distribution space approach to characterize error exponents and proposes optimal coding strategies for fixed-bit distributed hypothesis testing.

Findings

Optimal error exponents identified for different communication constraints.

Coding schemes based on empirical distribution encoding are effective.

Numerical examples illustrate the theoretical bounds and strategies.

Abstract

In this paper, we study a distributed learning problem constrained by constant communication bits. Specifically, we consider the distributed hypothesis testing (DHT) problem where two distributed nodes are constrained to transmit a constant number of bits to a central decoder. In such cases, we show that in order to achieve the optimal error exponents, it suffices to consider the empirical distributions of observed data sequences and encode them to the transmission bits. With such a coding strategy, we develop a geometric approach in the distribution spaces and establish an inner bound of error exponent regions. In particular, we show the optimal achievable error exponents and coding schemes for the following cases: (i) both nodes can transmit $\log_23$ bits; (ii) one of the nodes can transmit $1$ bit, and the other node is not constrained; (iii) the joint distribution of the nodes are conditionally independent given one hypothesis. Furthermore, we provide several numerical examples for illustrating the theoretical results. Our results provide theoretical guidance for designing practical distributed learning rules, and the developed approach also reveals new potentials for establishing error exponents for DHT with more general communication constraints.