Adaptive Communication Bounds for Distributed Online Learning

TL;DR

This paper investigates how to optimize communication in distributed online learning by balancing information exchange with learning performance, proposing criteria that adapt to problem difficulty.

Contribution

It introduces formal criteria for adaptive communication bounds in distributed online learning, aligning communication costs with problem hardness.

Findings

Criteria hold for a simplified version of a previous protocol

Communication bounds scale with the hardness of the prediction problem

Inherent trade-offs between communication and learning performance are characterized

Abstract

We consider distributed online learning protocols that control the exchange of information between local learners in a round-based learning scenario. The learning performance of such a protocol is intuitively optimal if approximately the same loss is incurred as in a hypothetical serial setting. If a protocol accomplishes this, it is inherently impossible to achieve a strong communication bound at the same time. In the worst case, every input is essential for the learning performance, even for the serial setting, and thus needs to be exchanged between the local learners. However, it is reasonable to demand a bound that scales well with the hardness of the serialized prediction problem, as measured by the loss received by a serial online learning algorithm. We provide formal criteria based on this intuition and show that they hold for a simplified version of a previously published protocol.