Derandomizing Multi-Distribution Learning

TL;DR

This paper investigates whether algorithms for multi-distribution learning can be converted from randomized to deterministic predictors, revealing computational hardness but also identifying conditions for successful derandomization.

Contribution

It demonstrates the computational hardness of derandomizing multi-distribution learning and provides a structural condition for efficient derandomization under certain circumstances.

Findings

Derandomization of multi-distribution learning is computationally hard.

A structural condition enables converting randomized predictors into deterministic ones.

The research bridges the gap between randomized and deterministic multi-distribution learning algorithms.

Abstract

Multi-distribution or collaborative learning involves learning a single predictor that works well across multiple data distributions, using samples from each during training. Recent research on multi-distribution learning, focusing on binary loss and finite VC dimension classes, has shown near-optimal sample complexity that is achieved with oracle efficient algorithms. That is, these algorithms are computationally efficient given an efficient ERM for the class. Unlike in classical PAC learning, where the optimal sample complexity is achieved with deterministic predictors, current multi-distribution learning algorithms output randomized predictors. This raises the question: can these algorithms be derandomized to produce a deterministic predictor for multiple distributions? Through a reduction to discrepancy minimization, we show that derandomizing multi-distribution learning is computationally hard, even when ERM is computationally efficient. On the positive side, we identify a structural condition enabling an efficient black-box reduction, converting existing randomized multi-distribution predictors into deterministic ones.