Cost-Aware Learning for Improved Identifiability with Multiple Experiments

TL;DR

This paper investigates how a cost-aware learning approach across multiple experiments can enhance model identifiability and reduce the gap between training and generalization errors, considering budget constraints.

Contribution

It introduces a framework analyzing sample complexity with multiple experiments and costs, demonstrating improved identifiability and error bounds using Rademacher complexity.

Findings

Learning from multiple experiments improves identifiability.

The generalization gap scales as O(C^{-1/2}) with total cost C.

Applicable to linear models, neural networks, and kernel methods.

Abstract

We analyze the sample complexity of learning from multiple experiments where the experimenter has a total budget for obtaining samples. In this problem, the learner should choose a hypothesis that performs well with respect to multiple experiments, and their related data distributions. Each collected sample is associated with a cost which depends on the particular experiments. In our setup, a learner performs $m$ experiments, while incurring a total cost $C$. We first show that learning from multiple experiments allows to improve identifiability. Additionally, by using a Rademacher complexity approach, we show that the gap between the training and generalization error is $O(C^{-1/2})$. We also provide some examples for linear prediction, two-layer neural networks and kernel methods.