A Unified Approach to Coreset Learning

TL;DR

This paper introduces a learning-based, generic method for constructing coresets that approximate average loss over queries, enabling practical, efficient data summarization for various machine learning tasks, including deep network pruning.

Contribution

It proposes a new, relaxed coreset definition and a learning paradigm for coreset construction, applicable across different problems and capable of compressing entire deep networks.

Findings

Learned coresets perform comparably or better than traditional algorithms.

The approach provides the first coreset for full deep network compression.

Formal guarantees support the effectiveness of the method.

Abstract

Coreset of a given dataset and loss function is usually a small weighed set that approximates this loss for every query from a given set of queries. Coresets have shown to be very useful in many applications. However, coresets construction is done in a problem dependent manner and it could take years to design and prove the correctness of a coreset for a specific family of queries. This could limit coresets use in practical applications. Moreover, small coresets provably do not exist for many problems. To address these limitations, we propose a generic, learning-based algorithm for construction of coresets. Our approach offers a new definition of coreset, which is a natural relaxation of the standard definition and aims at approximating the \emph{average} loss of the original data over the queries. This allows us to use a learning paradigm to compute a small coreset of a given set of inputs with respect to a given loss function using a training set of queries. We derive formal guarantees for the proposed approach. Experimental evaluation on deep networks and classic machine learning problems show that our learned coresets yield comparable or even better results than the existing algorithms with worst-case theoretical guarantees (that may be too pessimistic in practice). Furthermore, our approach applied to deep network pruning provides the first coreset for a full deep network, i.e., compresses all the network at once, and not layer by layer or similar divide-and-conquer methods.