Learning to Generate Synthetic Training Data using Gradient Matching and Implicit Differentiation

TL;DR

This paper introduces new data distillation techniques using gradient matching and implicit differentiation, which improve training efficiency and model performance with less data in deep learning tasks.

Contribution

The paper proposes novel data distillation methods based on generative teaching networks, gradient matching, and the Implicit Function Theorem, enhancing efficiency and effectiveness.

Findings

Methods are computationally more efficient than previous approaches.

Distilled data improves model performance on MNIST.

Techniques enable training with less data without sacrificing accuracy.

Abstract

Using huge training datasets can be costly and inconvenient. This article explores various data distillation techniques that can reduce the amount of data required to successfully train deep networks. Inspired by recent ideas, we suggest new data distillation techniques based on generative teaching networks, gradient matching, and the Implicit Function Theorem. Experiments with the MNIST image classification problem show that the new methods are computationally more efficient than previous ones and allow to increase the performance of models trained on distilled data.