Unifying distillation and privileged information

TL;DR

This paper introduces generalized distillation, unifying two machine learning techniques to improve learning from multiple sources, with theoretical insights and applications across various learning scenarios.

Contribution

It presents a unified framework combining distillation and privileged information, extending to unsupervised, semi-supervised, and multitask learning, with theoretical and empirical validation.

Findings

Effective in synthetic and real-world data

Unifies distillation and privileged information techniques

Applicable to multiple learning scenarios

Abstract

Distillation (Hinton et al., 2015) and privileged information (Vapnik & Izmailov, 2015) are two techniques that enable machines to learn from other machines. This paper unifies these two techniques into generalized distillation, a framework to learn from multiple machines and data representations. We provide theoretical and causal insight about the inner workings of generalized distillation, extend it to unsupervised, semisupervised and multitask learning scenarios, and illustrate its efficacy on a variety of numerical simulations on both synthetic and real-world data.