Distilling the Undistillable: Learning from a Nasty Teacher

TL;DR

This paper investigates the confidentiality of Nasty Teacher models in knowledge distillation, proposing methods to extract information despite defenses, and introduces improved techniques to enhance learning from such models.

Contribution

It develops novel methodologies HTC and SCM to effectively steal information from Nasty Teacher models, surpassing previous limitations and strengthening understanding of model confidentiality.

Findings

HTC and SCM increase learning from Nasty Teacher by up to 68.63%.

The proposed methods outperform existing approaches in extracting information.

An improved defense method is also proposed based on insights from the attack strategies.

Abstract

The inadvertent stealing of private/sensitive information using Knowledge Distillation (KD) has been getting significant attention recently and has guided subsequent defense efforts considering its critical nature. Recent work Nasty Teacher proposed to develop teachers which can not be distilled or imitated by models attacking it. However, the promise of confidentiality offered by a nasty teacher is not well studied, and as a further step to strengthen against such loopholes, we attempt to bypass its defense and steal (or extract) information in its presence successfully. Specifically, we analyze Nasty Teacher from two different directions and subsequently leverage them carefully to develop simple yet efficient methodologies, named as HTC and SCM, which increase the learning from Nasty Teacher by upto 68.63% on standard datasets. Additionally, we also explore an improvised defense method based on our insights of stealing. Our detailed set of experiments and ablations on diverse models/settings demonstrate the efficacy of our approach.