Adapting Models to Signal Degradation using Distillation

TL;DR

This paper introduces a method for domain adaptation using knowledge distillation with synthetically generated aligned data, improving recognition on degraded or low-quality images across various tasks.

Contribution

The authors propose a novel distillation-based domain adaptation technique that leverages synthetically generated aligned data, enabling models to adapt to signal degradation without requiring real aligned datasets.

Findings

Improves recognition accuracy on low-resolution and degraded images.

Outperforms strong baseline methods for domain adaptation.

Provides insights through visualizations and literature comparison.

Abstract

Model compression and knowledge distillation have been successfully applied for cross-architecture and cross-domain transfer learning. However, a key requirement is that training examples are in correspondence across the domains. We show that in many scenarios of practical importance such aligned data can be synthetically generated using computer graphics pipelines allowing domain adaptation through distillation. We apply this technique to learn models for recognizing low-resolution images using labeled high-resolution images, non-localized objects using labeled localized objects, line-drawings using labeled color images, etc. Experiments on various fine-grained recognition datasets demonstrate that the technique improves recognition performance on the low-quality data and beats strong baselines for domain adaptation. Finally, we present insights into workings of the technique through visualizations and relating it to existing literature.