Pixel-Wise Contrastive Distillation
Junqiang Huang, Zichao Guo

TL;DR
This paper introduces Pixel-Wise Contrastive Distillation (PCD), a self-supervised framework for dense prediction tasks that improves knowledge transfer at the pixel level using novel reshaping and attention mechanisms.
Contribution
The paper proposes a new pixel-level distillation method with a SpatialAdaptor and multi-head self-attention, enhancing dense prediction performance.
Findings
Outperforms previous self-supervised distillation methods on dense tasks
ResNet-18-FPN distilled by PCD achieves 37.4 AP_bbox on COCO
ResNet-18-FPN distilled by PCD achieves 34.0 AP_mask on COCO
Abstract
We present a simple but effective pixel-level self-supervised distillation framework friendly to dense prediction tasks. Our method, called Pixel-Wise Contrastive Distillation (PCD), distills knowledge by attracting the corresponding pixels from student's and teacher's output feature maps. PCD includes a novel design called SpatialAdaptor which ``reshapes'' a part of the teacher network while preserving the distribution of its output features. Our ablation experiments suggest that this reshaping behavior enables more informative pixel-to-pixel distillation. Moreover, we utilize a plug-in multi-head self-attention module that explicitly relates the pixels of student's feature maps to enhance the effective receptive field, leading to a more competitive student. PCD \textbf{outperforms} previous self-supervised distillation methods on various dense prediction tasks. A backbone of…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Neural Network Applications · Domain Adaptation and Few-Shot Learning · Human Pose and Action Recognition
MethodsConvolution · Softmax · RoIAlign · Region Proposal Network · Mask R-CNN
