Toward Robust Semi-supervised Regression via Dual-stream Knowledge Distillation

TL;DR

This paper proposes a dual-stream knowledge distillation framework for semi-supervised regression that effectively leverages unlabeled data and improves prediction robustness by distilling continuous and distributional knowledge.

Contribution

It introduces a novel dual-stream knowledge distillation approach with distribution alignment specifically designed for semi-supervised regression tasks.

Findings

Improves regression accuracy with limited labeled data.

Enhances robustness to noisy pseudo-labels.

Outperforms existing semi-supervised regression methods.

Abstract

Semi-supervised regression (SSR), which aims to predict continuous scores of samples while reducing reliance on a large amount of labeled data, has recently received considerable attention across various applications, including computer vision, natural language processing, and audio and medical analysis. Existing SSR methods typically train models on scarce labeled data by introducing constraint-based regularization or ordinal ranking to reduce overfitting. However, these approaches fail to fully exploit the abundance of unlabeled samples. While consistency-driven pseudo-labeling methods attempt to incorporate unlabeled data, they are highly sensitive to pseudo-label quality and noisy predictions. To address these challenges, we introduce a Dual-stream Knowledge Distillation framework (DKD), which is specially designed for the SSR task to distill knowledge from both continuous-valued knowledge and distribution information, better preserving regression magnitude information and improving sample efficiency. Specifically, in DKD, the teacher is optimized solely with ground-truth labels for label distribution estimation, while the student learns from a mixture of real labels and teacher-generated pseudo targets on unlabeled data. The distillation design ensures the effective supervision transfer, allowing the student to leverage pseudo labels more robustly. Then, we introduce an advanced Decoupled Distribution Alignment (DDA) to align the target class and non-target class between teacher and student on the distribution, enhancing the student's capacity to mitigate noise in pseudo-label supervision and learn a more well-calibrated regression predictor.