Beyond the Class Subspace: Teacher-Guided Training for Reliable Out-of-Distribution Detection in Single-Domain Models

TL;DR

This paper identifies a geometric failure mode in single-domain models called Domain-Sensitivity Collapse, and proposes Teacher-Guided Training to improve out-of-distribution detection without inference overhead.

Contribution

It introduces TGT, a novel training method that distills multi-domain knowledge into single-domain models to enhance OOD detection capabilities.

Findings

TGT significantly reduces far-OOD false positive rates across benchmarks.

TGT maintains or improves in-domain accuracy.

Theoretical analysis explains the failure mode and how TGT addresses it.

Abstract

Out-of-distribution (OOD) detection methods perform well on multi-domain benchmarks, yet many practical systems are trained on single-domain data. We show that this regime induces a geometric failure mode, Domain-Sensitivity Collapse (DSC): supervised training compresses features into a low-rank class subspace and suppresses directions that carry domain-shift signal. We provide theory showing that, under DSC, distance- and logit-based OOD scores lose sensitivity to domain shift. We then introduce Teacher-Guided Training (TGT), which distills class-suppressed residual structure from a frozen multi-domain teacher (DINOv2) into the student during training. The teacher and auxiliary head are discarded after training, adding no inference overhead. Across eight single-domain benchmarks, TGT yields large far-OOD FPR@95 reductions for distance-based scorers: MDS improves by 11.61 pp, ViM by 10.78 pp, and kNN by 12.87 pp (ResNet-50 average), while maintaining or slightly improving in-domain OOD and classification accuracy.