Leveraging Intermediate Representations for Better Out-of-Distribution Detection

TL;DR

This paper explores how intermediate neural network layers contain valuable information for detecting out-of-distribution samples, proposing a regularization method to enhance OoD detection performance.

Contribution

It introduces a novel approach that leverages and regularizes intermediate layer representations using an energy-based contrastive loss for improved OoD detection.

Findings

Intermediate layers have strong discriminative power for OoD detection.

Regularizing intermediate layers improves OoD detection accuracy.

Aggregating multiple layer responses enhances detection performance.

Abstract

In real-world applications, machine learning models must reliably detect Out-of-Distribution (OoD) samples to prevent unsafe decisions. Current OoD detection methods often rely on analyzing the logits or the embeddings of the penultimate layer of a neural network. However, little work has been conducted on the exploitation of the rich information encoded in intermediate layers. To address this, we analyze the discriminative power of intermediate layers and show that they can positively be used for OoD detection. Therefore, we propose to regularize intermediate layers with an energy-based contrastive loss, and by grouping multiple layers in a single aggregated response. We demonstrate that intermediate layer activations improves OoD detection performance by running a comprehensive evaluation across multiple datasets.