VRA: Variational Rectified Activation for Out-of-distribution Detection

TL;DR

This paper introduces VRA, a novel activation modification technique based on variational methods, to improve out-of-distribution detection by optimally suppressing and amplifying neuron activations, outperforming existing methods.

Contribution

Proposes VRA, a variational method-driven activation function that enhances OOD detection by optimally adjusting neuron activations, surpassing prior techniques like ReAct.

Findings

VRA outperforms existing post-hoc strategies on benchmark datasets.

VRA is compatible with various scoring functions and network architectures.

Experimental results validate the effectiveness of VRA in OOD detection.

Abstract

Out-of-distribution (OOD) detection is critical to building reliable machine learning systems in the open world. Researchers have proposed various strategies to reduce model overconfidence on OOD data. Among them, ReAct is a typical and effective technique to deal with model overconfidence, which truncates high activations to increase the gap between in-distribution and OOD. Despite its promising results, is this technique the best choice for widening the gap? To answer this question, we leverage the variational method to find the optimal operation and verify the necessity of suppressing abnormally low and high activations and amplifying intermediate activations in OOD detection, rather than focusing only on high activations like ReAct. This motivates us to propose a novel technique called ``Variational Rectified Activation (VRA)'', which simulates these suppression and amplification operations using piecewise functions. Experimental results on multiple benchmark datasets demonstrate that our method outperforms existing post-hoc strategies. Meanwhile, VRA is compatible with different scoring functions and network architectures. \textcolor[rgb]{0.93,0.0,0.47}{Our code can be found in Supplementary Material}.