Signal to Noise Ratio Loss Function

TL;DR

This paper introduces a novel loss function for classification that leverages bounds on probabilities to optimize the signal-to-noise ratio, improving true positive rates.

Contribution

It derives new bounds for probability estimates and formulates a loss function based on signal-to-noise ratio, addressing overlooked information in cross entropy loss.

Findings

Improved true positive rates in classification tasks.

The proposed loss function enhances model robustness.

Empirical results show better performance compared to traditional methods.

Abstract

This work proposes a new loss function targeting classification problems, utilizing a source of information overlooked by cross entropy loss. First, we derive a series of the tightest upper and lower bounds for the probability of a random variable in a given interval. Second, a lower bound is proposed for the probability of a true positive for a parametric classification problem, where the form of probability density function (pdf) of data is given. A closed form for finding the optimal function of unknowns is derived to maximize the probability of true positives. Finally, for the case that the pdf of data is unknown, we apply the proposed boundaries to find the lower bound of the probability of true positives and upper bound of the probability of false positives and optimize them using a loss function which is given by combining the boundaries. We demonstrate that the resultant loss function is a function of the signal to noise ratio both within and across logits. We empirically evaluate our proposals to show their benefit for classification problems.