DeepTopPush: Simple and Scalable Method for Accuracy at the Top

TL;DR

DeepTopPush is a scalable deep learning method designed to optimize accuracy at the top, effectively handling non-decomposable loss functions for applications like drug discovery and malware detection.

Contribution

It introduces a novel end-to-end training approach for accuracy at the top using a modified stochastic gradient descent and threshold estimation from minibatches.

Findings

Achieves high accuracy in visual recognition tasks.

Successfully detects 46% malware with very low false alarms.

Demonstrates effectiveness on real-world datasets.

Abstract

Accuracy at the top is a special class of binary classification problems where the performance is evaluated only on a small number of relevant (top) samples. Applications include information retrieval systems or processes with manual (expensive) postprocessing. This leads to minimizing the number of irrelevant samples above a threshold. We consider classifiers in the form of an arbitrary (deep) network and propose a new method DeepTopPush for minimizing the loss function at the top. Since the threshold depends on all samples, the problem is non-decomposable. We modify the stochastic gradient descent to handle the non-decomposability in an end-to-end training manner and propose a way to estimate the threshold only from values on the current minibatch and one delayed value. We demonstrate the excellent performance of DeepTopPush on visual recognition datasets and two real-world applications. The first one selects a small number of molecules for further drug testing. The second one uses real malware data, where we detected 46\% malware at an extremely low false alarm rate of $10^{-5}$.