MetaErr: Towards Predicting Error Patterns in Deep Neural Networks

TL;DR

MetaErr is a framework that predicts the failure of deep neural networks on specific data samples, aiding in understanding and improving model reliability in multimedia applications.

Contribution

We introduce MetaErr, a meta-model that predicts deep neural network failures without relying on model architecture or training details, enhancing error prediction capabilities.

Findings

MetaErr outperforms several baselines on three benchmark datasets.

MetaErr improves semi-supervised learning performance.

The framework is architecture-agnostic and effective across different tasks.

Abstract

Due to the unprecedented success of deep learning, it has become an integral component in several multimedia computing applications in todays world. Unfortunately, deep learning systems are not perfect and can fail, sometimes abruptly, without prior warning or explanation. While reducing the error rate of deep neural networks has been the primary focus of the multimedia community, the problem of predicting when a deep learning system is going to fail has received significantly less research attention. In this paper, we propose a simple yet effective framework, MetaErr, to address this under-explored problem in deep learning research. We train a meta-model whose goal is to predict whether a base deep neural network will succeed or fail in predicting a particular data sample, by observing the base models performance on a given learning task. The meta-model is completely agnostic of the architecture and training parameters of the base model. Such an error prediction system can be immensely useful in a variety of smart multimedia applications. Our empirical studies corroborate the promise and potential of our framework against competing baselines. We further demonstrate the usefulness of our framework to improve the performance of pseudo-labeling-based semi-supervised learning, and show that MetaErr outperforms several strong baselines on three benchmark computer vision datasets.