Capsa: A Unified Framework for Quantifying Risk in Deep Neural Networks

TL;DR

Capsa is a flexible framework that enables comprehensive risk quantification in deep neural networks by combining various uncertainty and bias estimation methods, improving risk-awareness in complex scenarios.

Contribution

Capsa introduces a unified, composable framework for quantifying multiple risk metrics in deep neural networks, simplifying integration and comparison of risk estimation algorithms.

Findings

Successfully implemented state-of-the-art uncertainty algorithms within Capsa.

Demonstrated the ability to combine multiple risk metrics in a single procedure.

Validated on complex perception datasets showing comprehensive risk awareness.

Abstract

The modern pervasiveness of large-scale deep neural networks (NNs) is driven by their extraordinary performance on complex problems but is also plagued by their sudden, unexpected, and often catastrophic failures, particularly on challenging scenarios. Existing algorithms that provide risk-awareness to NNs are complex and ad-hoc. Specifically, these methods require significant engineering changes, are often developed only for particular settings, and are not easily composable. Here we present capsa, a framework for extending models with risk-awareness. Capsa provides a methodology for quantifying multiple forms of risk and composing different algorithms together to quantify different risk metrics in parallel. We validate capsa by implementing state-of-the-art uncertainty estimation algorithms within the capsa framework and benchmarking them on complex perception datasets. We demonstrate capsa's ability to easily compose aleatoric uncertainty, epistemic uncertainty, and bias estimation together in a single procedure, and show how this approach provides a comprehensive awareness of NN risk.