Fibres of Failure: Classifying errors in predictive processes

TL;DR

Fibres of Failure (FiFa) is a novel method that uses Topological Data Analysis to classify and analyze failure modes in predictive models, aiding in understanding and correcting errors.

Contribution

The paper introduces FiFa, a new approach leveraging the Mapper algorithm to identify and interpret failure modes in predictive processes.

Findings

Successfully classified misclassifications in noisy MNIST images.

Demonstrated two applications: correction layer and failure mode inspection.

Provided insights into failure characteristics of predictive models.

Abstract

We describe Fibres of Failure (FiFa), a method to classify failure modes of predictive processes using the Mapper algorithm from Topological Data Analysis. Our method uses Mapper to build a graph model of input data stratified by prediction error. Groupings found in high-error regions of the Mapper model then provide distinct failure modes of the predictive process. We demonstrate FiFa on misclassifications of MNIST images with added noise, and demonstrate two ways to use the failure mode classification: either to produce a correction layer that adjusts predictions by similarity to the failure modes; or to inspect members of the failure modes to illustrate and investigate what characterizes each failure mode.