Principal Sensitivity Analysis

TL;DR

Principal Sensitivity Analysis (PSA) introduces a method to identify the most sensitive input directions of classifiers, helping to interpret learned knowledge in neural networks through visualization of principal sensitivity maps.

Contribution

The paper proposes a novel algorithm, PSA, for analyzing classifier sensitivity and decomposing learned knowledge using principal sensitivity maps.

Findings

PSA effectively visualizes classifier sensitivity.

Application to neural networks demonstrates interpretability.

Decomposition of knowledge in trained classifiers.

Abstract

We present a novel algorithm (Principal Sensitivity Analysis; PSA) to analyze the knowledge of the classifier obtained from supervised machine learning techniques. In particular, we define principal sensitivity map (PSM) as the direction on the input space to which the trained classifier is most sensitive, and use analogously defined k-th PSM to define a basis for the input space. We train neural networks with artificial data and real data, and apply the algorithm to the obtained supervised classifiers. We then visualize the PSMs to demonstrate the PSA's ability to decompose the knowledge acquired by the trained classifiers.