Towards Difficulty-Aware Analysis of Deep Neural Networks

TL;DR

This paper introduces a difficulty-aware evaluation framework for deep neural networks, incorporating data, model, and human perspectives, along with an interactive visualization tool to better understand and analyze model performance on challenging instances.

Contribution

It proposes a novel approach to evaluate neural networks by considering instance difficulty from multiple perspectives and provides an interactive tool for analysis.

Findings

Difficulty-aware evaluation reveals insights missed by traditional methods.

The visual tool helps identify challenging instances and potential issues.

Case studies validate the effectiveness of the proposed approach.

Abstract

Traditional instance-based model analysis focuses mainly on misclassified instances. However, this approach overlooks the varying difficulty associated with different instances. Ideally, a robust model should recognize and reflect the challenges presented by intrinsically difficult instances. It is also valuable to investigate whether the difficulty perceived by the model aligns with that perceived by humans. To address this, we propose incorporating instance difficulty into the deep neural network evaluation process, specifically for supervised classification tasks on image data. Specifically, we consider difficulty measures from three perspectives -- data, model, and human -- to facilitate comprehensive evaluation and comparison. Additionally, we develop an interactive visual tool, DifficultyEyes, to support the identification of instances of interest based on various difficulty patterns and to aid in analyzing potential data or model issues. Case studies demonstrate the effectiveness of our approach.