Controversy Rules - Discovering Regions Where Classifiers (Dis-)Agree Exceptionally

TL;DR

This paper introduces an algorithm based on Exceptional Model Mining to identify regions where classifiers strongly agree or disagree, providing insights into data and model behavior.

Contribution

The work presents a novel algorithm for discovering controversial regions among classifiers using the Exceptional Model Mining framework.

Findings

Identified regions of high classifier disagreement in public datasets

Revealed known and previously unreported phenomena in datasets

Demonstrated the usefulness of the approach in classification analysis

Abstract

Finding regions for which there is higher controversy among different classifiers is insightful with regards to the domain and our models. Such evaluation can falsify assumptions, assert some, or also, bring to the attention unknown phenomena. The present work describes an algorithm, which is based on the Exceptional Model Mining framework, and enables that kind of investigations. We explore several public datasets and show the usefulness of this approach in classification tasks. We show in this paper a few interesting observations about those well explored datasets, some of which are general knowledge, and other that as far as we know, were not reported before.