Multi-Dimensional Ability Diagnosis for Machine Learning Algorithms

TL;DR

This paper introduces Camilla, a multi-dimensional evaluation framework inspired by psychometric theories, to better measure and compare machine learning algorithms' multifaceted strengths beyond traditional accuracy metrics.

Contribution

The paper proposes a novel, task-agnostic diagnostic framework that uses neural networks and psychometric principles to assess multiple skills of algorithms simultaneously.

Findings

Camilla outperforms state-of-the-art baselines in reliability and stability.

It captures detailed strengths and weaknesses of algorithms.

The framework effectively quantifies sample difficulty and algorithm skills.

Abstract

Machine learning algorithms have become ubiquitous in a number of applications (e.g. image classification). However, due to the insufficient measurement of traditional metrics (e.g. the coarse-grained Accuracy of each classifier), substantial gaps are usually observed between the real-world performance of these algorithms and their scores in standardized evaluations. In this paper, inspired by the psychometric theories from human measurement, we propose a task-agnostic evaluation framework Camilla, where a multi-dimensional diagnostic metric Ability is defined for collaboratively measuring the multifaceted strength of each machine learning algorithm. Specifically, given the response logs from different algorithms to data samples, we leverage cognitive diagnosis assumptions and neural networks to learn the complex interactions among algorithms, samples and the skills (explicitly or implicitly pre-defined) of each sample. In this way, both the abilities of each algorithm on multiple skills and some of the sample factors (e.g. sample difficulty) can be simultaneously quantified. We conduct extensive experiments with hundreds of machine learning algorithms on four public datasets, and our experimental results demonstrate that Camilla not only can capture the pros and cons of each algorithm more precisely, but also outperforms state-of-the-art baselines on the metric reliability, rank consistency and rank stability.