An Item Response Theory-based R Module for Algorithm Portfolio Analysis

TL;DR

This paper presents AIRT-Module, an R-based tool utilizing Item Response Theory to evaluate and visualize the strengths and weaknesses of algorithms across diverse test instances, improving AI algorithm assessment.

Contribution

The paper introduces a novel IRT-based analysis tool for algorithm portfolio evaluation, adapting psychometric models to AI performance assessment.

Findings

Provides detailed algorithm capability insights

Visualizes algorithm strengths and weaknesses

Enhances comprehensive AI evaluation

Abstract

Experimental evaluation is crucial in AI research, especially for assessing algorithms across diverse tasks. Many studies often evaluate a limited set of algorithms, failing to fully understand their strengths and weaknesses within a comprehensive portfolio. This paper introduces an Item Response Theory (IRT) based analysis tool for algorithm portfolio evaluation called AIRT-Module. Traditionally used in educational psychometrics, IRT models test question difficulty and student ability using responses to test questions. Adapting IRT to algorithm evaluation, the AIRT-Module contains a Shiny web application and the R package airt. AIRT-Module uses algorithm performance measures to compute anomalousness, consistency, and difficulty limits for an algorithm and the difficulty of test instances. The strengths and weaknesses of algorithms are visualised using the difficulty spectrum of the test instances. AIRT-Module offers a detailed understanding of algorithm capabilities across varied test instances, thus enhancing comprehensive AI method assessment. It is available at https://sevvandi.shinyapps.io/AIRT/ .