Evaluating Counterfactual Explanation Methods on Incomplete Inputs

TL;DR

This paper evaluates how current counterfactual explanation methods perform with incomplete data inputs, revealing that all struggle to produce valid counterfactuals, highlighting the need for new robust approaches.

Contribution

It systematically assesses existing CX methods on incomplete inputs, demonstrating their limitations and motivating the development of more robust techniques.

Findings

Robust CX methods achieve higher validity than non-robust ones.

All evaluated methods struggle to find valid counterfactuals with incomplete inputs.

Results highlight the need for new CX methods capable of handling missing data.

Abstract

Existing algorithms for generating Counterfactual Explanations (CXs) for Machine Learning (ML) typically assume fully specified inputs. However, real-world data often contains missing values, and the impact of these incomplete inputs on the performance of existing CX methods remains unexplored. To address this gap, we systematically evaluate recent CX generation methods on their ability to provide valid and plausible counterfactuals when inputs are incomplete. As part of this investigation, we hypothesize that robust CX generation methods will be better suited to address the challenge of providing valid and plausible counterfactuals when inputs are incomplete. Our findings reveal that while robust CX methods achieve higher validity than non-robust ones, all methods struggle to find valid counterfactuals. These results motivate the need for new CX methods capable of handling incomplete inputs.