Evaluating Deep Taylor Decomposition for Reliability Assessment in the Wild

TL;DR

This paper evaluates the effectiveness of Deep Taylor Decomposition for interpretability in real-world scenarios, demonstrating its benefits in aiding journalists' decision-making and source evaluation.

Contribution

It provides an in-the-wild assessment of token attribution methods, specifically Deep Taylor Decomposition, in a professional setting involving journalists and news source reliability.

Findings

Faster and more accurate human decisions with the method

Increased critical attitude towards news sources

Positive qualitative feedback from journalists

Abstract

We argue that we need to evaluate model interpretability methods 'in the wild', i.e., in situations where professionals make critical decisions, and models can potentially assist them. We present an in-the-wild evaluation of token attribution based on Deep Taylor Decomposition, with professional journalists performing reliability assessments. We find that using this method in conjunction with RoBERTa-Large, fine-tuned on the Gossip Corpus, led to faster and better human decision-making, as well as a more critical attitude toward news sources among the journalists. We present a comparison of human and model rationales, as well as a qualitative analysis of the journalists' experiences with machine-in-the-loop decision making.