Creating Healthy Friction: Determining Stakeholder Requirements of Job Recommendation Explanations

TL;DR

This study evaluates how explanations in job recommender systems influence stakeholder decision-making, revealing that explanations serve better as decision support tools rather than persuasive devices, with minimal impact on accuracy and speed.

Contribution

The paper provides empirical evidence on stakeholder preferences for explanations in job recommender systems, emphasizing their role in fostering trust and transparency over improving decision speed or accuracy.

Findings

Real explanations do not significantly improve decision accuracy or speed.

Stakeholders perceive explanations as more trustworthy and useful, though not significantly.

Explanations act as healthy friction, supporting decision-making rather than persuasion.

Abstract

The increased use of information retrieval in recruitment, primarily through job recommender systems (JRSs), can have a large impact on job seekers, recruiters, and companies. As a result, such systems have been determined to be high-risk in recent legislature. This requires JRSs to be trustworthy and transparent, allowing stakeholders to understand why specific recommendations were made. To fulfill this requirement, the stakeholders' exact preferences and needs need to be determined. To do so, we evaluated an explainable job recommender system using a realistic, task-based, mixed-design user study (n=30) in which stakeholders had to make decisions based on the model's explanations. This mixed-methods evaluation consisted of two objective metrics - correctness and efficiency, along with three subjective metrics - trust, transparency, and usefulness. These metrics were evaluated twice per participant, once using real explanations and once using random explanations. The study included a qualitative analysis following a think-aloud protocol while performing tasks adapted to each stakeholder group. We find that providing stakeholders with real explanations does not significantly improve decision-making speed and accuracy. Our results showed a non-significant trend for the real explanations to outperform the random ones on perceived trust, usefulness, and transparency of the system for all stakeholder types. We determine that stakeholders benefit more from interacting with explanations as decision support capable of providing healthy friction, rather than as previously-assumed persuasive tools.