Leveraging Multi-Method Evaluation for Multi-Stakeholder Settings

TL;DR

This paper advocates for using a multi-method evaluation approach in multi-stakeholder recommendation settings to capture diverse goals and prevent blind spots in assessment.

Contribution

It introduces the concept of multi-method evaluation for complex multi-stakeholder recommendation scenarios, emphasizing its importance over single-method approaches.

Findings

Multi-method evaluation provides a more comprehensive assessment.

Single evaluation methods are insufficient for multi-stakeholder settings.

Combining multiple measures reveals diverse stakeholder perspectives.

Abstract

In this paper, we focus on recommendation settings with multiple stakeholders with possibly varying goals and interests, and argue that a single evaluation method or measure is not able to evaluate all relevant aspects in such a complex setting. We reason that employing a multi-method evaluation, where multiple evaluation methods or measures are combined and integrated, allows for getting a richer picture and prevents blind spots in the evaluation outcome.