The Impact of Expertise in the Loop for Exploring Machine Rationality

TL;DR

This study investigates how varying levels of human expertise influence the outcomes and satisfaction in human-in-the-loop optimization across different domains, revealing that experts pursue more diverse solutions and require more iterations.

Contribution

It provides empirical insights into the role of user expertise in human-in-the-loop optimization, highlighting differences in behavior and satisfaction levels that inform system design.

Findings

Experts perform more optimization iterations with explicit preferences.

Novices are more easily satisfied and terminate faster.

Expert behavior can serve as a performance indicator for system improvement.

Abstract

Human-in-the-loop optimization utilizes human expertise to guide machine optimizers iteratively and search for an optimal solution in a solution space. While prior empirical studies mainly investigated novices, we analyzed the impact of the levels of expertise on the outcome quality and corresponding subjective satisfaction. We conducted a study (N=60) in text, photo, and 3D mesh optimization contexts. We found that novices can achieve an expert level of quality performance, but participants with higher expertise led to more optimization iteration with more explicit preference while keeping satisfaction low. In contrast, novices were more easily satisfied and terminated faster. Therefore, we identified that experts seek more diverse outcomes while the machine reaches optimal results, and the observed behavior can be used as a performance indicator for human-in-the-loop system designers to improve underlying models. We inform future research to be cautious about the impact of user expertise when designing human-in-the-loop systems.