The Impact of Expertise in the Loop for Exploring Machine Rationality
Changkun Ou, Sven Mayer, Andreas Butz

TL;DR
This study investigates how varying levels of human expertise influence the outcomes and satisfaction in human-in-the-loop optimization across different domains, revealing that experts pursue more diverse solutions and require more iterations.
Contribution
It provides empirical insights into the role of user expertise in human-in-the-loop optimization, highlighting differences in behavior and satisfaction levels that inform system design.
Findings
Experts perform more optimization iterations with explicit preferences.
Novices are more easily satisfied and terminate faster.
Expert behavior can serve as a performance indicator for system improvement.
Abstract
Human-in-the-loop optimization utilizes human expertise to guide machine optimizers iteratively and search for an optimal solution in a solution space. While prior empirical studies mainly investigated novices, we analyzed the impact of the levels of expertise on the outcome quality and corresponding subjective satisfaction. We conducted a study (N=60) in text, photo, and 3D mesh optimization contexts. We found that novices can achieve an expert level of quality performance, but participants with higher expertise led to more optimization iteration with more explicit preference while keeping satisfaction low. In contrast, novices were more easily satisfied and terminated faster. Therefore, we identified that experts seek more diverse outcomes while the machine reaches optimal results, and the observed behavior can be used as a performance indicator for human-in-the-loop system designers…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
