Expectation Versus Reality: The Failed Evaluation of a Mixed-Initiative Visualization System

TL;DR

This study evaluated a mixed-initiative visualization system through user trials but failed to demonstrate its impact, highlighting the importance of interface design and user expectations in system usability.

Contribution

It provides insights into the challenges of evaluating adaptive visualization systems and emphasizes learning from unsuccessful experiments to guide future research.

Findings

The system did not significantly affect user interactions.

Interface design flaws impacted usability outcomes.

User expectations influenced system effectiveness.

Abstract

Our research aimed to present the design and evaluation of a mixed-initiative system that aids the user in handling complex datasets and dense visualization systems. We attempted to demonstrate this system with two trials of an online between-groups, two-by-two study, measuring the effects of this mixed-initiative system on user interactions and system usability. However, due to flaws in the interface design and the expectations that we put on users, we were unable to show that the adaptive system had an impact on user interactions or system usability. In this paper, we discuss the unexpected findings that we found from our "failed" experiments and examine how we can learn from our failures to improve further research.