Towards Better User Studies in Computer Graphics and Vision

TL;DR

This paper highlights the importance of rigorous user study design in computer graphics and vision research, advocating for better methodology, reporting, and utilization of user research to improve validity and project direction.

Contribution

It advocates for improved design and reporting of user studies, introduces methodologies from UX and HCI, and emphasizes their potential to enhance research validity and project planning.

Findings

Hasty user studies can lead to misleading conclusions.

Underutilization of foundational user research methods in vision and graphics.

Recommendations for community adoption of better user research practices.

Abstract

Online crowdsourcing platforms have made it increasingly easy to perform evaluations of algorithm outputs with survey questions like "which image is better, A or B?", leading to their proliferation in vision and graphics research papers. Results of these studies are often used as quantitative evidence in support of a paper's contributions. On the one hand we argue that, when conducted hastily as an afterthought, such studies lead to an increase of uninformative, and, potentially, misleading conclusions. On the other hand, in these same communities, user research is underutilized in driving project direction and forecasting user needs and reception. We call for increased attention to both the design and reporting of user studies in computer vision and graphics papers towards (1) improved replicability and (2) improved project direction. Together with this call, we offer an overview of methodologies from user experience research (UXR), human-computer interaction (HCI), and applied perception to increase exposure to the available methodologies and best practices. We discuss foundational user research methods (e.g., needfinding) that are presently underutilized in computer vision and graphics research, but can provide valuable project direction. We provide further pointers to the literature for readers interested in exploring other UXR methodologies. Finally, we describe broader open issues and recommendations for the research community.