Experiences with User Studies in Augmented Reality

TL;DR

This paper discusses challenges faced in designing and conducting user studies in augmented reality to improve the robustness and reliability of user experience evaluations as AR moves towards consumer markets.

Contribution

It identifies three key challenges in AR user studies based on the authors' research experiences, providing insights for future research design.

Findings

Identified three main challenges in AR user studies.

Insights applicable to perceptual, interaction, and design space studies.

Highlights importance of robust user experience evaluation methods.

Abstract

The research field of augmented reality (AR) is of increasing popularity, as seen, among others, in several recently published surveys. To produce further advancements in AR, it is not only necessary to create new systems or applications, but also to evaluate them. One important aspect in regards to the evaluation is the general understanding of how users experience a given AR application, which can also be seen by the increased number of papers focusing on this topic that were published in the last years. With the steadily growing understanding and development of AR in general, it is only a matter of time until AR devices make the leap into the consumer market where such an in-depth user understanding is even more essential. Thus, a better understanding of factors that could influence the design and results of user experience studies can help us to make them more robust and dependable in the future. In this position paper, we describe three challenges which researchers face while designing and conducting AR users studies. We encountered these challenges in our past and current research, including papers that focus on perceptual studies of visualizations, interaction studies, and studies exploring the use of AR applications and their design spaces.