Controlled Experimentation in Naturalistic Mobile Settings

TL;DR

This paper presents a set of validation protocols derived from observational data to ensure controlled mobile user experiments are natural and unobtrusive, improving experimental reliability in naturalistic settings.

Contribution

It introduces validation checks based on observed user behavior to verify the naturalness of controlled experiments on small mobile devices.

Findings

Validation protocols derived from 400 user observations

Checklists for experiment design validation

Enhanced confidence in naturalistic mobile experiments

Abstract

Performing controlled user experiments on small devices in naturalistic mobile settings has always proved to be a difficult undertaking for many Human Factors researchers. Difficulties exist, not least, because mimicking natural small device usage suffers from a lack of unobtrusive data to guide experimental design, and then validate that the experiment is proceeding naturally.Here we use observational data to derive a set of protocols and a simple checklist of validations which can be built into the design of any controlled experiment focused on the user interface of a small device. These, have been used within a series of experimental designs to measure the utility and application of experimental software. The key-point is the validation checks -- based on the observed behaviour of 400 mobile users -- to ratify that a controlled experiment is being perceived as natural by the user. While the design of the experimental route which the user follows is a major factor in the experimental setup, without check validations based on unobtrusive observed data there can be no certainty that an experiment designed to be natural is actually progressing as the design implies.