Rally and WebScience: A Platform and Toolkit for Browser-Based Research on Technology and Society Problems

TL;DR

This paper introduces Rally and sdkName, a platform and toolkit designed to facilitate browser-based research on technology and society issues, addressing methodological limitations of traditional approaches.

Contribution

It presents a new platform and toolkit that lower barriers for deploying browser-based studies, enabling more ecologically valid research in online environments.

Findings

Successful deployment of a pilot study with over 4 million webpage visits

Identification of implementation barriers in current browser-based research methods

Positive feedback from researchers using the new systems

Abstract

Empirical technology and society research is in a methodological crisis. Problems increasingly involve closed platforms, targeted content, and context-specific behavior. Prevailing research methods, such as surveys, tasks, and web crawls, pose design and ecological validity limitations. Deploying studies in participant browsers and devices is a promising direction. These vantage points can observe individualized experiences and implement UI interventions in real settings. We survey scholarship that uses these methods, annotating 284 sampled papers. Our analysis demonstrates their potential, but also recurring implementation barriers and shortcomings. We then present Rally and sdkName, a platform and toolkit for browser-based research. These systems lower implementation barriers and advance the science of measuring online behavior. Finally, we evaluate Rally and sdkName against our design goals. We report results from a one-month pilot study on news engagement, analyzing 4,466,200 webpage visits from 1,817 participants. We also present observations from interviews with researchers using these systems.