Lessons from Four Years of PHONELAB Experimentation

TL;DR

This paper shares lessons learned from four years of operating PHONELAB, a large-scale smartphone testbed, highlighting design, experimentation process, research supported, and practical insights for future mobile systems research.

Contribution

It provides a comprehensive account of running a large user-facing smartphone testbed, including design choices, operational challenges, and practical lessons learned over four years.

Findings

Lessons learned through trial and error

Insights into testbed design and operation

Guidance for future mobile systems experiments

Abstract

Over the last four years we have operated a public smartphone platform testbed called PHONELAB. PHONELAB consists of up to several-hundred participants who run an experimental platform image on their primary smartphone. The experimental platform consists of both instrumentation and experimental changes to platform components, including core Android services and Linux. This paper describes the design of the testbed, the process of conducting PHONELAB experiments, and some of the research the testbed has supported. We also offer many lessons learned along the way, almost all of which have been learned the hard way--through trial and a lot of error. We expect our experiences will help those contemplating operating large user-facing testbeds, anyone conducting experiments on smartphones, and many mobile systems researchers.