How Much is Performance Worth to Users? A Quantitative Approach

TL;DR

This paper introduces three methodologies to quantify the monetary value users assign to system performance, enabling better design trade-offs between user experience and system constraints.

Contribution

It presents a scalable, incentive-compatible methodology, a lab-based simulation, and a survey approach to measure user valuation of performance loss.

Findings

Users accept up to 10-30% performance loss for specific monetary values.

Incentive-compatible methods provide more accurate measurements.

Surveys tend to underestimate user willingness to accept performance loss.

Abstract

Architects and systems designers artfully balance multiple competing design constraints during the design process but are unable to translate between system metrics and end user experience. This work presents three methodologies to fill in this gap. The first is an incentive-compatible methodology that determines a "ground truth" measurement of users' value of speed in terms of US dollars, and find that users would accept a performance losses of 10%, 20%, and 30% to their personal computer in exchange for \$2.27, \$4.07, and \$4.43 per day, respectively. However, while highly accurate the methodology is a painstaking process and does not scale with large numbers of participants. To allow for scalability, we introduce a second methodology -- a lab-based simulation experiment -- which finds that users would accept a permanent performance loss of 10%, 20%, and 30% to their personal computer in exchange for \$127, \$169, and \$823, respectively. Finally, to allow for even greater scalability, we introduce a third methodology -- a survey -- and observe that the lack of incentive compatibility and the lack of hands-on experience with throttled device performance skews the results significantly, thus demonstrating the need for lab-based or incentive compatible study designs. By quantifying the tradeoff between user satisfaction and performance, we enable architects and systems designers to make more nuanced tradeoffs between design requirements.