Reflow: Automatically Improving Touch Interactions in Mobile Applications through Pixel-based Refinements

TL;DR

Reflow is a system that automatically applies pixel-based, personalized UI refinements to mobile apps, significantly improving touch efficiency with minimal disruption, based on layout extraction and re-rendering.

Contribution

Reflow introduces a novel pixel-based approach for automatic, personalized UI refinements that enhance touch interactions without altering original design intent.

Findings

10% faster selection times on average

Minimal layout disruption observed

Effective in improving touch efficiency

Abstract

Touch is the primary way that users interact with smartphones. However, building mobile user interfaces where touch interactions work well for all users is a difficult problem, because users have different abilities and preferences. We propose a system, Reflow, which automatically applies small, personalized UI adaptations, called refinements -- to mobile app screens to improve touch efficiency. Reflow uses a pixel-based strategy to work with existing applications, and improves touch efficiency while minimally disrupting the design intent of the original application. Our system optimizes a UI by (i) extracting its layout from its screenshot, (ii) refining its layout, and (iii) re-rendering the UI to reflect these modifications. We conducted a user study with 10 participants and a heuristic evaluation with 6 experts and found that applications optimized by Reflow led to, on average, 9% faster selection time with minimal layout disruption. The results demonstrate that Reflow's refinements useful UI adaptations to improve touch interactions.