Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning

TL;DR

Screen2Words is a multimodal learning approach that automatically generates concise language descriptions of mobile UI screens, leveraging a large dataset and deep models to improve understanding and communication of UI content.

Contribution

The paper introduces a novel multimodal learning method for mobile UI summarization, supported by a large-scale annotated dataset and comprehensive model evaluations.

Findings

High-quality summaries generated by the models

Large-scale dataset with over 112k annotations

Effective multimodal learning approach for UI understanding

Abstract

Mobile User Interface Summarization generates succinct language descriptions of mobile screens for conveying important contents and functionalities of the screen, which can be useful for many language-based application scenarios. We present Screen2Words, a novel screen summarization approach that automatically encapsulates essential information of a UI screen into a coherent language phrase. Summarizing mobile screens requires a holistic understanding of the multi-modal data of mobile UIs, including text, image, structures as well as UI semantics, motivating our multi-modal learning approach. We collected and analyzed a large-scale screen summarization dataset annotated by human workers. Our dataset contains more than 112k language summarization across $\sim$22k unique UI screens. We then experimented with a set of deep models with different configurations. Our evaluation of these models with both automatic accuracy metrics and human rating shows that our approach can generate high-quality summaries for mobile screens. We demonstrate potential use cases of Screen2Words and open-source our dataset and model to lay the foundations for further bridging language and user interfaces.