SummAct: Uncovering User Intentions Through Interactive Behaviour Summarisation

TL;DR

SummAct is a hierarchical method that automatically uncovers user intentions from interactive behaviour, improving understanding of user goals in GUIs and enabling applications like forecasting and behaviour retrieval.

Contribution

The paper introduces SummAct, a novel hierarchical approach combining large language models and UI element attention to summarise low-level actions into high-level user intentions.

Findings

SummAct outperforms baselines by up to 21.9% across various interfaces.

It enables applications like behaviour forecasting and behaviour retrieval.

SummAct effectively captures detailed user intentions from interaction data.

Abstract

Recent work has highlighted the potential of modelling interactive behaviour analogously to natural language. We propose interactive behaviour summarisation as a novel computational task and demonstrate its usefulness for automatically uncovering latent user intentions while interacting with graphical user interfaces. To tackle this task, we introduce SummAct, a novel hierarchical method to summarise low-level input actions into high-level intentions. SummAct first identifies sub-goals from user actions using a large language model and in-context learning. High-level intentions are then obtained by fine-tuning the model using a novel UI element attention to preserve detailed context information embedded within UI elements during summarisation. Through a series of evaluations, we demonstrate that SummAct significantly outperforms baselines across desktop and mobile interfaces as well as interactive tasks by up to 21.9%. We further show three exciting interactive applications benefited from SummAct: interactive behaviour forecasting, automatic behaviour synonym identification, and language-based behaviour retrieval.