TextOnly: A Unified Function Portal for Text-Related Functions on Smartphones

TL;DR

TextOnly is a unified smartphone interface that interprets user text inputs to perform diverse functions across applications, leveraging large language models and BERT for improved accuracy and personalization.

Contribution

It introduces a novel unified text-based function portal on smartphones that integrates LLM and BERT models for accurate, personalized, and efficient function execution from raw text inputs.

Findings

Achieved 71.35% top-1 accuracy in real-world user studies

Demonstrated continuous improvement in accuracy and inference speed

Participants preferred TextOnly over manual and voice-based methods

Abstract

Text boxes serve as portals to diverse functionalities in today's smartphone applications. However, when it comes to specific functionalities, users always need to navigate through multiple steps to access particular text boxes for input. We propose TextOnly, a unified function portal that enables users to access text-related functions from various applications by simply inputting text into a sole text box. For instance, entering a restaurant name could trigger a Google Maps search, while a greeting could initiate a conversation in WhatsApp. Despite their brevity, TextOnly maximizes the utilization of these raw text inputs, which contain rich information, to interpret user intentions effectively. TextOnly integrates large language models(LLM) and a BERT model. The LLM consistently provides general knowledge, while the BERT model can continuously learn user-specific preferences and enable quicker predictions. Real-world user studies demonstrated TextOnly's effectiveness with a top-1 accuracy of 71.35%, and its ability to continuously improve both its accuracy and inference speed. Participants perceived TextOnly as having satisfactory usability and expressed a preference for TextOnly over manual executions. Compared with voice assistants, TextOnly supports a greater range of text-related functions and allows for more concise inputs.