Exploring Mobile Touch Interaction with Large Language Models

TL;DR

This paper introduces a novel method for controlling large language models on mobile devices using touch gestures directly on text, demonstrating its feasibility and user-friendliness through a user study.

Contribution

It proposes a new touch gesture control space for LLMs on mobile devices and evaluates specific mappings with visual feedback, advancing mobile LLM interaction methods.

Findings

Touch-based control is feasible and user-friendly.

Length + word indicator improves control effectiveness.

Gesture-based interaction can enhance mobile LLM usability.

Abstract

Interacting with Large Language Models (LLMs) for text editing on mobile devices currently requires users to break out of their writing environment and switch to a conversational AI interface. In this paper, we propose to control the LLM via touch gestures performed directly on the text. We first chart a design space that covers fundamental touch input and text transformations. In this space, we then concretely explore two control mappings: spread-to-generate and pinch-to-shorten, with visual feedback loops. We evaluate this concept in a user study (N=14) that compares three feedback designs: no visualisation, text length indicator, and length + word indicator. The results demonstrate that touch-based control of LLMs is both feasible and user-friendly, with the length + word indicator proving most effective for managing text generation. This work lays the foundation for further research into gesture-based interaction with LLMs on touch devices.