Human-Centered LLM-Agent User Interface: A Position Paper

TL;DR

This paper advocates for a human-centered LLM-agent user interface that proactively assists users by discovering workflows and proposing interactions, demonstrated through a flute tutoring system.

Contribution

It introduces the concept of a proactive, human-centered LLM-agent UI that surpasses passive command following, exemplified by the Flute X GPT system.

Findings

Proposes a new LAUI paradigm emphasizing proactive assistance.

Demonstrates LAUI with a multi-modal flute tutoring system.

Highlights potential for improved user-system interaction.

Abstract

Large Language Model (LLM) -in-the-loop applications have been shown to effectively interpret the human user's commands, make plans, and operate external tools/systems accordingly. Still, the operation scope of the LLM agent is limited to passively following the user, requiring the user to frame his/her needs with regard to the underlying tools/systems. We note that the potential of an LLM-Agent User Interface (LAUI) is much greater. A user mostly ignorant to the underlying tools/systems should be able to work with a LAUI to discover an emergent workflow. Contrary to the conventional way of designing an explorable GUI to teach the user a predefined set of ways to use the system, in the ideal LAUI, the LLM agent is initialized to be proficient with the system, proactively studies the user and his/her needs, and proposes new interaction schemes to the user. To illustrate LAUI, we present Flute X GPT, a concrete example using an LLM agent, a prompt manager, and a flute-tutoring multi-modal software-hardware system to facilitate the complex, real-time user experience of learning to play the flute.