ReactGenie: A Development Framework for Complex Multimodal Interactions Using Large Language Models
Jackie Junrui Yang, Yingtian Shi, Yuhan Zhang, Karina Li, Daniel Wan, Rosli, Anisha Jain, Shuning Zhang, Tianshi Li, James A. Landay, Monica S. Lam

TL;DR
ReactGenie is a framework that simplifies the development of complex multimodal interfaces by translating user commands into a specialized programming language, enabling richer interactions and faster task completion.
Contribution
It introduces NLPL and a neural semantic parser to separate multimodal input from the computational model, easing development of advanced multimodal applications.
Findings
Developers learned to build applications in under 2.5 hours.
Users completed tasks faster with ReactGenie.
ReactGenie supports richer multimodal commands.
Abstract
By combining voice and touch interactions, multimodal interfaces can surpass the efficiency of either modality alone. Traditional multimodal frameworks require laborious developer work to support rich multimodal commands where the user's multimodal command involves possibly exponential combinations of actions/function invocations. This paper presents ReactGenie, a programming framework that better separates multimodal input from the computational model to enable developers to create efficient and capable multimodal interfaces with ease. ReactGenie translates multimodal user commands into NLPL (Natural Language Programming Language), a programming language we created, using a neural semantic parser based on large-language models. The ReactGenie runtime interprets the parsed NLPL and composes primitives in the computational model to implement complex user commands. As a result, ReactGenie…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Speech and dialogue systems
