ShadAR: LLM-driven shader generation to transform visual perception in Augmented Reality

TL;DR

ShadAR leverages large language models to generate real-time shaders from natural language, enabling flexible and inclusive visual perception transformations in augmented reality applications.

Contribution

This paper introduces ShadAR, a novel AR system that uses LLMs to generate shaders from natural language, enhancing flexibility and user control in visual perception simulation.

Findings

Successfully generates shaders from natural language commands

Enables real-time visual perception transformations in AR

Improves inclusiveness and creative possibilities in AR

Abstract

Augmented Reality (AR) can simulate various visual perceptions, such as how individuals with colorblindness see the world. However, these simulations require developers to predefine each visual effect, limiting flexibility. We present ShadAR, an AR application enabling real-time transformation of visual perception through shader generation using large language models (LLMs). ShadAR allows users to express their visual intent via natural language, which is interpreted by an LLM to generate corresponding shader code. This shader is then compiled real-time to modify the AR headset viewport. We present our LLM-driven shader generation pipeline and demonstrate its ability to transform visual perception for inclusiveness and creativity.