Language-Guided Multimodal Texture Authoring via Generative Models

TL;DR

This paper introduces a language-guided system that generates multimodal textures, combining haptic and visual outputs from natural language prompts, streamlining texture design and enabling intuitive, prompt-based control.

Contribution

It presents a novel multimodal texture authoring framework that links haptic and visual modalities through shared language-aligned latent representations, facilitating semantic consistency and user-friendly design.

Findings

Participants rated textures consistent with prompts for roughness, hardness, and slipperiness.

The system enables immediate visual and tactile feedback from a single natural language prompt.

Language effectively controls both haptic and visual texture attributes, supporting a prompt-first workflow.

Abstract

Authoring realistic haptic textures typically requires low-level parameter tuning and repeated trial-and-error, limiting speed, transparency, and creative reach. We present a language-driven authoring system that turns natural-language prompts into multimodal textures: two coordinated haptic channels - sliding vibrations via force/speed-conditioned autoregressive (AR) models and tapping transients - and a text-prompted visual preview from a diffusion model. A shared, language-aligned latent links modalities so a single prompt yields semantically consistent haptic and visual signals; designers can write goals (e.g., "gritty but cushioned surface," "smooth and hard metal surface") and immediately see and feel the result through a 3D haptic device. To verify that the learned latent encodes perceptually meaningful structure, we conduct an anchor-referenced, attribute-wise evaluation for roughness, slipperiness, and hardness. Participant ratings are projected to the interpretable line between two real-material references, revealing consistent trends - asperity effects in roughness, compliance in hardness, and surface-film influence in slipperiness. A human-subject study further indicates coherent cross-modal experience and low effort for prompt-based iteration. The results show that language can serve as a practical control modality for texture authoring: prompts reliably steer material semantics across haptic and visual channels, enabling a prompt-first, designer-oriented workflow that replaces manual parameter tuning with interpretable, text-guided refinement.