Brickify: Enabling Expressive Design Intent Specification through Direct Manipulation on Design Tokens

TL;DR

Brickify introduces a visual-centric interaction paradigm that enables designers to specify and manipulate design intent directly through design tokens, improving efficiency and intuitiveness over text-based methods.

Contribution

It presents a novel system that extracts and manipulates visual elements as design tokens, facilitating expressive and direct visual design specification.

Findings

Designers found Brickify more efficient than text prompts.

Brickify enables intuitive manipulation of visual elements.

The system effectively interprets and executes visual lexicons.

Abstract

Expressing design intent using natural language prompts requires designers to verbalize the ambiguous visual details concisely, which can be challenging or even impossible. To address this, we introduce Brickify, a visual-centric interaction paradigm -- expressing design intent through direct manipulation on design tokens. Brickify extracts visual elements (e.g., subject, style, and color) from reference images and converts them into interactive and reusable design tokens that can be directly manipulated (e.g., resize, group, link, etc.) to form the visual lexicon. The lexicon reflects users' intent for both what visual elements are desired and how to construct them into a whole. We developed Brickify to demonstrate how AI models can interpret and execute the visual lexicon through an end-to-end pipeline. In a user study, experienced designers found Brickify more efficient and intuitive than text-based prompts, allowing them to describe visual details, explore alternatives, and refine complex designs with greater ease and control.