Voxify3D: Pixel Art Meets Volumetric Rendering

TL;DR

Voxify3D introduces a novel differentiable framework that combines 3D mesh optimization with pixel art supervision to generate stylized voxel art with semantic and aesthetic fidelity.

Contribution

It presents a unique integration of orthographic supervision, CLIP alignment, and palette-constrained quantization for improved voxel art generation from meshes.

Findings

Achieved 37.12 CLIP-IQA score and 77.90% user preference in experiments.

Supported controllable abstraction with 2-8 colors and 20x-50x resolutions.

Abstract

Voxel art is a distinctive stylization widely used in games and digital media, yet automated generation from 3D meshes remains challenging due to conflicting requirements of geometric abstraction, semantic preservation, and discrete color coherence. Existing methods either over-simplify geometry or fail to achieve the pixel-precise, palette-constrained aesthetics of voxel art. We introduce Voxify3D, a differentiable two-stage framework bridging 3D mesh optimization with 2D pixel art supervision. Our core innovation lies in the synergistic integration of three components: (1) orthographic pixel art supervision that eliminates perspective distortion for precise voxel-pixel alignment; (2) patch-based CLIP alignment that preserves semantics across discretization levels; (3) palette-constrained Gumbel-Softmax quantization enabling differentiable optimization over discrete color spaces with controllable palette strategies. This integration addresses fundamental challenges: semantic preservation under extreme discretization, pixel-art aesthetics through volumetric rendering, and end-to-end discrete optimization. Experiments show superior performance (37.12 CLIP-IQA, 77.90% user preference) across diverse characters and controllable abstraction (2-8 colors, 20x-50x resolutions). Project page: https://yichuanh.github.io/Voxify-3D/