ORBIS: Output-Guided Token Reduction with Distribution-Aware Matching for Video Diffusion Acceleration

TL;DR

ORBIS is a specialized accelerator that significantly speeds up video diffusion models by improving token reduction accuracy and efficiency, leading to faster processing and lower energy consumption.

Contribution

It introduces a novel output-guided token reduction method with a distribution-aware matching algorithm and hardware design, achieving higher token reduction and acceleration.

Findings

ORBIS achieves about 2x higher token reduction ratio than state-of-the-art.

It delivers up to 4.5x speedup and 79.3% energy reduction on NVIDIA A100.

The hardware design occupies only 2.4% of total area with negligible accuracy loss.

Abstract

Diffusion Transformer (DiT) has emerged as a powerful model architecture for generating high-quality images and videos. In the case of video DiT, 3D Spatio-Temporal Attention increases token length in proportion to the number of frames, sharply increasing computational cost. Token reduction methods mitigate this cost by exploiting spatial redundancy, but existing approaches rely on inaccurate similarity estimates and lightweight matching algorithms, resulting in poor matching quality and only marginal acceleration. To overcome these limitations, we propose ORBIS, an SW-HW co-designed accelerator for video DiT. ORBIS leverages the output activation from the previous timestep to obtain more accurate inter-token similarity, substantially improving matching quality and enabling a higher token reduction ratio. We further introduce a Distribution-Aware Token Matching (DATM) algorithm that captures global token distribution and explicitly minimizes token-pair loss for additional gains. To fully hide DATM latency, we design specialized, deeply pipelined hardware and minimize its hardware cost through quantization, occupying only 2.4% of total area with negligible accuracy loss. Extensive experiments show that ORBIS achieves about 2x higher token reduction ratio than the state-of-the-art approach, AsymRnR, while delivering up to 4.5x speedup and 79.3% energy reduction compared to an NVIDIA A100 GPU.