GSPN-2: Efficient Parallel Sequence Modeling

TL;DR

GSPN-2 significantly improves the efficiency of global spatial context modeling in vision transformers by redesigning the algorithm and GPU implementation, reducing computational overhead while maintaining accuracy.

Contribution

It introduces a joint algorithm-system redesign of GSPN, including a single 2D kernel and compact channel propagation, achieving higher efficiency in vision transformer applications.

Findings

Matches transformer-level accuracy with lower computational cost.

Reduces GPU kernel launches and data transfers significantly.

Effective across image classification and text-to-image synthesis.

Abstract

Efficient vision transformer remains a bottleneck for high-resolution images and long-video related real-world applications. Generalized Spatial Propagation Network (GSPN) addresses this by replacing quadratic self-attention with a line-scan propagation scheme, bringing the cost close to linear in the number of rows or columns, while retaining accuracy. Despite this advancement, the existing GSPN implementation still suffers from (i) heavy overhead due to repeatedly launching GPU kernels, (ii) excessive data transfers from global GPU memory, and (iii) redundant computations caused by maintaining separate propagation weights for each channel. We introduce GSPN-2, a joint algorithm-system redesign. In particular, we eliminate thousands of micro-launches from the previous implementation into one single 2D kernel, explicitly pin one warp to each channel slice, and stage the previous column's activations in shared memory. On the model side, we introduce a compact channel propagation strategy that replaces per-channel matrices, trimming parameters, and align naturally with the affinity map used in transformer attention. Experiments demonstrate GSPN-2's effectiveness across image classification and text-to-image synthesis tasks, matching transformer-level accuracy with significantly lower computational cost. GSPN-2 establishes a new efficiency frontier for modeling global spatial context in vision applications through its unique combination of structured matrix transformations and GPU-optimized implementation. Project page: https://whj363636.github.io/GSPN2/