Untwisting RoPE: Frequency Control for Shared Attention in DiTs

TL;DR

This paper analyzes Rotary Positional Embeddings (RoPE) in transformer models, revealing that high-frequency components cause unintended reference copying in shared attention, and proposes a frequency modulation method to improve style transfer.

Contribution

It provides a frequency-based analysis of RoPE, identifies the cause of reference copying, and introduces a modulation technique to control attention behavior in diffusion models.

Findings

Frequency components of RoPE influence attention behavior.

Modulating RoPE frequencies reduces reference copying.

Enhanced style transfer without content duplication.

Abstract

Positional encodings are essential to transformer-based generative models, yet their behavior in multimodal and attention-sharing settings is not fully understood. In this work, we present a principled analysis of Rotary Positional Embeddings (RoPE), showing that RoPE naturally decomposes into frequency components with distinct positional sensitivities. We demonstrate that this frequency structure explains why shared-attention mechanisms, where a target image is generated while attending to tokens from a reference image, can lead to reference copying, in which the model reproduces content from the reference instead of extracting only its stylistic cues. Our analysis reveals that the high-frequency components of RoPE dominate the attention computation, forcing queries to attend mainly to spatially aligned reference tokens and thereby inducing this unintended copying behavior. Building on these insights, we introduce a method for selectively modulating RoPE frequency bands so that attention reflects semantic similarity rather than strict positional alignment. Applied to modern transformer-based diffusion architectures, where all tokens share attention, this modulation restores stable and meaningful shared attention. As a result, it enables effective control over the degree of style transfer versus content copying, yielding a proper style-aligned generation process in which stylistic attributes are transferred without duplicating reference content.