HAAP: Vision-context Hierarchical Attention Autoregressive with Adaptive Permutation for Scene Text Recognition

TL;DR

The paper introduces HAAP, a novel model for scene text recognition that uses adaptive permutation and hierarchical attention to improve accuracy and efficiency by better capturing visual and contextual dependencies.

Contribution

HAAP employs Implicit Permutation Neurons and Cross-modal Hierarchical Attention to enhance position-context-image interaction, avoiding training oscillations and iterative refinement overhead.

Findings

Achieves state-of-the-art accuracy on multiple datasets.

Reduces complexity and latency compared to previous methods.

Effectively captures global semantic information without iterative refinement.

Abstract

Scene Text Recognition (STR) is challenging in extracting effective character representations from visual data when text is unreadable. Permutation language modeling (PLM) is introduced to refine character predictions by jointly capturing contextual and visual information. However, in PLM, the use of random permutations causes training fit oscillation, and the iterative refinement (IR) operation also introduces additional overhead. To address these issues, this paper proposes the Hierarchical Attention autoregressive Model with Adaptive Permutation (HAAP) to enhance position-context-image interaction capability, improving autoregressive LM generalization. First, we propose Implicit Permutation Neurons (IPN) to generate adaptive attention masks that dynamically exploit token dependencies, enhancing the correlation between visual information and context. Adaptive correlation representation helps the model avoid training fit oscillation. Second, the Cross-modal Hierarchical Attention mechanism (CHA) is introduced to capture the dependencies among position queries, contextual semantics and visual information. CHA enables position tokens to aggregate global semantic information, avoiding the need for IR. Extensive experimental results show that the proposed HAAP achieves state-of-the-art (SOTA) performance in terms of accuracy, complexity, and latency on several datasets.