Correction of Transformer-Based Models with Smoothing Pseudo-Projector

TL;DR

This paper introduces a pseudo-projector technique inspired by multigrid methods to improve the robustness and training dynamics of transformer-based models without altering their core architecture.

Contribution

It proposes a novel pseudo-projector method that acts as a hidden-representation corrector, enhancing transformer models' robustness and training efficiency through a multigrid-inspired approach.

Findings

Improves training dynamics of transformer models.

Enhances robustness in text classification tasks.

No adverse effects observed during experiments.

Abstract

The pseudo-projector is a lightweight modification that can be integrated into existing language models and other neural networks without altering their core architecture. It can be viewed as a hidden-representation corrector that reduces sensitivity to noise by suppressing directions induced by label-irrelevant input content. The design is inspired by the multigrid (MG) paradigm, originally developed to accelerate the convergence of iterative solvers for partial differential equations and boundary value problems, and later extended to more general linear systems through algebraic multigrid methods. We refer to the method as a pseudo-projector because its linear prototype corresponds to a strictly idempotent orthogonal projector, whereas the practical formulation employs learnable restriction and prolongation operators and therefore does not, in general, satisfy the properties of an exact orthogonal projection. We evaluate the proposed approach on transformer-based text classification tasks, as well as controlled synthetic benchmarks, demonstrating its effectiveness in improving training dynamics and robustness. Experimental results, together with supporting theoretical heuristics, indicate consistent improvements in training behavior across a range of settings, with no adverse effects observed otherwise. Our next step will be to extend this approach to language models.