Likelihood-guided Regularization in Attention Based Models

TL;DR

This paper introduces a likelihood-guided variational Ising regularization method for Vision Transformers, improving generalization, interpretability, and uncertainty quantification through structured Bayesian sparsification.

Contribution

It proposes a novel Ising-based regularization framework that adaptively prunes parameters and enhances model performance over traditional methods.

Findings

Improved accuracy and generalization on benchmark datasets.

Enhanced interpretability and structured feature selection.

Better-calibrated probability estimates and uncertainty quantification.

Abstract

The transformer architecture has demonstrated strong performance in classification tasks involving structured and high-dimensional data. However, its success often hinges on large- scale training data and careful regularization to prevent overfitting. In this paper, we intro- duce a novel likelihood-guided variational Ising-based regularization framework for Vision Transformers (ViTs), which simultaneously enhances model generalization and dynamically prunes redundant parameters. The proposed variational Ising-based regularization approach leverages Bayesian sparsification techniques to impose structured sparsity on model weights, allowing for adaptive architecture search during training. Unlike traditional dropout-based methods, which enforce fixed sparsity patterns, the variational Ising-based regularization method learns task-adaptive regularization, improving both efficiency and interpretability. We evaluate our approach on benchmark vision datasets, including MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100, demonstrating improved generalization under sparse, complex data and allowing for principled uncertainty quantification on both weights and selection parameters. Additionally, we show that the Ising regularizer leads to better-calibrated probability estimates and structured feature selection through uncertainty-aware attention mechanisms. Our results highlight the effectiveness of structured Bayesian sparsification in enhancing transformer-based architectures, offering a principled alternative to standard regularization techniques.