Attention-likelihood relationship in transformers

TL;DR

This paper investigates how transformer-based language models represent out-of-context words, revealing a correlation between token likelihood and attention, and showing that unexpected tokens reduce attention to their own information, impacting robustness.

Contribution

It uncovers the relationship between token likelihood and attention in transformers, providing insights into model robustness and behavior with out-of-context words.

Findings

Unexpected tokens lead to reduced attention to their own information.

Likelihood and attention are correlated in transformer models.

Findings have implications for assessing LLM robustness.

Abstract

We analyze how large language models (LLMs) represent out-of-context words, investigating their reliance on the given context to capture their semantics. Our likelihood-guided text perturbations reveal a correlation between token likelihood and attention values in transformer-based language models. Extensive experiments reveal that unexpected tokens cause the model to attend less to the information coming from themselves to compute their representations, particularly at higher layers. These findings have valuable implications for assessing the robustness of LLMs in real-world scenarios. Fully reproducible codebase at https://github.com/Flegyas/AttentionLikelihood.