Trapped by simplicity: When Transformers fail to learn from noisy features

TL;DR

This paper investigates the ability of transformers to learn noise-robust boolean functions, revealing their limitations due to simplicity bias and sensitivity issues, and proposes methods to mitigate these failures.

Contribution

It demonstrates that transformers can learn certain noise-robust functions but struggle with others, and introduces techniques to overcome their bias towards simpler solutions.

Findings

Transformers succeed at noise-robust learning for sparse parity and majority functions.

Transformers fail at learning random k-juntas with noisy features.

Adding a sensitivity-penalizing loss improves transformers' ability to learn noisy functions.

Abstract

Noise is ubiquitous in data used to train large language models, but it is not well understood whether these models are able to correctly generalize to inputs generated without noise. Here, we study noise-robust learning: are transformers trained on data with noisy features able to find a target function that correctly predicts labels for noiseless features? We show that transformers succeed at noise-robust learning for a selection of $k$-sparse parity and majority functions, compared to LSTMs which fail at this task for even modest feature noise. However, we find that transformers typically fail at noise-robust learning of random $k$-juntas, especially when the boolean sensitivity of the optimal solution is smaller than that of the target function. We argue that this failure is due to a combination of two factors: transformers' bias toward simpler functions, combined with an observation that the optimal function for noise-robust learning typically has lower sensitivity than the target function for random boolean functions. We test this hypothesis by exploiting transformers' simplicity bias to trap them in an incorrect solution, but show that transformers can escape this trap by training with an additional loss term penalizing high-sensitivity solutions. Overall, we find that transformers are particularly ineffective for learning boolean functions in the presence of feature noise.