Can Transformers Break Encryption Schemes via In-Context Learning?

TL;DR

This paper investigates whether transformer-based language models can learn and break classical encryption schemes like mono-alphabetic and Vigenère ciphers through in-context learning, demonstrating their potential to infer hidden substitution mappings from few examples.

Contribution

It introduces a novel application of in-context learning to cryptographic functions, evaluating transformers' ability to decode classical ciphers from limited data.

Findings

Transformers can infer substitution ciphers from few examples.

Models successfully decode new cipher texts after training on small datasets.

The approach highlights the potential of ICL for structured inference tasks in cryptography.

Abstract

In-context learning (ICL) has emerged as a powerful capability of transformer-based language models, enabling them to perform tasks by conditioning on a small number of examples presented at inference time, without any parameter updates. Prior work has shown that transformers can generalize over simple function classes like linear functions, decision trees, even neural networks, purely from context, focusing on numerical or symbolic reasoning over underlying well-structured functions. Instead, we propose a novel application of ICL into the domain of cryptographic function learning, specifically focusing on ciphers such as mono-alphabetic substitution and Vigen\`ere ciphers, two classes of private-key encryption schemes. These ciphers involve a fixed but hidden bijective mapping between plain text and cipher text characters. Given a small set of (cipher text, plain text) pairs, the goal is for the model to infer the underlying substitution and decode a new cipher text word. This setting poses a structured inference challenge, which is well-suited for evaluating the inductive biases and generalization capabilities of transformers under the ICL paradigm. Code is available at https://github.com/adistomar/CS182-project.