FoldSAE: Learning to Steer Protein Folding Through Sparse Representations

TL;DR

FoldSAE integrates sparse autoencoders with RFdiffusion to interpret and control protein secondary structure formation, enhancing understanding and precision in protein design.

Contribution

This work introduces a novel framework combining SAE with RFdiffusion for interpretability and control of protein folding, which was not previously achieved.

Findings

Uncovered interpretable features related to secondary structures.

Developed a hyperparameter-based steering mechanism for protein design.

Demonstrated improved interpretability of RFdiffusion internal representations.

Abstract

RFdiffusion is a popular and well-established model for generation of protein structures. However, this generative process offers limited insight into its internal representations and how they contribute to the final protein structure. Concurrently, recent work in mechanistic interpretability has successfully used Sparse Autoencoders (SAEs) to discover interpretable features within neural networks. We combine these concepts by applying SAE to the internal representations of RFdiffusion to uncover secondary structure-specific features and establish a relationship between them and generated protein structures. Building on these insights, we introduce a novel steering mechanism that enables precise control of secondary structure formation through a tunable hyperparameter, while simultaneously revealing interpretable block and neuron-level representations within RFdiffusion. Our work pioneers a new framework for making RFdiffusion more interpretable, demonstrating how understanding internal features can be directly translated into precise control over the protein design process.