Freeze, Diffuse, Decode: Geometry-Aware Adaptation of Pretrained Transformer Embeddings for Antimicrobial Peptide Design

TL;DR

This paper introduces FDD, a diffusion-based method that adaptively fine-tunes pretrained transformer embeddings for antimicrobial peptide design, preserving their geometric structure and improving downstream task performance.

Contribution

The paper proposes a novel diffusion-based framework, FDD, that enhances transfer learning by maintaining geometric integrity of embeddings during adaptation.

Findings

FDD produces low-dimensional, interpretable embeddings.

FDD improves property prediction and retrieval.

FDD enables effective latent-space interpolation.

Abstract

Pretrained transformers provide rich, general-purpose embeddings, which are transferred to downstream tasks. However, current transfer strategies: fine-tuning and probing, either distort the pretrained geometric structure of the embeddings or lack sufficient expressivity to capture task-relevant signals. These issues become even more pronounced when supervised data are scarce. Here, we introduce Freeze, Diffuse, Decode (FDD), a novel diffusion-based framework that adapts pre-trained embeddings to downstream tasks while preserving their underlying geometric structure. FDD propagates supervised signal along the intrinsic manifold of frozen embeddings, enabling a geometry-aware adaptation of the embedding space. Applied to antimicrobial peptide design, FDD yields low-dimensional, predictive, and interpretable representations that support property prediction, retrieval, and latent-space interpolation.