Activation Manifold Projection: Liberating Task-Specific Behaviors from LLM Architectures

TL;DR

This paper introduces CAST, a novel framework that enables zero-shot transfer of task-specific behaviors encoded by LoRA adapters across different LLM architectures by learning a direct activation manifold mapping.

Contribution

CAST provides a direct, nonlinear activation space transfer method that outperforms weight-space approaches and enables zero-shot adapter transfer between diverse LLMs.

Findings

CAST achieves 85-95% of LoRA performance after transfer.

It outperforms existing weight-space transfer techniques.

Enables zero-shot transfer between heterogeneous LLM architectures.

Abstract

The proliferation of Large Language Model (LLM) architectures presents a fundamental challenge: valuable, task-specific behaviors learned through fine-tuning methods like Low-Rank Adaptation (LoRA) are effectively trapped within their source model's architecture, herein referred to architectural lock-in. Existing transfer methods attempt to bridge this gap by aligning the static weight spaces of models, a brittle and indirect approach that relies on tenuous correlations between parameter geometries. This paper introduces a fundamentally different and more direct paradigm: the Cartridge Activation Space Transfer (CAST), a novel framework that liberates LoRA-encoded behaviors by learning a direct, nonlinear mapping between the activation manifolds, the geometric structures formed by the model's internal neuron activations, of two distinct LLM architectures. CAST treats a pre-trained LoRA as a frozen "behavioral kernel." It learns a set of lightweight, bidirectional projection heads that translate the target model's activation stream into the source model's latent space, apply the frozen kernel, and project the result back. This process, trained on a general text corpus without any task-specific data, effectively decouples the learned skill from the source architecture. We demonstrate that CAST enables true "zero-shot" translation of any standard LoRA adapter. Our experiments, including transfers between heterogeneous model families like Llama-2 and Mistral, show that CAST-translated adapters achieve 85-95\% of the performance of a LoRA fully retrained on the target model, quantitatively outperforming current weight-space transfer techniques and establishing a new state-of-the-art in model interoperability.