Towards Building Non-Fine-Tunable Foundation Models

TL;DR

This paper introduces a novel pre-training framework called Private Mask Pre-Training (PMP) that creates foundation models resistant to unauthorized fine-tuning, maintaining performance while limiting adaptation gains.

Contribution

The paper proposes PMP, a method that uses a private sparse subnetwork mask during pre-training to prevent effective unauthorized fine-tuning of foundation models.

Findings

PMP preserves base model performance.

PMP significantly degrades unauthorized fine-tuning.

The non-fine-tunability strength is adjustable via mask ratio.

Abstract

Open-sourcing foundation models (FMs) enables broad reuse but also exposes model trainers to economic and safety risks from unrestricted downstream fine-tuning. We address this problem by building non-fine-tunable foundation models: models that remain broadly usable in their released form while yielding limited adaptation gains under task-agnostic unauthorized fine-tuning. We propose Private Mask Pre-Training (PMP), a pre-training framework that concentrates representation learning into a sparse subnetwork identified early in training. The binary mask defining this subnetwork is kept private, and only the final dense weights are released. This forces unauthorized fine-tuning without access to the mask to update parameters misaligned with pretraining subspace, inducing an intrinsic mismatch between the fine-tuning objective and the pre-training geometry. We provide theoretical analysis showing that this mismatch destabilizes gradient-based adaptation and bounds fine-tuning gains. Empirical results on large language models demonstrating that PMP preserves base model performance while consistently degrading unauthorized fine-tuning across a wide range of downstream tasks, with the strength of non-fine-tunability controlled by the mask ratio.