Weight Space Representation Learning via Neural Field Adaptation

TL;DR

This paper explores using weights as representations in neural fields, demonstrating that low-rank adaptation techniques like LoRA can induce meaningful structure in weight space, improving generative quality.

Contribution

It introduces a novel approach of constraining weight space with pre-trained models and LoRA, leading to enhanced representation quality and generation performance.

Findings

LoRA weights achieve high-quality representations.

Multiplicative LoRA weights improve generative results.

Structured weight space exhibits semantic properties.

Abstract

In this work, we investigate the potential of weights to serve as effective representations, focusing on neural fields. Our key insight is that constraining the optimization space through a pre-trained base model and low-rank adaptation (LoRA) can induce structure in weight space. Across reconstruction, generation, and analysis tasks on 2D and 3D data, we find that multiplicative LoRA weights achieve high representation quality while exhibiting distinctiveness and semantic structure. When used with latent diffusion models, multiplicative LoRA weights enable higher-quality generation than existing weight-space methods.