From MLP to NeoMLP: Leveraging Self-Attention for Neural Fields

TL;DR

This paper introduces NeoMLP, a novel neural network architecture inspired by self-attention mechanisms, designed to improve neural fields for encoding complex signals and enhancing downstream task performance.

Contribution

NeoMLP transforms traditional MLPs into a complete graph with self-attention, enabling scalable conditioning and superior performance in neural field applications.

Findings

Effective fitting of high-resolution multi-modal data.

Outperforms state-of-the-art methods in downstream tasks.

Open-source implementation available.

Abstract

Neural fields (NeFs) have recently emerged as a state-of-the-art method for encoding spatio-temporal signals of various modalities. Despite the success of NeFs in reconstructing individual signals, their use as representations in downstream tasks, such as classification or segmentation, is hindered by the complexity of the parameter space and its underlying symmetries, in addition to the lack of powerful and scalable conditioning mechanisms. In this work, we draw inspiration from the principles of connectionism to design a new architecture based on MLPs, which we term NeoMLP. We start from an MLP, viewed as a graph, and transform it from a multi-partite graph to a complete graph of input, hidden, and output nodes, equipped with high-dimensional features. We perform message passing on this graph and employ weight-sharing via self-attention among all the nodes. NeoMLP has a built-in mechanism for conditioning through the hidden and output nodes, which function as a set of latent codes, and as such, NeoMLP can be used straightforwardly as a conditional neural field. We demonstrate the effectiveness of our method by fitting high-resolution signals, including multi-modal audio-visual data. Furthermore, we fit datasets of neural representations, by learning instance-specific sets of latent codes using a single backbone architecture, and then use them for downstream tasks, outperforming recent state-of-the-art methods. The source code is open-sourced at https://github.com/mkofinas/neomlp.