Explicit and data-Efficient Encoding via Gradient Flow

TL;DR

This paper presents a decoder-only, gradient flow-based method for explicit, data-efficient encoding that avoids the limitations of traditional autoencoders, especially beneficial in physical sciences with scarce data.

Contribution

It introduces a novel ODE-based encoding approach using gradient flow, eliminating the need for encoder inversion and improving data efficiency in scientific applications.

Findings

Outperforms traditional autoencoders in data-scarce scenarios

Uses adaptive solvers for robustness with stiff ODEs

Achieves faster convergence with a second-order ODE variant

Abstract

The autoencoder model typically uses an encoder to map data to a lower dimensional latent space and a decoder to reconstruct it. However, relying on an encoder for inversion can lead to suboptimal representations, particularly limiting in physical sciences where precision is key. We introduce a decoder-only method using gradient flow to directly encode data into the latent space, defined by ordinary differential equations (ODEs). This approach eliminates the need for approximate encoder inversion. We train the decoder via the adjoint method and show that costly integrals can be avoided with minimal accuracy loss. Additionally, we propose a $2^{nd}$ order ODE variant, approximating Nesterov's accelerated gradient descent for faster convergence. To handle stiff ODEs, we use an adaptive solver that prioritizes loss minimization, improving robustness. Compared to traditional autoencoders, our method demonstrates explicit encoding and superior data efficiency, which is crucial for data-scarce scenarios in the physical sciences. Furthermore, this work paves the way for integrating machine learning into scientific workflows, where precise and efficient encoding is critical. \footnote{The code for this work is available at \url{https://github.com/k-flouris/gfe}.}