Deep Kernel Methods Learn Better: From Cards to Process Optimization

TL;DR

This paper compares deep kernel learning and variational autoencoders, showing that active learning in DKL creates more optimized, smooth latent spaces beneficial for complex domain-specific optimization tasks.

Contribution

It demonstrates that deep kernel learning with active learning produces superior latent manifolds for optimization compared to traditional VAEs.

Findings

DKL with active learning yields more compact latent spaces.

Active learning enhances the smoothness and optimization readiness of latent manifolds.

Results extend from simple cards data to complex physical system trajectories.

Abstract

The ability of deep learning methods to perform classification and regression tasks relies heavily on their capacity to uncover manifolds in high-dimensional data spaces and project them into low-dimensional representation spaces. In this study, we investigate the structure and character of the manifolds generated by classical variational autoencoder (VAE) approaches and deep kernel learning (DKL). In the former case, the structure of the latent space is determined by the properties of the input data alone, while in the latter, the latent manifold forms as a result of an active learning process that balances the data distribution and target functionalities. We show that DKL with active learning can produce a more compact and smooth latent space which is more conducive to optimization compared to previously reported methods, such as the VAE. We demonstrate this behavior using a simple cards data set and extend it to the optimization of domain-generated trajectories in physical systems. Our findings suggest that latent manifolds constructed through active learning have a more beneficial structure for optimization problems, especially in feature-rich target-poor scenarios that are common in domain sciences, such as materials synthesis, energy storage, and molecular discovery. The jupyter notebooks that encapsulate the complete analysis accompany the article.