Neuro-Visualizer: An Auto-encoder-based Loss Landscape Visualization Method

TL;DR

Neuro-Visualizer is a novel auto-encoder-based method for visualizing neural network loss landscapes, offering higher fidelity and flexibility than traditional linear techniques, and providing new insights into model training.

Contribution

The paper introduces Neuro-Visualizer, a non-linear visualization approach that overcomes limitations of linear methods, enhancing understanding of neural network loss landscapes.

Findings

Outperforms existing linear and non-linear baselines

Provides insights that corroborate or challenge existing claims

Demonstrates effectiveness across various machine learning problems

Abstract

In recent years, there has been a growing interest in visualizing the loss landscape of neural networks. Linear landscape visualization methods, such as principal component analysis, have become widely used as they intuitively help researchers study neural networks and their training process. However, these linear methods suffer from limitations and drawbacks due to their lack of flexibility and low fidelity at representing the high dimensional landscape. In this paper, we present a novel auto-encoder-based non-linear landscape visualization method called Neuro-Visualizer that addresses these shortcoming and provides useful insights about neural network loss landscapes. To demonstrate its potential, we run experiments on a variety of problems in two separate applications of knowledge-guided machine learning (KGML). Our findings show that Neuro-Visualizer outperforms other linear and non-linear baselines and helps corroborate, and sometime challenge, claims proposed by machine learning community. All code and data used in the experiments of this paper are available at an anonymous link https://anonymous.4open.science/r/NeuroVisualizer-FDD6