FuNNscope: Visual microscope for interactively exploring the loss landscape of fully connected neural networks

TL;DR

This paper introduces FuNNscope, an interactive visualization tool that explores the loss landscape of small neural networks, providing insights into their geometry and training dynamics through novel 1D and 2D slicing methods.

Contribution

It extends existing visualization techniques with interpretable charts and demonstrates their effectiveness on small models, enabling interactive exploration of loss landscapes.

Findings

Identified symmetries around the zero vector

Analyzed layer influences on the global landscape

Observed how gradient descent navigates high-loss obstacles

Abstract

Despite their effective use in various fields, many aspects of neural networks are poorly understood. One important way to investigate the characteristics of neural networks is to explore the loss landscape. However, most models produce a high-dimensional non-convex landscape which is difficult to visualize. We discuss and extend existing visualization methods based on 1D- and 2D slicing with a novel method that approximates the actual loss landscape geometry by using charts with interpretable axes. Based on the assumption that observations on small neural networks can generalize to more complex systems and provide us with helpful insights, we focus on small models in the range of a few dozen weights, which enables computationally cheap experiments and the use of an interactive dashboard. We observe symmetries around the zero vector, the influence of different layers on the global landscape, the different weight sensitivities around a minimizer, and how gradient descent navigates high-loss obstacles. The user study resulted in an average SUS (System Usability Scale) score with suggestions for improvement and opened up a number of possible application scenarios, such as autoencoders and ensemble networks.