Comgra: A Tool for Analyzing and Debugging Neural Networks

TL;DR

Comgra is an open-source Python library that facilitates in-depth inspection, debugging, and interpretability of neural networks by visualizing internal activations, gradients, and training dynamics within a user-friendly GUI.

Contribution

It introduces a novel, comprehensive tool for analyzing neural networks' internal states, enhancing debugging and interpretability workflows.

Findings

Enables visualization of internal activations and gradients

Supports comparison of training stages and individual samples

Facilitates rapid hypothesis testing without retraining

Abstract

Neural Networks are notoriously difficult to inspect. We introduce comgra, an open source python library for use with PyTorch. Comgra extracts data about the internal activations of a model and organizes it in a GUI (graphical user interface). It can show both summary statistics and individual data points, compare early and late stages of training, focus on individual samples of interest, and visualize the flow of the gradient through the network. This makes it possible to inspect the model's behavior from many different angles and save time by rapidly testing different hypotheses without having to rerun it. Comgra has applications for debugging, neural architecture design, and mechanistic interpretability. We publish our library through Python Package Index (PyPI) and provide code, documentation, and tutorials at https://github.com/FlorianDietz/comgra.