NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internals

TL;DR

This paper presents NNsight and NDIF, open-source tools that facilitate scientific analysis of large neural network internals by providing scalable, transparent access without the need for hosting models locally.

Contribution

The authors introduce NNsight and NDIF, novel open-source systems that enable efficient, scalable, and transparent study of large neural networks' internals, bridging a significant gap in AI research.

Findings

Enabled new research methods on large models

Demonstrated scalable inference with NDIF

Benchmarked performance against previous approaches

Abstract

We introduce NNsight and NDIF, technologies that work in tandem to enable scientific study of the representations and computations learned by very large neural networks. NNsight is an open-source system that extends PyTorch to introduce deferred remote execution. The National Deep Inference Fabric (NDIF) is a scalable inference service that executes NNsight requests, allowing users to share GPU resources and pretrained models. These technologies are enabled by the Intervention Graph, an architecture developed to decouple experimental design from model runtime. Together, this framework provides transparent and efficient access to the internals of deep neural networks such as very large language models (LLMs) without imposing the cost or complexity of hosting customized models individually. We conduct a quantitative survey of the machine learning literature that reveals a growing gap in the study of the internals of large-scale AI. We demonstrate the design and use of our framework to address this gap by enabling a range of research methods on huge models. Finally, we conduct benchmarks to compare performance with previous approaches. Code, documentation, and tutorials are available at https://nnsight.net/.