TL;DR
Neural teleportation, derived from quiver representation theory, is a simple operation that preserves neural network functions while significantly altering the loss landscape, enabling new insights into optimization and training dynamics.
Contribution
This paper introduces neural teleportation, a novel mathematical operation based on representation theory, revealing its effects on loss landscapes and training processes.
Findings
Teleportation preserves network function while changing the loss landscape.
It enables exploration of loss level curves and modifies local minima.
Teleportation boosts gradients and sharpens global minima during training.
Abstract
In this paper, we explore a process called neural teleportation, a mathematical consequence of applying quiver representation theory to neural networks. Neural teleportation "teleports" a network to a new position in the weight space and preserves its function. This phenomenon comes directly from the definitions of representation theory applied to neural networks and it turns out to be a very simple operation that has remarkable properties. We shed light on surprising and counter-intuitive consequences neural teleportation has on the loss landscape. In particular, we show that teleportation can be used to explore loss level curves, that it changes the local loss landscape, sharpens global minima and boosts back-propagated gradients at any moment during the learning process. Our results can be reproduced with the code available here: https://github.com/vitalab/neuralteleportation
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
Methods*Communicated@Fast*How Do I Communicate to Expedia?
