Lattice-Based Pruning in Recurrent Neural Networks via Poset Modeling

TL;DR

This paper introduces a lattice-based pruning method for RNNs that models their structure as posets, enabling more effective reduction of network complexity while maintaining high accuracy.

Contribution

It presents a novel poset and lattice framework for RNN pruning, capturing structural properties beyond weight magnitude for improved network sparsity and performance.

Findings

Moderate pruning retains over 98% accuracy on MNIST.

Aggressive pruning increases sparsity with minimal accuracy loss.

Structural pruning outperforms magnitude-based methods in preserving functionality.

Abstract

Recurrent neural networks (RNNs) are central to sequence modeling tasks, yet their high computational complexity poses challenges for scalability and real-time deployment. Traditional pruning techniques, predominantly based on weight magnitudes, often overlook the intrinsic structural properties of these networks. We introduce a novel framework that models RNNs as partially ordered sets (posets) and constructs corresponding dependency lattices. By identifying meet irreducible neurons, our lattice-based pruning algorithm selectively retains critical connections while eliminating redundant ones. The method is implemented using both binary and continuous-valued adjacency matrices to capture different aspects of network connectivity. Evaluated on the MNIST dataset, our approach exhibits a clear trade-off between sparsity and classification accuracy. Moderate pruning maintains accuracy above 98%, while aggressive pruning achieves higher sparsity with only a modest performance decline. Unlike conventional magnitude-based pruning, our method leverages the structural organization of RNNs, resulting in more effective preservation of functional connectivity and improved efficiency in multilayer networks with top-down feedback. The proposed lattice-based pruning framework offers a rigorous and scalable approach for reducing RNN complexity while sustaining robust performance, paving the way for more efficient hierarchical models in both machine learning and computational neuroscience.