Never Reset Again: A Mathematical Framework for Continual Inference in Recurrent Neural Networks

TL;DR

This paper introduces a novel adaptive loss function for RNNs that enables continual inference without resets, maintaining accuracy over long sequences and improving streaming application performance.

Contribution

The paper presents a new loss function combining cross-entropy and Kullback-Leibler divergence that allows RNNs to operate continuously without state resets, a significant advancement over existing methods.

Findings

Outperforms reset-based methods in continual tasks

Maintains stable representations over extended sequences

Enhances RNN capabilities for streaming applications

Abstract

Recurrent Neural Networks (RNNs) are widely used for sequential processing but face fundamental limitations with continual inference due to state saturation, requiring disruptive hidden state resets. However, reset-based methods impose synchronization requirements with input boundaries and increase computational costs at inference. To address this, we propose an adaptive loss function that eliminates the need for resets during inference while preserving high accuracy over extended sequences. By combining cross-entropy and Kullback-Leibler divergence, the loss dynamically modulates the gradient based on input informativeness, allowing the network to differentiate meaningful data from noise and maintain stable representations over time. Experimental results demonstrate that our reset-free approach outperforms traditional reset-based methods when applied to a variety of RNNs, particularly in continual tasks, enhancing both the theoretical and practical capabilities of RNNs for streaming applications.