# Understanding Feature Selection and Feature Memorization in Recurrent Neural Networks

## TL;DR

This paper introduces the F1B test to analyze how recurrent neural networks handle feature selection and memorization, revealing a conflict that can be mitigated by gating mechanisms or increasing state dimensions.

## Contribution

The paper proposes the F1B test and provides a comparative analysis of different RNN models' capabilities in feature selection and memorization.

## Key findings

- Gated RNNs resolve the conflict via adaptive gating.
- Vanilla RNNs assign different tasks to different state dimensions.
- A fundamental conflict exists between feature selection and memorization.

## Abstract

In this paper, we propose a test, called Flagged-1-Bit (F1B) test, to study the intrinsic capability of recurrent neural networks in sequence learning. Four different recurrent network models are studied both analytically and experimentally using this test. Our results suggest that in general there exists a conflict between feature selection and feature memorization in sequence learning. Such a conflict can be resolved either using a gating mechanism as in LSTM, or by increasing the state dimension as in Vanilla RNN. Gated models resolve this conflict by adaptively adjusting their state-update equations, whereas Vanilla RNN resolves this conflict by assigning different dimensions different tasks. Insights into feature selection and memorization in recurrent networks are given.