# Modelling Identity Rules with Neural Networks

## TL;DR

This paper demonstrates that standard neural networks struggle with learning identity-based abstract patterns, and introduces Relation Based Pattern (RBP) extensions that enable neural networks to successfully learn and generalize these patterns.

## Contribution

The paper proposes RBP extensions to neural networks that effectively learn and generalize identity-based patterns, addressing a key limitation of standard architectures.

## Key findings

- RBP-augmented neural networks achieve perfect classification on synthetic patterns.
- RBP improves performance on real-world sequence prediction tasks.
- Standard neural networks fail to generalize identity rules, confirmed by experiments.

## Abstract

In this paper, we show that standard feed-forward and recurrent neural networks fail to learn abstract patterns based on identity rules. We propose Relation Based Pattern (RBP) extensions to neural network structures that solve this problem and answer, as well as raise, questions about integrating structures for inductive bias into neural networks. Examples of abstract patterns are the sequence patterns ABA and ABB where A or B can be any object. These were introduced by Marcus et al (1999) who also found that 7 month old infants recognise these patterns in sequences that use an unfamiliar vocabulary while simple recurrent neural networks do not.This result has been contested in the literature but it is confirmed by our experiments. We also show that the inability to generalise extends to different, previously untested, settings. We propose a new approach to modify standard neural network architectures, called Relation Based Patterns (RBP) with different variants for classification and prediction. Our experiments show that neural networks with the appropriate RBP structure achieve perfect classification and prediction performance on synthetic data, including mixed concrete and abstract patterns. RBP also improves neural network performance in experiments with real-world sequence prediction tasks. We discuss these finding in terms of challenges for neural network models and identify consequences from this result in terms of developing inductive biases for neural network learning.