# DNN Architecture for High Performance Prediction on Natural Videos Loses Submodule's Ability to Learn Discrete-World Dataset

## TL;DR

This paper investigates the limitations of the PredNet neural architecture in generalizing predictive capabilities across different types of video datasets, revealing a trade-off between high performance on natural videos and artificial rule-based datasets.

## Contribution

It demonstrates that PredNet's architecture enables high performance on natural videos but impairs learning on artificial datasets like the Game of Life, highlighting a trade-off in neural design.

## Key findings

- PredNet generalizes well across different natural video datasets.
- PredNet fails to learn artificial datasets like the Game of Life.
- A submodule trained alone can learn the Game of Life effectively.

## Abstract

Is cognition a collection of loosely connected functions tuned to different tasks, or can there be a general learning algorithm? If such an hypothetical general algorithm did exist, tuned to our world, could it adapt seamlessly to a world with different laws of nature? We consider the theory that predictive coding is such a general rule, and falsify it for one specific neural architecture known for high-performance predictions on natural videos and replication of human visual illusions: PredNet. Our results show that PredNet's high performance generalizes without retraining on a completely different natural video dataset. Yet PredNet cannot be trained to reach even mediocre accuracy on an artificial video dataset created with the rules of the Game of Life (GoL). We also find that a submodule of PredNet, a Convolutional Neural Network trained alone, reaches perfect accuracy on the GoL while being mediocre for natural videos, showing that PredNet's architecture itself is responsible for both the high performance on natural videos and the loss of performance on the GoL. Just as humans cannot predict the dynamics of the GoL, our results suggest that there might be a trade-off between high performance on sensory inputs with different sets of rules.