# Differentiable Grammars for Videos

## TL;DR

This paper introduces a differentiable algorithm that learns formal regular grammars directly from videos, enabling interpretable, generative models that improve activity forecasting accuracy on challenging datasets.

## Contribution

It presents a novel, fully differentiable method for learning formal grammars from continuous video data, enhancing interpretability and predictive performance.

## Key findings

- Outperforms state-of-the-art on several datasets
- Provides more accurate activity forecasting
- Generates interpretable sequential structures

## Abstract

This paper proposes a novel algorithm which learns a formal regular grammar from real-world continuous data, such as videos. Learning latent terminals, non-terminals, and production rules directly from continuous data allows the construction of a generative model capturing sequential structures with multiple possibilities. Our model is fully differentiable, and provides easily interpretable results which are important in order to understand the learned structures. It outperforms the state-of-the-art on several challenging datasets and is more accurate for forecasting future activities in videos. We plan to open-source the code. https://sites.google.com/view/differentiable-grammars