Rethink Deep Learning with Invariance in Data Representation

TL;DR

This paper discusses the importance of invariance in data representations, highlighting its historical significance, current neglect in deep learning, and its resurgence through Geometric Deep Learning for improved robustness and interpretability.

Contribution

It provides a historical perspective on invariance in data representations and identifies research challenges, promising directions, and applications in Geometric Deep Learning.

Findings

Invariance principles are crucial for robust data representations.

Deep learning has historically neglected invariance, leading to limitations.

Geometric Deep Learning reintroduces invariance for improved system performance.

Abstract

Integrating invariance into data representations is a principled design in intelligent systems and web applications. Representations play a fundamental role, where systems and applications are both built on meaningful representations of digital inputs (rather than the raw data). In fact, the proper design/learning of such representations relies on priors w.r.t. the task of interest. Here, the concept of symmetry from the Erlangen Program may be the most fruitful prior -- informally, a symmetry of a system is a transformation that leaves a certain property of the system invariant. Symmetry priors are ubiquitous, e.g., translation as a symmetry of the object classification, where object category is invariant under translation. The quest for invariance is as old as pattern recognition and data mining itself. Invariant design has been the cornerstone of various representations in the era before deep learning, such as the SIFT. As we enter the early era of deep learning, the invariance principle is largely ignored and replaced by a data-driven paradigm, such as the CNN. However, this neglect did not last long before they encountered bottlenecks regarding robustness, interpretability, efficiency, and so on. The invariance principle has returned in the era of rethinking deep learning, forming a new field known as Geometric Deep Learning (GDL). In this tutorial, we will give a historical perspective of the invariance in data representations. More importantly, we will identify those research dilemmas, promising works, future directions, and web applications.