Deep Learning of Representations: Looking Forward

TL;DR

This paper discusses the future challenges and directions in deep learning, focusing on scaling models, improving optimization, inference, sampling, and disentangling factors of variation.

Contribution

It identifies key challenges in scaling and optimizing deep learning models and proposes future research directions to address these issues.

Findings

Highlighting the importance of scaling deep models to larger datasets.

Addressing optimization difficulties like ill-conditioning and local minima.

Suggesting new directions for inference, sampling, and disentangling factors.

Abstract

Deep learning research aims at discovering learning algorithms that discover multiple levels of distributed representations, with higher levels representing more abstract concepts. Although the study of deep learning has already led to impressive theoretical results, learning algorithms and breakthrough experiments, several challenges lie ahead. This paper proposes to examine some of these challenges, centering on the questions of scaling deep learning algorithms to much larger models and datasets, reducing optimization difficulties due to ill-conditioning or local minima, designing more efficient and powerful inference and sampling procedures, and learning to disentangle the factors of variation underlying the observed data. It also proposes a few forward-looking research directions aimed at overcoming these challenges.