Training Deep Networks from Zero to Hero: avoiding pitfalls and going beyond

TL;DR

This paper provides a comprehensive tutorial on training deep neural networks, emphasizing techniques to improve performance on small or challenging datasets through various architectural and training strategies.

Contribution

It offers an overview of both basic and recent methods for training deep networks, including architectural innovations and advanced training procedures, tailored for difficult data scenarios.

Findings

Effective data preparation and optimization improve model performance.

Recent architectural choices like transformers and alternative convolutions enhance capabilities.

Advanced training methods such as curriculum and self-supervised learning yield better results.

Abstract

Training deep neural networks may be challenging in real world data. Using models as black-boxes, even with transfer learning, can result in poor generalization or inconclusive results when it comes to small datasets or specific applications. This tutorial covers the basic steps as well as more recent options to improve models, in particular, but not restricted to, supervised learning. It can be particularly useful in datasets that are not as well-prepared as those in challenges, and also under scarce annotation and/or small data. We describe basic procedures: as data preparation, optimization and transfer learning, but also recent architectural choices such as use of transformer modules, alternative convolutional layers, activation functions, wide and deep networks, as well as training procedures including as curriculum, contrastive and self-supervised learning.