Training Efficiency and Robustness in Deep Learning

TL;DR

This paper explores methods to enhance training efficiency and robustness in deep learning, including data prioritization, optimization improvements, and adversarial robustness strategies, with theoretical and practical insights.

Contribution

It introduces novel techniques like hard negative mining, redundancy-aware sampling, and gradient clustering, and provides theoretical analysis of robustness in linear models.

Findings

Prioritizing informative data accelerates convergence and improves generalization.

Hard negative mining adds no computational overhead to training.

Optimal robustness in linear models depends on choice of optimizer, regularization, or architecture.

Abstract

Deep Learning has revolutionized machine learning and artificial intelligence, achieving superhuman performance in several standard benchmarks. It is well-known that deep learning models are inefficient to train; they learn by processing millions of training data multiple times and require powerful computational resources to process large batches of data in parallel at the same time rather than sequentially. Deep learning models also have unexpected failure modes; they can be fooled into misbehaviour, producing unexpectedly incorrect predictions. In this thesis, we study approaches to improve the training efficiency and robustness of deep learning models. In the context of learning visual-semantic embeddings, we find that prioritizing learning on more informative training data increases convergence speed and improves generalization performance on test data. We formalize a simple trick called hard negative mining as a modification to the learning objective function with no computational overhead. Next, we seek improvements to optimization speed in general-purpose optimization methods in deep learning. We show that a redundancy-aware modification to the sampling of training data improves the training speed and develops an efficient method for detecting the diversity of training signal, namely, gradient clustering. Finally, we study adversarial robustness in deep learning and approaches to achieve maximal adversarial robustness without training with additional data. For linear models, we prove guaranteed maximal robustness achieved only by appropriate choice of the optimizer, regularization, or architecture.