Towards Robust Out-of-Distribution Generalization: Data Augmentation and Neural Architecture Search Approaches

TL;DR

This paper explores methods to improve deep learning robustness against out-of-distribution data by disentangling features, applying gradient-based data augmentation, and optimizing neural architectures specifically for OoD scenarios.

Contribution

It introduces a novel feature disentanglement technique, gradient-based augmentation, and a neural architecture search method tailored for OoD robustness.

Findings

The proposed feature disentanglement improves OoD generalization.

Gradient-based augmentation enhances robustness to distribution shifts.

Architecture search guided by synthetic OoD data yields more robust models.

Abstract

Deep learning has been demonstrated with tremendous success in recent years. Despite so, its performance in practice often degenerates drastically when encountering out-of-distribution (OoD) data, i.e. training and test data are sampled from different distributions. In this thesis, we study ways toward robust OoD generalization for deep learning, i.e., its performance is not susceptible to distribution shift in the test data. We first propose a novel and effective approach to disentangle the spurious correlation between features that are not essential for recognition. It employs decomposed feature representation by orthogonalizing the two gradients of losses for category and context branches. Furthermore, we perform gradient-based augmentation on context-related features (e.g., styles, backgrounds, or scenes of target objects) to improve the robustness of learned representations. Results show that our approach generalizes well for different distribution shifts. We then study the problem of strengthening neural architecture search in OoD scenarios. We propose to optimize the architecture parameters that minimize the validation loss on synthetic OoD data, under the condition that corresponding network parameters minimize the training loss. Moreover, to obtain a proper validation set, we learn a conditional generator by maximizing their losses computed by different neural architectures. Results show that our approach effectively discovers robust architectures that perform well for OoD generalization.