Hierarchically Robust Representation Learning

TL;DR

This paper introduces a hierarchically robust optimization approach to learn deep features that maintain performance across different data distributions, addressing the limitations of empirical risk minimization.

Contribution

It proposes a novel distributionally robust optimization framework with Wasserstein constraints to enhance the generality of deep features for diverse tasks.

Findings

Robust features outperform standard features on various benchmarks

The method improves transferability across different data distributions

Experimental results validate the effectiveness of the proposed approach

Abstract

With the tremendous success of deep learning in visual tasks, the representations extracted from intermediate layers of learned models, that is, deep features, attract much attention of researchers. Previous empirical analysis shows that those features can contain appropriate semantic information. Therefore, with a model trained on a large-scale benchmark data set (e.g., ImageNet), the extracted features can work well on other tasks. In this work, we investigate this phenomenon and demonstrate that deep features can be suboptimal due to the fact that they are learned by minimizing the empirical risk. When the data distribution of the target task is different from that of the benchmark data set, the performance of deep features can degrade. Hence, we propose a hierarchically robust optimization method to learn more generic features. Considering the example-level and concept-level robustness simultaneously, we formulate the problem as a distributionally robust optimization problem with Wasserstein ambiguity set constraints, and an efficient algorithm with the conventional training pipeline is proposed. Experiments on benchmark data sets demonstrate the effectiveness of the robust deep representations.