Robust Optimization over Multiple Domains

TL;DR

This paper introduces a robust optimization framework for training a single machine learning model that performs well across multiple domains, addressing challenges in cloud computing applications.

Contribution

It develops a novel framework that optimizes models over adversarial domain distributions and analyzes convergence rates for both convex and non-convex models.

Findings

Effective in multi-domain visual categorization and digit recognition tasks

Convergence rates established for convex and non-convex models

Robustness improved with regularizers over adversarial distributions

Abstract

In this work, we study the problem of learning a single model for multiple domains. Unlike the conventional machine learning scenario where each domain can have the corresponding model, multiple domains (i.e., applications/users) may share the same machine learning model due to maintenance loads in cloud computing services. For example, a digit-recognition model should be applicable to hand-written digits, house numbers, car plates, etc. Therefore, an ideal model for cloud computing has to perform well at each applicable domain. To address this new challenge from cloud computing, we develop a framework of robust optimization over multiple domains. In lieu of minimizing the empirical risk, we aim to learn a model optimized to the adversarial distribution over multiple domains. Hence, we propose to learn the model and the adversarial distribution simultaneously with the stochastic algorithm for efficiency. Theoretically, we analyze the convergence rate for convex and non-convex models. To our best knowledge, we first study the convergence rate of learning a robust non-convex model with a practical algorithm. Furthermore, we demonstrate that the robustness of the framework and the convergence rate can be further enhanced by appropriate regularizers over the adversarial distribution. The empirical study on real-world fine-grained visual categorization and digits recognition tasks verifies the effectiveness and efficiency of the proposed framework.