Online Alternating Direction Method

TL;DR

This paper introduces online algorithms based on the alternating directions method (ADM) for large-scale optimization, providing convergence and regret bounds in online settings with feasible and infeasible solutions.

Contribution

It develops a new proof technique for ADM with O(1/T) convergence and extends ADM to online optimization with regret analysis for both feasible and infeasible cases.

Findings

Established regret bounds for objective and constraints

Demonstrated O(1/T) convergence rate for ADM in batch setting

Preliminary experiments show effective online performance

Abstract

Online optimization has emerged as powerful tool in large scale optimization. In this paper, we introduce efficient online algorithms based on the alternating directions method (ADM). We introduce a new proof technique for ADM in the batch setting, which yields the O(1/T) convergence rate of ADM and forms the basis of regret analysis in the online setting. We consider two scenarios in the online setting, based on whether the solution needs to lie in the feasible set or not. In both settings, we establish regret bounds for both the objective function as well as constraint violation for general and strongly convex functions. Preliminary results are presented to illustrate the performance of the proposed algorithms.