Online Parallel Multi-Task Relationship Learning via Alternating Direction Method of Multipliers

TL;DR

This paper introduces an online multi-task learning framework using ADMM that models task relations dynamically, outperforming gradient-based methods in accuracy and efficiency, and extends to decentralized settings for scalability.

Contribution

It proposes a novel ADMM-based online multi-task learning algorithm that models task relations dynamically and is suitable for decentralized distributed environments.

Findings

Outperforms SGD-based methods in accuracy and efficiency.

Effective in both centralized and decentralized architectures.

Demonstrated on synthetic and real-world datasets.

Abstract

Online multi-task learning (OMTL) enhances streaming data processing by leveraging the inherent relations among multiple tasks. It can be described as an optimization problem in which a single loss function is defined for multiple tasks. Existing gradient-descent-based methods for this problem might suffer from gradient vanishing and poor conditioning issues. Furthermore, the centralized setting hinders their application to online parallel optimization, which is vital to big data analytics. Therefore, this study proposes a novel OMTL framework based on the alternating direction multiplier method (ADMM), a recent breakthrough in optimization suitable for the distributed computing environment because of its decomposable and easy-to-implement nature. The relations among multiple tasks are modeled dynamically to fit the constant changes in an online scenario. In a classical distributed computing architecture with a central server, the proposed OMTL algorithm with the ADMM optimizer outperforms SGD-based approaches in terms of accuracy and efficiency. Because the central server might become a bottleneck when the data scale grows, we further tailor the algorithm to a decentralized setting, so that each node can work by only exchanging information with local neighbors. Experimental results on a synthetic and several real-world datasets demonstrate the efficiency of our methods.