Fast Line Search for Multi-Task Learning

TL;DR

This paper introduces a novel line search method for multi-task learning that uses latent representation space to find step sizes, resulting in faster and more accurate solutions without significant computational overhead.

Contribution

The paper proposes a new line search algorithm based on latent space for multi-task learning, improving convergence speed and accuracy over traditional methods.

Findings

The latent space line search outperforms classical backtracking in accuracy and speed.

The proposed method maintains competitive computational time.

It achieves better results than constant learning rate methods.

Abstract

Multi-task learning is a powerful method for solving several tasks jointly by learning robust representation. Optimization of the multi-task learning model is a more complex task than a single-task due to task conflict. Based on theoretical results, convergence to the optimal point is guaranteed when step size is chosen through line search. But, usually, line search for the step size is not the best choice due to the large computational time overhead. We propose a novel idea for line search algorithms in multi-task learning. The idea is to use latent representation space instead of parameter space for finding step size. We examined this idea with backtracking line search. We compare this fast backtracking algorithm with classical backtracking and gradient methods with a constant learning rate on MNIST, CIFAR-10, Cityscapes tasks. The systematic empirical study showed that the proposed method leads to more accurate and fast solution, than the traditional backtracking approach and keep competitive computational time and performance compared to the constant learning rate method.