Small Towers Make Big Differences
Yuyan Wang, Zhe Zhao, Bo Dai, Christopher Fifty, Dong Lin, Lichan, Hong, Ed H. Chi

TL;DR
This paper explores the trade-off between efficiency and generalization in multi-task learning, proposing under-parameterized self-auxiliaries to improve Pareto efficiency without sacrificing generalization.
Contribution
It introduces a novel under-parameterized self-auxiliary method that enhances Pareto efficiency in multi-task models while maintaining generalization.
Findings
Small towers of under-parameterized self-auxiliaries significantly improve Pareto efficiency.
The method is task-agnostic and compatible with existing multi-task algorithms.
Empirical results demonstrate notable performance gains across various applications.
Abstract
Multi-task learning aims at solving multiple machine learning tasks at the same time. A good solution to a multi-task learning problem should be generalizable in addition to being Pareto optimal. In this paper, we provide some insights on understanding the trade-off between Pareto efficiency and generalization as a result of parameterization in multi-task deep learning models. As a multi-objective optimization problem, enough parameterization is needed for handling task conflicts in a constrained solution space; however, from a multi-task generalization perspective, over-parameterization undermines the benefit of learning a shared representation which helps harder tasks or tasks with limited training examples. A delicate balance between multi-task generalization and multi-objective optimization is therefore needed for finding a better trade-off between efficiency and generalization. To…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsInfrastructure Resilience and Vulnerability Analysis · Disaster Management and Resilience
