Hierarchical Optimization-Derived Learning

TL;DR

This paper introduces Hierarchical ODL (HODL), a unified framework that jointly models and analyzes optimization-derived model construction and learning, providing the first theoretical convergence guarantees and demonstrating improved performance in vision and learning tasks.

Contribution

The paper proposes a novel hierarchical framework for optimization-derived learning that couples model construction and learning, with rigorous convergence analysis and practical validation.

Findings

Proves joint convergence of optimization and learning components.

Demonstrates flexibility of HODL in challenging tasks.

Validates theoretical properties through extensive experiments.

Abstract

In recent years, by utilizing optimization techniques to formulate the propagation of deep model, a variety of so-called Optimization-Derived Learning (ODL) approaches have been proposed to address diverse learning and vision tasks. Although having achieved relatively satisfying practical performance, there still exist fundamental issues in existing ODL methods. In particular, current ODL methods tend to consider model construction and learning as two separate phases, and thus fail to formulate their underlying coupling and depending relationship. In this work, we first establish a new framework, named Hierarchical ODL (HODL), to simultaneously investigate the intrinsic behaviors of optimization-derived model construction and its corresponding learning process. Then we rigorously prove the joint convergence of these two sub-tasks, from the perspectives of both approximation quality and stationary analysis. To our best knowledge, this is the first theoretical guarantee for these two coupled ODL components: optimization and learning. We further demonstrate the flexibility of our framework by applying HODL to challenging learning tasks, which have not been properly addressed by existing ODL methods. Finally, we conduct extensive experiments on both synthetic data and real applications in vision and other learning tasks to verify the theoretical properties and practical performance of HODL in various application scenarios.