# A hybrid penalty method for a class of optimization problems with multiple rank constraints

## TL;DR

This paper introduces a hybrid penalty method combining penalty and pseudo-projection techniques to efficiently solve optimization problems with multiple rank constraints on Hankel matrices, relevant in system identification and signal processing.

## Contribution

The paper proposes a novel hybrid penalty approach that integrates pseudo-projection methods for handling multiple rank constraints, with convergence analysis and practical efficiency demonstrated.

## Key findings

- Efficient computation of pseudo-projection onto low-rank Hankel matrices using existing software.
- Successful application of the hybrid method to numerical examples showing improved performance.
- Convergence results established for the proposed hybrid penalty method.

## Abstract

In this paper, we consider the problem of minimizing a smooth objective over multiple rank constraints on Hankel-structured matrices. This kind of problems arises in system identification, system theory and signal processing, where the rank constraints are typically "hard constraints". To solve these problems, we propose a hybrid penalty method that combines a penalty method with a post-processing scheme. Specifically, we solve the penalty subproblems until the penalty parameter reaches a given threshold, and then switch to a local alternating "pseudo-projection'' method to further reduce constraint violation. Pseudo-projection is a generalization of the concept of projection. We show that a pseudo-projection onto a {\em single} low-rank Hankel-structured matrix constraint can be computed efficiently by existing softwares such as SLRA (Markovsky and Usevich, 2014), under mild assumptions. We also demonstrate how the penalty subproblems in the hybrid penalty method can be solved by pseudo-projection-based optimization methods, and then present some convergence results for our hybrid penalty method. Finally, the efficiency of our method is illustrated by numerical examples.