Hierarchical Visual Prompt Learning for Continual Video Instance Segmentation

TL;DR

This paper introduces HVPL, a hierarchical visual prompt learning model that effectively mitigates catastrophic forgetting in continual video instance segmentation by leveraging frame-level and video-level prompts and context decoding.

Contribution

The paper proposes a novel hierarchical prompt learning framework with task-specific prompts and orthogonal gradient correction to address continual learning challenges in VIS.

Findings

HVPL outperforms baseline methods in continual VIS tasks.

The orthogonal gradient correction improves retention of old class knowledge.

Video context decoding enhances inter-class relationship modeling.

Abstract

Video instance segmentation (VIS) has gained significant attention for its capability in tracking and segmenting object instances across video frames. However, most of the existing VIS approaches unrealistically assume that the categories of object instances remain fixed over time. Moreover, they experience catastrophic forgetting of old classes when required to continuously learn object instances belonging to new categories. To resolve these challenges, we develop a novel Hierarchical Visual Prompt Learning (HVPL) model that overcomes catastrophic forgetting of previous categories from both frame-level and video-level perspectives. Specifically, to mitigate forgetting at the frame level, we devise a task-specific frame prompt and an orthogonal gradient correction (OGC) module. The OGC module helps the frame prompt encode task-specific global instance information for new classes in each individual frame by projecting its gradients onto the orthogonal feature space of old classes. Furthermore, to address forgetting at the video level, we design a task-specific video prompt and a video context decoder. This decoder first embeds structural inter-class relationships across frames into the frame prompt features, and then propagates task-specific global video contexts from the frame prompt features to the video prompt. Through rigorous comparisons, our HVPL model proves to be more effective than baseline approaches. The code is available at https://github.com/JiahuaDong/HVPL.