A Solution to CVPR'2023 AQTC Challenge: Video Alignment for Multi-Step Inference

TL;DR

This paper presents a novel video alignment method for multi-step inference in egocentric instructional videos, significantly improving AI assistant guidance and achieving second place in the CVPR 2023 AQTC challenge.

Contribution

It introduces an enhanced video alignment approach using VideoCLIP, question grounding, feature reweighting, and GRU-based inference for better task completion.

Findings

Secured 2nd place in CVPR'2023 AQTC challenge.

Demonstrated superior performance over existing methods.

Effective multi-step inference in instructional videos.

Abstract

Affordance-centric Question-driven Task Completion (AQTC) for Egocentric Assistant introduces a groundbreaking scenario. In this scenario, through learning instructional videos, AI assistants provide users with step-by-step guidance on operating devices. In this paper, we present a solution for enhancing video alignment to improve multi-step inference. Specifically, we first utilize VideoCLIP to generate video-script alignment features. Afterwards, we ground the question-relevant content in instructional videos. Then, we reweight the multimodal context to emphasize prominent features. Finally, we adopt GRU to conduct multi-step inference. Through comprehensive experiments, we demonstrate the effectiveness and superiority of our method, which secured the 2nd place in CVPR'2023 AQTC challenge. Our code is available at https://github.com/zcfinal/LOVEU-CVPR23-AQTC.