HPE-CogVLM: Advancing Vision Language Models with a Head Pose Grounding Task

TL;DR

This paper introduces HPE-CogVLM, a novel vision language model framework that significantly improves head pose estimation accuracy by integrating object detection and HPE tasks through a specialized model merging technique.

Contribution

The paper presents a new LoRA layer-based merging method for VLMs that effectively combines object detection and head pose estimation capabilities, overcoming previous limitations.

Findings

Achieves 31.5% reduction in MAE over state-of-the-art CNN model

Outperforms direct LoRA fine-tuning and task arithmetic merging methods

Successfully integrates HPE and object detection in a unified VLM framework

Abstract

Head pose estimation (HPE) requires a sophisticated understanding of 3D spatial relationships to generate precise yaw, pitch, and roll angles. Previous HPE models, primarily CNN-based, rely on cropped close-up human head images as inputs and often lack robustness in real-world scenario. Vision Language Models (VLMs) can analyze entire images while focusing on specific objects through their attention mechanisms. In this paper, we propose a novel framework to improve the HPE accuracy by leveraging the object detection grounding capability of a VLM, referred to as CogVLM. We empirically find that directly LoRA fine-tuning of this VLM for the HPE task fails to achieve desirable HPE accuracy, while some model merging methods can improve accuracy but frequently produce blended invalid response formats, struggling to handle both object detection and HPE tasks simultaneously. To integrate HPE capability into CogVLM effectively, we develop a novel LoRA layer-based model merging method. This merging approach applies a high cosine similarity threshold and a 'winner-takes-all' layer selection strategy, aligning attention to the HPE task while preserving original object detection knowledge. It successfully resolves issues with blended invalid response formats and improves accuracy. Results show that our HPE-CogVLM achieves a 31.5% reduction in Mean Absolute Error over the current state-of-the-art CNN model, 6DRepNet, in cross-dataset evaluation. Furthermore, HPE-CogVLM outperforms both directly LoRA fine-tuned and task arithmetic-based merged VLMs across all HPE metrics.