An Ensemble Approach to Short-form Video Quality Assessment Using Multimodal LLM

TL;DR

This paper proposes an ensemble method combining multimodal large language models and traditional BVQA models to improve short-form video quality assessment, demonstrating enhanced generalization and revealing content-dependent model contributions.

Contribution

It introduces a novel ensemble approach that adaptively integrates MLLM and BVQA predictions, addressing challenges in diverse short-form video quality assessment.

Findings

Ensemble approach outperforms individual models in generalization

Preprocessing impacts MLLM performance significantly

Content-aware weights reveal limitations of existing BVQA models

Abstract

The rise of short-form videos, characterized by diverse content, editing styles, and artifacts, poses substantial challenges for learning-based blind video quality assessment (BVQA) models. Multimodal large language models (MLLMs), renowned for their superior generalization capabilities, present a promising solution. This paper focuses on effectively leveraging a pretrained MLLM for short-form video quality assessment, regarding the impacts of pre-processing and response variability, and insights on combining the MLLM with BVQA models. We first investigated how frame pre-processing and sampling techniques influence the MLLM's performance. Then, we introduced a lightweight learning-based ensemble method that adaptively integrates predictions from the MLLM and state-of-the-art BVQA models. Our results demonstrated superior generalization performance with the proposed ensemble approach. Furthermore, the analysis of content-aware ensemble weights highlighted that some video characteristics are not fully represented by existing BVQA models, revealing potential directions to improve BVQA models further.