EditBoard: Towards a Comprehensive Evaluation Benchmark for Text-Based Video Editing Models

TL;DR

EditBoard is a comprehensive benchmark designed to evaluate text-based video editing models across multiple dimensions, addressing the lack of holistic assessment tools in this emerging field.

Contribution

We introduce EditBoard, the first detailed evaluation benchmark with nine metrics across four dimensions for assessing text-based video editing models.

Findings

Provides a standardized evaluation framework for video editing models.

Includes nine automatic metrics and three new fidelity measures.

Enables detailed analysis of model strengths and weaknesses.

Abstract

The rapid development of diffusion models has significantly advanced AI-generated content (AIGC), particularly in Text-to-Image (T2I) and Text-to-Video (T2V) generation. Text-based video editing, leveraging these generative capabilities, has emerged as a promising field, enabling precise modifications to videos based on text prompts. Despite the proliferation of innovative video editing models, there is a conspicuous lack of comprehensive evaluation benchmarks that holistically assess these models' performance across various dimensions. Existing evaluations are limited and inconsistent, typically summarizing overall performance with a single score, which obscures models' effectiveness on individual editing tasks. To address this gap, we propose EditBoard, the first comprehensive evaluation benchmark for text-based video editing models. EditBoard encompasses nine automatic metrics across four dimensions, evaluating models on four task categories and introducing three new metrics to assess fidelity. This task-oriented benchmark facilitates objective evaluation by detailing model performance and providing insights into each model's strengths and weaknesses. By open-sourcing EditBoard, we aim to standardize evaluation and advance the development of robust video editing models.