AR as an Evaluation Playground: Bridging Metrics and Visual Perception of Computer Vision Models

TL;DR

ARCADE is an augmented reality platform designed to improve computer vision model evaluation by making it more human-centered, reproducible, and capable of revealing perceptual flaws that traditional metrics might miss.

Contribution

The paper introduces ARCADE, a modular AR-based evaluation platform that integrates metric and visual perception assessments for CV models, enhancing evaluation accuracy and reproducibility.

Findings

ARCADE can identify perceptual flaws in CV models missed by traditional metrics.

User study with 15 participants demonstrates ARCADE's usability and effectiveness.

Case studies on depth and lighting estimation validate ARCADE's capability to reveal model weaknesses.

Abstract

Quantitative metrics are central to evaluating computer vision (CV) models, but they often fail to capture real-world performance due to protocol inconsistencies and ground-truth noise. While visual perception studies can complement these metrics, they often require end-to-end systems that are time-consuming to implement and setups that are difficult to reproduce. We systematically summarize key challenges in evaluating CV models and present the design of ARCADE, an evaluation platform that leverages augmented reality (AR) to enable easy, reproducible, and human-centered CV evaluation. ARCADE uses a modular architecture that provides cross-platform data collection, pluggable model inference, and interactive AR tasks, supporting both metric and visual perception evaluation. We demonstrate ARCADE through a user study with 15 participants and case studies on two representative CV tasks, depth and lighting estimation, showing that ARCADE can reveal perceptual flaws in model quality that are often missed by traditional metrics. We also evaluate ARCADE's usability and performance, showing its flexibility as a reliable real-time platform.