ShiftedBronzes: Benchmarking and Analysis of Domain Fine-Grained Classification in Open-World Settings

TL;DR

This paper introduces ShiftedBronzes, a benchmark dataset for fine-grained bronze ware dating, and analyzes the performance of various out-of-distribution detection methods in open-world, domain-specific scenarios.

Contribution

It presents a new benchmark dataset, ShiftedBronzes, with diverse OOD data for bronze ware dating, and provides comprehensive analysis of OOD detection methods' behaviors in this domain.

Findings

VLM-based and generation-based methods show distinct behaviors on bronze ware data.

General OOD detection methods face challenges in domain-specific tasks.

Benchmark results validate and extend previous conclusions on OOD detection techniques.

Abstract

In real-world applications across specialized domains, addressing complex out-of-distribution (OOD) challenges is a common and significant concern. In this study, we concentrate on the task of fine-grained bronze ware dating, a critical aspect in the study of ancient Chinese history, and developed a benchmark dataset named ShiftedBronzes. By extensively expanding the bronze Ding dataset, ShiftedBronzes incorporates two types of bronze ware data and seven types of OOD data, which exhibit distribution shifts commonly encountered in bronze ware dating scenarios. We conduct benchmarking experiments on ShiftedBronzes and five commonly used general OOD datasets, employing a variety of widely adopted post-hoc, pre-trained Vision Large Model (VLM)-based and generation-based OOD detection methods. Through analysis of the experimental results, we validate previous conclusions regarding post-hoc, VLM-based, and generation-based methods, while also highlighting their distinct behaviors on specialized datasets. These findings underscore the unique challenges of applying general OOD detection methods to domain-specific tasks such as bronze ware dating. We hope that the ShiftedBronzes benchmark provides valuable insights into both the field of bronze ware dating and the and the development of OOD detection methods. The dataset and associated code will be available later.