Benchmarks, Performance Evaluation and Contests for 3D Shape Retrieval

TL;DR

This paper reviews the development of benchmarks, evaluation measures, and contests for 3D shape retrieval, highlighting community efforts and recent results from SHREC and NIST benchmarks.

Contribution

It provides an overview of benchmarking procedures, discusses current efforts, and details the organization and outcomes of recent 3D shape retrieval contests.

Findings

SHREC tracks facilitate performance comparison of algorithms.

Different evaluation measures are used to assess retrieval effectiveness.

Results demonstrate progress and challenges in 3D shape retrieval.

Abstract

Benchmarking of 3D Shape retrieval allows developers and researchers to compare the strengths of different algorithms on a standard dataset. Here we describe the procedures involved in developing a benchmark and issues involved. We then discuss some of the current 3D shape retrieval benchmarks efforts of our group and others. We also review the different performance evaluation measures that are developed and used by researchers in the community. After that we give an overview of the 3D shape retrieval contest (SHREC) tracks run under the EuroGraphics Workshop on 3D Object Retrieval and give details of tracks that we organized for SHREC 2010. Finally we demonstrate some of the results based on the different SHREC contest tracks and the NIST shape benchmark.