VDMS: Efficient Big-Visual-Data Access for Machine Learning Workloads

TL;DR

VDMS is a system designed to improve access and analytics of large visual datasets for machine learning, offering faster query performance and specialized storage formats.

Contribution

Introduces VDMS, a visual data management system optimized for machine learning workflows with graph-based metadata search and machine-friendly storage formats.

Findings

2x faster complex query performance

Supports rich vision analytics in medical imaging

Enables efficient machine learning pipelines for visual data

Abstract

We introduce the Visual Data Management System (VDMS), which enables faster access to big-visual-data and adds support to visual analytics. This is achieved by searching for relevant visual data via metadata stored as a graph, and enabling faster access to visual data through new machine-friendly storage formats. VDMS differs from existing large scale photo serving, video streaming, and textual big-data management systems due to its primary focus on supporting machine learning and data analytics pipelines that use visual data (images, videos, and feature vectors), treating these as first class entities. We describe how to use VDMS via its user friendly interface and how it enables rich and efficient vision analytics through a machine learning pipeline for processing medical images. We show the improved performance of 2x in complex queries over a comparable set-up.