3D Model Assisted Image Segmentation

TL;DR

This paper introduces a fully automatic 3D model assisted image segmentation method that registers a 3D model to an image to improve segmentation accuracy, especially when object parts share similar pixel features.

Contribution

The paper presents a novel gradient-based loss function for 3D model registration and integrates it with level-set segmentation, eliminating the need for prior training or user interaction.

Findings

Effective segmentation of complex objects like cars from arbitrary views.

No prior training or user input required for the segmentation process.

Successful application demonstrated on real car photographs.

Abstract

The problem of segmenting a given image into coherent regions is important in Computer Vision and many industrial applications require segmenting a known object into its components. Examples include identifying individual parts of a component for process control work in a manufacturing plant and identifying parts of a car from a photo for automatic damage detection. Unfortunately most of an object's parts of interest in such applications share the same pixel characteristics, having similar colour and texture. This makes segmenting the object into its components a non-trivial task for conventional image segmentation algorithms. In this paper, we propose a "Model Assisted Segmentation" method to tackle this problem. A 3D model of the object is registered over the given image by optimising a novel gradient based loss function. This registration obtains the full 3D pose from an image of the object. The image can have an arbitrary view of the object and is not limited to a particular set of views. The segmentation is subsequently performed using a level-set based method, using the projected contours of the registered 3D model as initialisation curves. The method is fully automatic and requires no user interaction. Also, the system does not require any prior training. We present our results on photographs of a real car.