Achieving Competitive Play Through Bottom-Up Approach in Semantic Segmentation

TL;DR

This paper demonstrates that a bottom-up approach using PuckNet can achieve competitive object detection and segmentation performance, and effectively enable AI to play video games like SuperTuxKart.

Contribution

The paper introduces PuckNet, a fully convolutional neural network that detects extreme points for object recognition, challenging the dominance of top-down methods in vision tasks.

Findings

PuckNet achieves a 36.4% bounding box AP on COCO test-dev.

It outperforms vanilla bounding box methods in Mask AP, reaching 17.6%.

Guided segmentation improves Mask AP to 32.1%.

Abstract

With the renaissance of neural networks, object detection has slowly shifted from a bottom-up recognition problem to a top-down approach. Best in class algorithms enumerate a near-complete list of objects and classify each into object/not object. In this paper, we show that strong performance can still be achieved using a bottom-up approach for vision-based object recognition tasks and achieve competitive video game play. We propose PuckNet, which is used to detect four extreme points (top, left, bottom, and right-most points) and one center point of objects using a fully convolutional neural network. Object detection is then a purely keypoint-based appearance estimation problem, without implicit feature learning or region classification. The method proposed herein performs on-par with the best in class region-based detection methods, with a bounding box AP of 36.4% on COCO test-dev. In addition, the extreme points estimated directly resolve into a rectangular object mask, with a COCO Mask AP of 17.6%, outperforming the Mask AP of vanilla bounding boxes. Guided segmentation of extreme points further improves this to 32.1% Mask AP. We applied the PuckNet vision system to the SuperTuxKart video game to test it's capacity to achieve competitive play in dynamic and co-operative multiplayer environments.