Segmenting Sky Pixels in Images

TL;DR

This paper enhances sky segmentation in outdoor images, especially under challenging conditions like night and weather, by improving a state-of-the-art model using non-ideal datasets, leading to significant accuracy gains.

Contribution

The study adapts and improves RefineNet for sky segmentation on non-ideal datasets, achieving notable accuracy improvements under diverse conditions.

Findings

10-15% improvement in average MCR on SkyFinder dataset

35% increase in average mIOU over baseline

Significant outperformance in night and weather conditions

Abstract

Outdoor scene parsing models are often trained on ideal datasets and produce quality results. However, this leads to a discrepancy when applied to the real world. The quality of scene parsing, particularly sky classification, decreases in night time images, images involving varying weather conditions, and scene changes due to seasonal weather. This project focuses on approaching these challenges by using a state-of-the-art model in conjunction with a non-ideal dataset: SkyFinder and a subset from SUN database with Sky object. We focus specifically on sky segmentation, the task of determining sky and not-sky pixels, and improving upon an existing state-of-the-art model: RefineNet. As a result of our efforts, we have seen an improvement of 10-15% in the average MCR compared to the prior methods on SkyFinder dataset. We have also improved from an off-the shelf-model in terms of average mIOU by nearly 35%. Further, we analyze our trained models on images w.r.t two aspects: times of day and weather, and find that, in spite of facing same challenges as prior methods, our trained models significantly outperform them.