Vision Based Machine Learning Algorithms for Out-of-Distribution Generalisation

TL;DR

This paper compares vision-based machine learning methods for out-of-distribution generalization, highlighting the poor performance of simple CNN models on domain shifts and evaluating various approaches on benchmark datasets.

Contribution

It provides a comprehensive comparison of domain generalization techniques and conventional deep learning models, emphasizing CNN limitations in out-of-distribution scenarios.

Findings

CNN models perform poorly on domain shifts

Extensive methods outperform simple CNNs in OOD tasks

Benchmark experiments on PACS and Office-Home datasets

Abstract

There are many computer vision applications including object segmentation, classification, object detection, and reconstruction for which machine learning (ML) shows state-of-the-art performance. Nowadays, we can build ML tools for such applications with real-world accuracy. However, each tool works well within the domain in which it has been trained and developed. Often, when we train a model on a dataset in one specific domain and test on another unseen domain known as an out of distribution (OOD) dataset, models or ML tools show a decrease in performance. For instance, when we train a simple classifier on real-world images and apply that model on the same classes but with a different domain like cartoons, paintings or sketches then the performance of ML tools disappoints. This presents serious challenges of domain generalisation (DG), domain adaptation (DA), and domain shifting. To enhance the power of ML tools, we can rebuild and retrain models from scratch or we can perform transfer learning. In this paper, we present a comparison study between vision-based technologies for domain-specific and domain-generalised methods. In this research we highlight that simple convolutional neural network (CNN) based deep learning methods perform poorly when they have to tackle domain shifting. Experiments are conducted on two popular vision-based benchmarks, PACS and Office-Home. We introduce an implementation pipeline for domain generalisation methods and conventional deep learning models. The outcome confirms that CNN-based deep learning models show poor generalisation compare to other extensive methods.