Exploring Social Media Image Categorization Using Large Models with Different Adaptation Methods: A Case Study on Cultural Nature's Contributions to People

TL;DR

This paper investigates the use of large models and adaptation methods for categorizing social media images related to human-nature interactions, introducing a new dataset and evaluating various solutions for improved performance.

Contribution

It introduces FLIPS, a new dataset of social media images, and evaluates multiple large model adaptation methods for social media image categorization.

Findings

Large models show promise in categorizing complex social media images.

Different adaptation methods vary in cost and scalability.

The proposed solutions improve categorization performance on the FLIPS dataset.

Abstract

Social media images provide valuable insights for modeling, mapping, and understanding human interactions with natural and cultural heritage. However, categorizing these images into semantically meaningful groups remains highly complex due to the vast diversity and heterogeneity of their visual content as they contain an open-world human and nature elements. This challenge becomes greater when categories involve abstract concepts and lack consistent visual patterns. Related studies involve human supervision in the categorization process and the lack of public benchmark datasets make comparisons between these works unfeasible. On the other hand, the continuous advances in large models, including Large Language Models (LLMs), Large Visual Models (LVMs), and Large Visual Language Models (LVLMs), provide a large space of unexplored solutions. In this work 1) we introduce FLIPS a dataset of Flickr images that capture the interaction between human and nature, and 2) evaluate various solutions based on different types and combinations of large models using various adaptation methods. We assess and report their performance in terms of cost, productivity, scalability, and result quality to address the challenges of social media image categorization.