Understanding Image2Video Domain Shift in Food Segmentation: An Instance-level Analysis on Apples

TL;DR

This paper investigates why food segmentation models trained on images perform poorly on videos, revealing that temporal inconsistencies like illumination changes cause identity fragmentation and proposing remedies without full video supervision.

Contribution

The study provides an instance-level analysis of domain shift in food segmentation from images to videos, highlighting the importance of temporal coherence and proposing practical solutions.

Findings

High frame-wise accuracy does not ensure temporal stability.

Illumination and reflection cause mask flickering and identity errors.

Conventional metrics overestimate real-world video performance.

Abstract

Food segmentation models trained on static images have achieved strong performance on benchmark datasets; however, their reliability in video settings remains poorly understood. In real-world applications such as food monitoring and instance counting, segmentation outputs must be temporally consistent, yet image-trained models often break down when deployed on videos. In this work, we analyze this failure through an instance segmentation and tracking perspective, focusing on apples as a representative food category. Models are trained solely on image-level food segmentation data and evaluated on video sequences using an instance segmentation with tracking-by-matching framework, enabling object-level temporal analysis. Our results reveal that high frame-wise segmentation accuracy does not translate to stable instance identities over time. Temporal appearance variations, particularly illumination changes, specular reflections, and texture ambiguity, lead to mask flickering and identity fragmentation, resulting in significant errors in apple counting. These failures are largely overlooked by conventional image-based metrics, which substantially overestimate real-world video performance. Beyond diagnosing the problem, we examine practical remedies that do not require full video supervision, including post-hoc temporal regularization and self-supervised temporal consistency objectives. Our findings suggest that the root cause of failure lies in image-centric training objectives that ignore temporal coherence, rather than model capacity. This study highlights a critical evaluation gap in food segmentation research and motivates temporally-aware learning and evaluation protocols for video-based food analysis.