PortionNet: Distilling 3D Geometric Knowledge for Food Nutrition Estimation

TL;DR

PortionNet is a novel framework that learns 3D geometric features from point clouds during training but only requires RGB images at inference, enabling accurate food volume and energy estimation without specialized hardware.

Contribution

It introduces a cross-modal knowledge distillation approach that enables 3D geometric reasoning from RGB images alone, improving food nutrition estimation accuracy.

Findings

Outperforms previous methods on MetaFood3D in volume and energy estimation.

Demonstrates strong generalization across datasets in energy estimation.

Achieves state-of-the-art results without requiring depth sensors.

Abstract

Accurate food nutrition estimation from single images is challenging due to the loss of 3D information. While depth-based methods provide reliable geometry, they remain inaccessible on most smartphones because of depth-sensor requirements. To overcome this challenge, we propose PortionNet, a novel cross-modal knowledge distillation framework that learns geometric features from point clouds during training while requiring only RGB images at inference. Our approach employs a dual-mode training strategy where a lightweight adapter network mimics point cloud representations, enabling pseudo-3D reasoning without any specialized hardware requirements. PortionNet achieves state-of-the-art performance on MetaFood3D, outperforming all previous methods in both volume and energy estimation. Cross-dataset evaluation on SimpleFood45 further demonstrates strong generalization in energy estimation.