Food Image Generation on Multi-Noun Categories

TL;DR

This paper addresses the challenge of generating accurate food images for multi-noun categories by introducing a novel method that incorporates domain knowledge and layout refinement, resulting in improved generation quality.

Contribution

The paper proposes FoCULR, a method that enhances food image generation by integrating food domain knowledge and layout refinement to better handle multi-noun categories.

Findings

Improved accuracy in generating multi-noun food images.

Enhanced spatial layout correctness in generated images.

Better alignment with real-world food dataset distributions.

Abstract

Generating realistic food images for categories with multiple nouns is surprisingly challenging. For instance, the prompt "egg noodle" may result in images that incorrectly contain both eggs and noodles as separate entities. Multi-noun food categories are common in real-world datasets and account for a large portion of entries in benchmarks such as UEC-256. These compound names often cause generative models to misinterpret the semantics, producing unintended ingredients or objects. This is due to insufficient multi-noun category related knowledge in the text encoder and misinterpretation of multi-noun relationships, leading to incorrect spatial layouts. To overcome these challenges, we propose FoCULR (Food Category Understanding and Layout Refinement) which incorporates food domain knowledge and introduces core concepts early in the generation process. Experimental results demonstrate that the integration of these techniques improves image generation performance in the food domain.