Rethinking Cooking State Recognition with Vision Transformers

TL;DR

This paper introduces a Vision Transformer-based approach for cooking state recognition in kitchen environments, leveraging global attention and transfer learning to significantly improve accuracy over previous methods.

Contribution

It applies Vision Transformers with transfer learning and data augmentation to enhance cooking state recognition, achieving state-of-the-art performance.

Findings

Achieved 94.3% accuracy on the Cooking State Recognition Challenge Dataset.

Outperformed existing state-of-the-art methods.

Demonstrated the effectiveness of global attention in distinguishing similar cooking states.

Abstract

To ensure proper knowledge representation of the kitchen environment, it is vital for kitchen robots to recognize the states of the food items that are being cooked. Although the domain of object detection and recognition has been extensively studied, the task of object state classification has remained relatively unexplored. The high intra-class similarity of ingredients during different states of cooking makes the task even more challenging. Researchers have proposed adopting Deep Learning based strategies in recent times, however, they are yet to achieve high performance. In this study, we utilized the self-attention mechanism of the Vision Transformer (ViT) architecture for the Cooking State Recognition task. The proposed approach encapsulates the globally salient features from images, while also exploiting the weights learned from a larger dataset. This global attention allows the model to withstand the similarities between samples of different cooking objects, while the employment of transfer learning helps to overcome the lack of inductive bias by utilizing pretrained weights. To improve recognition accuracy, several augmentation techniques have been employed as well. Evaluation of our proposed framework on the `Cooking State Recognition Challenge Dataset' has achieved an accuracy of 94.3%, which significantly outperforms the state-of-the-art.