Leveraging Foundation Models To learn the shape of semi-fluid deformable objects

TL;DR

This paper explores using foundation models to characterize semi-fluid deformable objects, enabling keypoint detection without extensive labeled datasets, and demonstrates promising results in deformable object manipulation.

Contribution

It introduces a novel approach leveraging foundation models as teachers for deformable object characterization, eliminating the need for pre-training and labeled data.

Findings

Student network achieved 13.4 pixel keypoint error.

Teacher model attained 75.26% mIoU in object mask retrieval.

Knowledge distillation improved deformable object characterization.

Abstract

One of the difficulties imposed on the manipulation of deformable objects is their characterization and the detection of representative keypoints for the purpose of manipulation. A keen interest was manifested by researchers in the last decade to characterize and manipulate deformable objects of non-fluid nature, such as clothes and ropes. Even though several propositions were made in the regard of object characterization, however researchers were always confronted with the need of pixel-level information of the object through images to extract relevant information. This usually is accomplished by means of segmentation networks trained on manually labeled data for this purpose. In this paper, we address the subject of characterizing weld pool to define stable features that serve as information for further motion control objectives. We achieve this by employing different pipelines. The first one consists of characterizing fluid deformable objects through the use of a generative model that is trained using a teacher-student framework. And in the second one we leverage foundation models by using them as teachers to characterize the object in the image, without the need of any pre-training and any dataset. The performance of knowledge distillation from foundation models into a smaller generative model shows prominent results in the characterization of deformable objects. The student network was capable of learning to retrieve the keypoitns of the object with an error of 13.4 pixels. And the teacher was evaluated based on its capacities to retrieve pixel level information represented by the object mask, with a mean Intersection Over Union (mIoU) of 75.26%.