Memorized action chunking with Transformers: Imitation learning for vision-based tissue surface scanning

TL;DR

This paper introduces MACT, a Transformer-based imitation learning method that improves automated tissue surface scanning by leveraging past visual data and hybrid positional embeddings, demonstrating significant success in simulations and real-world tests.

Contribution

The paper presents MACT, a novel Transformer-based imitation learning approach that effectively handles long-horizon, fine-grained tissue scanning tasks with limited demonstrations.

Findings

MACT outperforms baseline models in simulation tasks.

In real-world tests, MACT achieved 60-80% success with only 50 demonstrations.

MACT demonstrates promise for adaptive surgical scanning applications.

Abstract

Optical sensing technologies are emerging technologies used in cancer surgeries to ensure the complete removal of cancerous tissue. While point-wise assessment has many potential applications, incorporating automated large area scanning would enable holistic tissue sampling. However, such scanning tasks are challenging due to their long-horizon dependency and the requirement for fine-grained motion. To address these issues, we introduce Memorized Action Chunking with Transformers (MACT), an intuitive yet efficient imitation learning method for tissue surface scanning tasks. It utilizes a sequence of past images as historical information to predict near-future action sequences. In addition, hybrid temporal-spatial positional embeddings were employed to facilitate learning. In various simulation settings, MACT demonstrated significant improvements in contour scanning and area scanning over the baseline model. In real-world testing, with only 50 demonstration trajectories, MACT surpassed the baseline model by achieving a 60-80% success rate on all scanning tasks. Our findings suggest that MACT is a promising model for adaptive scanning in surgical settings.