# A Vision-Based Deep Learning Framework for Monitoring and Recognition of Chemical Laboratory Operations

## TL;DR

This paper introduces a deep learning system that uses video to monitor and recognize pipetting actions in chemistry labs, improving safety and consistency.

## Contribution

The novel framework uses spatiotemporal features and bidirectional LSTM networks for real-time monitoring of pipetting operations.

## Key findings

- The framework reliably distinguishes standard from non-standard pipetting behaviors across multiple error categories.
- It shows improved robustness compared to static or frame-level analysis methods.
- The system is feasible for scalable and objective monitoring of laboratory procedures.

## Abstract

Standardized operating procedures are essential for ensuring safety and reproducibility in chemical laboratory experiments. However, real-time monitoring of manual laboratory operations, such as pipetting, remains challenging due to complex human–tool interactions, temporal dependencies between procedural steps, and operator variability. In this study, we propose a vision-based deep learning framework that leverages spatiotemporal features for automated monitoring of pipetting operations using non-contact visual sensing. Briefly, human poses and pipette interactions are extracted from video recordings using a YOLO-based perception model, while temporal execution patterns are captured through bidirectional long short-term memory networks. Experimental results demonstrate that the proposed approach can reliably distinguish between standard and non-standard pipetting behaviors across multiple predefined error categories and shows improved robustness compared with static or frame-level analysis. Overall, this work demonstrates the feasibility of vision-based AI systems for objective and scalable monitoring of laboratory pipetting operations, with potential applicability to other manual laboratory procedures.