Advanced Hybrid Transformer LSTM Technique with Attention and TS Mixer for Drilling Rate of Penetration Prediction

TL;DR

This paper introduces a novel hybrid deep learning framework combining LSTM, Transformer, and TS-Mixer modules with attention mechanisms to improve the accuracy of drilling rate of penetration prediction, addressing the complex, nonlinear drilling data characteristics.

Contribution

The study develops a new hybrid LSTM-Trans-Mixer-Attention model that effectively captures multi-scale temporal dependencies and feature interactions for ROP prediction, outperforming existing models.

Findings

Achieved an Rsquare of 0.9991 in real-world datasets.

Attained a MAPE of 1.447%, indicating high prediction accuracy.

Demonstrated significant performance improvements over baseline models.

Abstract

Rate of Penetration (ROP) prediction is critical for drilling optimization yet remains challenging due to the nonlinear, dynamic, and heterogeneous characteristics of drilling data. Conventional empirical, physics-based, and standard machine learning models rely on oversimplified assumptions or intensive feature engineering, constraining their capacity to model long-term dependencies and intricate feature interactions. To address these issues, this study presents a new deep learning Hybrid LSTM-Trans-Mixer-Att framework that first processes input data through a customized Long Short-Term Memory (LSTM) network to capture multi-scale temporal dependencies aligned with drilling cycles. Subsequently, an Enhanced Transformer encoder with drilling-specific positional encodings and real-time optimization refines the features. Concurrently, a parallel Time-Series Mixer (TS-Mixer) block introduced facilitates efficient cross-feature interaction modeling of static and categorical parameters, including lithological indices and mud properties. The feature representations extracted from the Enhanced Transformer and TS-Mixer modules are integrated through a dedicated fusion layer. Finally, an adaptive attention mechanism then dynamically assigns contextual weights to salient features, enhancing discriminative representation learning and enabling high-fidelity ROP prediction. The proposed framework combines sequential memory, static feature interactions, global context learning, and dynamic feature weighting, providing a comprehensive solution for the heterogeneous and event-driven nature of drilling dynamics. Experimental validation on real-world drilling datasets demonstrates superior performance, achieving an Rsquare of 0.9991 and a MAPE of 1.447%, significantly outperforming existing baseline and hybrid models.