Contact-Aided Navigation of Flexible Robotic Endoscope Using Deep Reinforcement Learning in Dynamic Stomach

TL;DR

This paper presents a deep reinforcement learning approach that uses contact force feedback to improve the navigation accuracy and stability of flexible endoscopes in dynamic, deformable stomach environments, achieving high success rates.

Contribution

The study introduces a novel contact-aided navigation strategy using DRL and contact force feedback, trained in a physics-based simulation for flexible endoscopes in deformable stomachs, outperforming previous methods.

Findings

Achieved 100% success rate within 3 mm in static and dynamic environments.

Maintained 85% success rate in unseen, challenging scenarios.

Significantly outperformed baseline navigation policies.

Abstract

Navigating a flexible robotic endoscope (FRE) through the gastrointestinal tract is critical for surgical diagnosis and treatment. However, navigation in the dynamic stomach is particularly challenging because the FRE must learn to effectively use contact with the deformable stomach walls to reach target locations. To address this, we introduce a deep reinforcement learning (DRL) based Contact-Aided Navigation (CAN) strategy for FREs, leveraging contact force feedback to enhance motion stability and navigation precision. The training environment is established using a physics-based finite element method (FEM) simulation of a deformable stomach. Trained with the Proximal Policy Optimization (PPO) algorithm, our approach achieves high navigation success rates (within 3 mm error between the FRE's end-effector and target) and significantly outperforms baseline policies. In both static and dynamic stomach environments, the CAN agent achieved a 100% success rate with 1.6 mm average error, and it maintained an 85% success rate in challenging unseen scenarios with stronger external disturbances. These results validate that the DRL-based CAN strategy substantially enhances FRE navigation performance over prior methods.

Figures

Tables

Table 1. Table I: Comparative study of policies SE vs. DE

Policy	SR in SE [%] $\uparrow$	AE [mm] $\downarrow$	SR in DE [%] $\uparrow$	AE [mm] $\downarrow$
A	26.2	2.5	34.5	2.2
B	98	1.6	81.5	1.7
C	100	1.6	100	1.6
D	91.8	1.6	97.8	1.6
E	100	1.6	100	1.6

Table 2. Table II: Comparative study of policies UE1 vs. UE2

Policy	SR in UE1 [%] $\uparrow$	AE [mm] $\downarrow$	SR in UE2 [%] $\uparrow$	AE [mm] $\downarrow$
A	32.5	2.2	33.8	2.5
B	75.6	1.8	68.8	2.1
C	81	1.9	78	1.8
D	71.3	2.1	55	2.0
E	84.5	1.9	85	1.9

Equations

$$u(x)=\sum _{i=1}^n{N}_i(x)u_i,$$

$$M\frac{{\mathrm{d}}^{2}u}{\mathrm{d}{t}^2}+C\frac{\mathrm{d}u}{\mathrm{d}t}+K_{e}u=F_{external},$$

$$g(u)=p,u=u_A,u_B.$$

$$F_c=G^T\lambda ,$$

$$\overrightarrow{F_p}(t)=\{\begin{array}{ll}{\vec{f}}_0,& \text{if }\left(\frac{t}{T}\right)\mathrm{mod}2\in [0,1),\\ -{\vec{f}}_0,& \text{if }\left(\frac{t}{T}\right)\mathrm{mod}2\in [1,2).\end{array}$$

$$R_{dis}$$

$$R_B$$

$$R_S$$

Full text

I Introduction

Over the past decade, gastrointestinal endoscopy has become a cornerstone for both surgical diagnostics and treatment [1, 2]. Traditional manual endoscopy faces significant challenges in accessing certain regions within the stomach due to the tissue deformation, the dynamic movement of that stomach and non-intuitive manual manipulation. To tackle this, an increasing number of flexible robotic endoscopes (FRE) were developed [3, 1, 2]. The FRE, a type of continuum robot, offers inherent compliance, enabling it to adapt its shape to tortuous gastrointestinal (GI) tracts [4]. The active segment has three degrees of freedom (DoFs) of our FRE, which include two bendings and one translational movement. As shown in Fig. 1(a), the FRE consists of 1000 mm passive segments and a 79 mm active segment (7% of entire endoscope), with a bending range over $\pm 90^{\circ}$ [2]. The average length of the male stomach is 190 mm, and the average length of the greater curvature of the male stomach is 344 mm [5]. Without utilizing contact with the stomach walls, the reachable workspace of the endoscope tip is greatly limited. As illustrated in Fig. 1(a), purely executing tip bending, base rotation, and insertion motions (without wall contact) cannot guide the endoscope tip to certain targets on the stomach wall. The passive segment simply floats in the large gastric cavity, and the tip cannot reach deep targets by free-space motions alone. In contrast, if the endoscope can brace or push against the stomach walls (Fig. 1(b)), the environment contact provides a form of “external constraint” or fulcrum that the robot can leverage to extend its reach and stability. In other words, contact is indispensable for navigating FREs inside expansive and deformable environments like the stomach. This insight underpins the concept of Contact-Aided Navigation (CAN): using the endoscope’s contact with the surroundings (here, the stomach wall) to help guide the robot towards the target.

Several research efforts have recognized the importance of contact in continuum robot navigation. For instance, Zhang et al. implemented a motion controller for a tendon-driven continuum catheter that explicitly modeled contacts using FEM, enabling real-time contact force estimation and control without relying on any external sensors [6]. Recent work in motion planning has also embraced contact-rich strategies. Rao et al. developed a contact-aided motion planner that leverages environmental contact to improve path planning for tendon-driven continuum robots [7]. In scenarios where avoiding collisions is impossible, contact-inclusive planners have proven more robust than traditional purely collision-avoiding methods. For example, researchers have shown that a soft “vine” robot can navigate more reliably by exploiting contact with obstacles rather than treating them as forbidden zones [8].

Furthermore, new sensing and control frameworks are emerging to handle contacts: Wang et al. proposed a perception-action loop that compares expected vs. actual robot shape to detect contact events and adjust a soft robot’s configuration in real time [9]. Most existing research on CAN focuses on sparse contact situations [8, 10, 9], or studies CAN in small, static cavity regions [11, 6]. Extending CAN strategies to a large, deformable, and dynamically moving environment like the stomach poses new challenges that have not been fully addressed by existing approaches.

Deep reinforcement learning (DRL) has emerged as a promising tool for enabling FREs to learn CAN strategies in complex, dynamic, contact-rich environments. DRL allows robotic systems to learn sophisticated policies through trial-and-error interactions with the environment [12, 13, 14, 15, 16, 17, 18]. Unlike traditional path planning methods that rely on predefined models or heuristics, DRL is a data-driven approach that offers the flexibility and adaptability needed to navigate dynamic, uncertain environments. Notably, Ng et al. demonstrated the feasibility of training a tendon-driven flexible endoscope agent using PPO in the SOFA simulation environment [12]. However, this work did not leverage an explicit CAN strategy and was evaluated in a simpler scenario. It lacked the dynamic physiological motions of the stomach and did not integrate force feedback into the policy’s observations. As a result, the learned policy could struggle when collisions occur or when the environment is dynamic. By doing so, our method enables the FRE to intelligently utilize contacts with the stomach wall to navigate toward targets, even as the stomach undergoes deformations and motion.

In this work, we leverage the contact feedback to improve endoscopic navigation capability in deformable and dynamic stomachs. The major contributions are as follows:

• •

Innovative DRL-based CAN strategy: We develop a novel CAN approach, integrating force feedback into the DRL framework. This enables the FRE to leverage contact forces and interaction dynamics, improving adaptability and performance in dynamic stomachs.

• •

Dynamic simulation environment: We construct a dynamic and deformable stomach environment in physics-based SOFA simulation that assembles breathing, heartbeat, and subtle movements of the human body during FRE navigation. The training results show the FRE needs real-time adaptation to environmental deformations and external disturbances, which is critical for autonomous endoscopic navigation in clinical scenarios.

• •

Generalization and robustness: We conduct extensive zero-shot evaluations in unstructured environments (UE) with unseen target locations and varying external force dynamics, demonstrating the adaptability of policy trained by CAN strategy.

II Method

This section shows the dynamic stomach environment, which is constructed using the physics-based SOFA simulator. FEM models the deformable stomach and real-time contact detection accurately captures the interactions between the robot and its surroundings. Then, a DRL training pipeline in constructed environments is illustrated.

II-A Force-informed dynamic simulation environment

Efficient contact detection pipelines and model order reduction methods enable real-time deformation and force modeling [20]. FEM is widely used for soft robotics simulation due to its ability to represent complex geometries and non-linear material behaviors. A denser mesh can capture finer curvature details. As shown in Fig. 2, FEM discretizes a curved structure into smaller, manageable elements. To reduce computational complexity, these elements are defined as nodes, dividing the continuous domain of the soft body using a discretization method.

[TABLE]

where $u(x)$ represents the displacement at position x, $N_{i}(x)$ is shape function and $u_{i}$ is nodal displacement. External forces, such as pulling forces and gravity, induce deformation in the soft body. In volumetric meshes, tetrahedral elements define the DoFs of the deformable object. Within the simulation, force propagates from the outer regions to the inner regions. To model the deformable behavior of the stomach, a volumetric mesh is generated using Gmsh. The system dynamics are governed by the equation of motion:

[TABLE]

where $M$ is the mass matrix, $u$ is the displacement vector as defined in Eq. 1, $C$ is the damping matrix, $K_{e}$ is the stiffness matrix to specify material properties such as Young’s modulus and Poisson’s ratio, and $F_{external}$ represents external forces. Internal forces propagate within the volumetric tetrahedral mesh and are modeled as $f_{int}=K_{e}u$. These forces are used to compute the displacement of each DoF at each time step, ensuring accurate force propagation and system dynamics.

As shown in Fig. 3(a), the FRE interacts with a deformable stomach, which introduces displacement $\Delta u_{contact}$ of the target point due to contact force. Real-time contact handling in FEM ensures that two soft bodies do not interpenetrate and efficiently manage collisions. Contact and collision generate reaction forces that push the contact points away from the surface. A constraint-based method is employed to maintain zero penetration by defining a constraint function $g(u)$ between two objects $u_{A}$ and $u_{B}$. Given a gap p between them, the constraint function is formulated as

[TABLE]

The Lagrange multiplier method is added into the $F_{external}$ of Eq. 2 to handle contact forces $F_{c}$, which is formulated by

[TABLE]

where $G$ is matrix of contact constraints, and $\lambda$ is the Lagrange multipliers.

To facilitate reinforcement learning in a force-informed dynamic environment, Algorithm 1 outlines the simulation setup, which integrates FRE with contact information for training agents. The FRE consists of four cable-driven actuation units and one axial DoF. The environment is designed to simulate endoluminal conditions. The simulation begins with the initialization of the environment (ROOTNODE function in SOFA). Essential plugins are added to the main header to enable required functionalities for solving FEM force field, visualization, collision detection etc. Threshold values for collision detection are set at 2 mm for alarms and 0.5 mm for contact, with a friction coefficient of 0.1 to mimic realistic surface interactions (handled by the collisionPipeline in SOFA). Dynamic properties of the environment include a deformable phantom model representing the stomach, which is simulated using FEM with collision and visual components (DynamicEnvNode). Specific indices of the volumetric stomach mesh are assigned as fixed constraints. When subjected only to constant forces, the internal forces in the stomach reach an equilibrium state, stabilizing the structure. As shown in Fig. 3(b), to introduce dynamic environmental conditions, discrete periodic external forces with a period of $2T$ to introduce environmental dynamics, which is defined as

[TABLE]

FRE includes a deformable section actuated by tendon-driven mechanisms, which control the front-end motion of the robot (SoftRobotNode). The robot’s operation is managed by a controller within the reinforcement learning framework (GymStepController). Real-time contact data is collected using the ContactListener component, which extracts collision information from collisionPointsModel (i.e., collision between stomach and FRE). This ensures that contact force data is continuously monitored and integrated into the simulation pipeline. The resultant force and contact feedback are then used as state inputs for reinforcement learning, allowing the agent to adapt to dynamic endoluminal conditions and optimize navigation strategies.

II-B Deep reinforcement learning

On-policy DRL trains an agent to navigate by learning from interactions with its environment. The Markov Decision Process (MDP) provides a framework for modeling sequential decision-making problems in RL. Elements $S,A,R,P,\gamma$ are defined in the feedback control loop of CAN problems, which can be formulated as a long-horizon discounted MDP. $s_{t}\in S$ is the state at the time $t$, $a_{t}\in A$ is the action in time $t$, and the reward function $r_{t}=R(s_{t},a_{t})$. The transition dynamics, $P(s_{t+1}|s_{t},a_{t})$, describe the probability of reaching state $s_{t+1}$ given the current state $s_{t}$ and $a_{t}$. The discount factor $\gamma\in[0,1]$ determines the less importance of past rewards, ensuring long-term optimization in the decision-making process.

At the beginning of training, a simulation scene is created using a predefined method described in Algorithm 1. To ensure robust policy generalization, the target position is randomized via a reset function, and the initial state of the environment is recorded as the observation vector. As shown in Fig. 4, the state $s_{t}$ consists of $\vec{p_{t}},\vec{v_{t}},L_{t},\vec{F_{t}},C$, where $\vec{p_{t}}$ is the position vector between end-effector (EE) and target, $\vec{v_{t}}$ is the EE’s instantaneous velocity, $L_{t}$ represents four cable lengths, $\vec{F_{t}}$ is the resultant contact force vector, and $C$ is the contact indicator. The reward function $r_{t}$ consists of $R_{dis},R_{B},R_{S}$, defined as:

[TABLE]

$R_{dis}$ specifies the negative reward of the distance between EE and the target point. $R_{B}$ indicates a boundary penalty when EE leaves the pylorus stomach. $R_{S}$ is a positive reward in the case of reaching the targets.

The control policy is defined as a deep neural network. Based on neural networks, FRE can predict the dynamics of the system [23], where perception is the input, and action is the output. As shown in Algorithm 2, the training process begins with the initialization of a simulation environment using Gymnasium [21], specifically the custom environment endogym-v0, which integrates the SOFA simulation framework. This environment allows for real-time interaction between the policy and the simulated FRE. The algorithm iteratively trains an optimal control policy, denoted $\pi$, by mapping the current observation to an action. The action is then applied to the environment, which updates the robot’s state. Sequential data pairs are stored in the data buffer. We adopt the Proximal Policy Optimization (PPO) algorithm due to its robust, stable performance in high-dimensional, continuous control tasks [24].

III Experiment

We conducted a series of experiments in the simulation environment to assess the performance of the proposed CAN strategy. The experiments are designed to answer three questions: (1) Does incorporating contact force feedback into the state improve navigation performance? (2) How does training in a dynamic environment (with moving walls) compare to training in a static environment? (3) Can the learned policy generalize to new, unseen scenarios (targets and dynamics) that were not encountered during training?

III-A Experiment setup

III-A1 Environment

The FREs were trained and evaluated in environments with varying conditions, defined as follows:

• •

Free environment (FE): an empty environment where the FRE can reach targets without contact.

• •

Static environment (SE): ensures deformation only occurs when the robotic endoscope interacts with it. A deformable stomach model is stationary when no external periodic forces are applied.

• •

Dynamic environment (DE): As described in Section II-A, periodic sinusoidal forces were applied to the deformable stomach model to simulate respiration and natural peristaltic movements. This introduced dynamic deformations in the environment, independent of direct contact with the robotic endoscope. As the target moves, the endoscope must adapt to the constantly changing geometry of the stomach, similar to realistic conditions.

III-A2 Training and simulation parameters

The simulation and training environments were implemented using a custom Gym-compatible interface [21], leveraging the deformable behavior modeled by FEM in the SOFA framework [20]. The number of maximum steps of training episodes is set to 128. A continuous action space with five degrees of freedom is constrained in the range of [-0.4, 0.4]. The state space consisted of 6 features: the position vector from the robot’s tip to the target, the velocity of the robot tip, cable length displacement, prior axial actions, binary contact indicators, and normalized contact forces. Without losing realistic considerations, contact forces can be obtained in real FRE, as demonstrated in soft robotic sensing and contact estimation [9, 25, 26].

As described in Eq. 5, the dynamics of stomach was simulated by a LinearForceAlternator controller. This alternation induced continuous deformations, requiring the FRE to adapt its trajectory in real-time. A random selection from the predefined list of indices, $IndexList_{A}$, on the stomach surface defined the variation in navigation goals. Episodes terminated when the distance between the robotic tip and the target was less than 3 mm, the tip of the endoscope was outside of the stomach’s boundaries, or the maximum number of steps was exceeded. The network consisted of four layers with sizes [256, 128, 64, 32]. DRL was trained on a 12th Gen Intel® Core™ i7-12700K CPU with 20 cores, the training rate was maintained at 4 Hz.

III-A3 Policies

To evaluate the importance of force-informed observation and dynamic environment, the training process was performed under five settings:

• •

Policy A ($\pi_{A}$): trained in FE, without any contact.

• •

Policy B ($\pi_{B}$): trained in the SE, but without access to force and contact information [12].

• •

Policy C ($\pi_{C}$): trained in the SE, with state information of the resultant force and the contact indicator.

• •

Policy D ($\pi_{D}$): trained in the DE, but without access to force and contact information.

• •

Policy E ($\pi_{E}$): trained in the DE, with state information of the resultant force and contact indicator (our CAN strategy for learning in a challenging environment with the benefit of force awareness).

III-B Training performances

The training results of the five policies demonstrate their convergence at varying levels of environmental complexity and input state spaces. As shown in Fig. 5(a), $\pi_{A}$ trained in FE, exhibited the fastest convergence around 300k timesteps. The reward of $\pi_{A}$ is shown in Eq. 6. This indicates that the FE posed minimal challenges, allowing the policy to learn an effective navigation strategy with fewer interactions.

In contrast, the training curves of $\pi_{B},\pi_{C},\pi_{D},\pi_{E}$ in Fig. 5(b), reveal a more gradual convergence. Remarkably, policy E trained with force feedback in state and the most complex environment, converging after 1M timesteps.

Despite these differences, all the policies in Fig. 5(b) ultimately converged to similar reward levels within their respective training environments. This indicates that DRL effectively optimized navigation strategies under a consistent reward structure, suggesting that the policies acquired comparable navigation capabilities in their respective settings. As shown in Eq. (6)-(8), under the same reward function, all policies learned to optimize their paths, efficiently explored the stomach region, and maintained a stable success rate (SR) in different target locations. Each policy converged to an episode reward of approximately $10^{4}$, demonstrating the robustness of DRL in the navigation task. This outcome highlights that, despite variations in environmental complexity and state representations, the trained policies achieved similar performance within their training conditions.

III-C Navigation performance

This section evaluates the navigation performances. The experiments assess navigation in training environments (TE). Then, policies with zero-shot are evaluated in UE with different external force dynamics and unseen target sets. The SR and average error (AE) are used as metrics to compare the performance of the policies. By analyzing the results, we aim to demonstrate the impact of incorporating force and contact feedback into the state representation, particularly in dynamic and complex navigation scenarios (maximum 80 timesteps for each trial).

III-C1 Navigation in TE

$\pi_{A}$ trained without environmental contact or force feedback, achieved a high SR of 98.3% in free space, with an impressive average error of 0.4 mm. This indicates that $\pi_{A}$ is highly effective in simple navigation tasks without contact, where no external forces or environmental interactions are involved.

As shown in Table I, in the SE, $\pi_{C}$, and $\pi_{E}$ achieved a 100% SR and an AE of 1.6 mm, demonstrating their superior precision and reliability. $\pi_{B}$ trained in the SE, achieved a slightly lower SR of 98%. $\pi_{A}$ trained in FE, performed poorly, with an SR of only 26.2% and an AE of 2.5 mm. The lack of contact and force feedback when training $\pi_{A}$ limits its adaptability to complex, dynamic, and constrained environments, which highlights the importance of training in contact environments for improving navigation performance.

In the DE, $\pi_{C}$, $\pi_{E}$ maintained a 100% SR and 1.6 mm AE, adapting effectively to the complex dynamic environment. In contrast, $\pi_{D}$ showed a slightly lower SR of 97.8%, and $\pi_{B}$ achieved only 81.5% SR, indicating reduced adaptability in dynamic scenarios without access to force and contact data. $\pi_{A}$’s performance was limited to an SR of 34.5%, reinforcing the importance of training in a contact environment and adding contact force feedback to state information.

III-C2 Navigation in UE

To evaluate the policies, we construct unstructured environment 1 (UE1) and unstructured environment 2 (UE2) that apply doubled and triple external periodic forces respectively. UE1 and UE2 use unseen target set $IndexList_{B}$. As shown in Table II, $\pi_{E}$ outperformed all others with SR of 84.5% in UE1. $\pi_{C}$ also demonstrated strong performance with an SR of 81%, but $\pi_{B}$ and $\pi_{D}$, trained without force feedback, achieved lower SRs of 75.6% and 71.3%. In UE2, with triple periodic forces, the best policy $\pi_{E}$ maintained the highest SR of 85%. $\pi_{A}$ showed the weakest adaptability to UE, achieving around 30% SR.

Policies trained with this information demonstrated higher SRs and precision in UE, which emphasize the critical role of force and contact feedback in enabling policies to adapt to complex and dynamic environments.

III-C3 RFE navigation demonstration

Fig. 6(a) presents selected navigation trajectories of the FRE in a dynamic stomach. The trajectories are color-coded, where paths sharing the same color correspond to the same initial target positions. Targets are randomly assigned within the designated blue region. As the robot moves toward a target, it continuously adapts to the changing environment while interacting with the deformable stomach walls. Green points along the trajectories indicate the FRE starts to come into contact with the stomach. The varying trajectory patterns reflect the FRE’s ability to respond dynamically to environmental deformations and external disturbances.

Fig. 6(b) illustrates the navigation process using pre-trained $\pi_{E}$. The robot actively maneuvers toward the target, represented by a white point, which transitions to yellow once the EE reaches within 3 mm of the target position. Yellow arrows indicate the reaction forces exerted between the endoscope and the stomach wall, providing insight into how contact forces influence navigation. The trajectories demonstrate CAN strategy that force information is needed in DE. These forces arise due to the interaction between FRE and the stomach. By utilizing DRL, the policy enables the robot to adjust its motion based on the force information.

Fig. 6(c) shows a normalized force magnitude and remaining distance between EE and targets across navigation trajectories. The transparent blue and green lines represent individual trajectory data for normalized force magnitude and distance, respectively. The solid blue line indicates the smoothed average of normalized force magnitude across all trajectories, obtained via cubic spline interpolation, while the dashed green line represents the average normalized distance to the target. The data highlights the trend of increasing force magnitude as the robot tip approaches the target, showcasing the robot’s reliance on contact-based guidance. The robot tip is far from the target, and the force magnitude is minimal. As the robot moves forward, the contact area between the stomach wall and the passive end of the robot increases, leading to a rise in force magnitude. This trend highlights the progressive interaction between the robot and its environment, with force values intensifying as the robot approaches the target, which shows a clear inverse relationship: the closer the tip gets to the target, the greater the force experienced. This behavior underscores the significance of force information as a navigation signal in FRE navigation.

IV CONCLUSIONS

We proposed the CAN strategy to navigate an FRE in dynamic stomach. To address the challenges of non-intuitive manual control, a force-informed DRL framework was developed to enhance autonomy. Contact forces serve as critical guidance signals, allowing the FRE to perform complex movements and improve navigation performance in UE. The effectiveness of force feedback in overcoming the uncertainties in deformable stomachs for FRE is shown.

Despite its advantages, the proposed DRL-based CAN method has limitations. The longer convergence times in complex environments highlight the increased computational cost and learning effort required for policy adaptation. Additionally, while contact forces improve navigation, visual information is also an important observation for navigation in real FRE. Combining information of contact and vision remains an area for investigation. Future research will focus on extending the current state-based DRL policy to a multi-modal DRL framework that integrates both force and vision information. This enhancement will enable the FRE to leverage complementary sensory inputs for more robust navigation. Additionally, investigating safe navigation algorithms is essential to minimize the risk of tissue injury, ensuring that CAN remains both effective and non-invasive.