Jacobian Exploratory Dual-Phase Reinforcement Learning for Dynamic Endoluminal Navigation of Deformable Continuum Robots

TL;DR

This paper introduces JEDP-RL, a novel reinforcement learning framework that improves planning and navigation of deformable continuum robots by estimating deformation Jacobians, leading to faster convergence and better generalization in dynamic surgical simulations.

Contribution

The paper presents a dual-phase RL method that estimates deformation Jacobians during training, enhancing planning for deformable robots with nonlinear mechanics.

Findings

3.2x faster policy convergence compared to PPO

25% fewer steps to reach target in simulations

92% success rate under material variations

Abstract

Deformable continuum robots (DCRs) present unique planning challenges due to nonlinear deformation mechanics and partial state observability, violating the Markov assumptions of conventional reinforcement learning (RL) methods. While Jacobian-based approaches offer theoretical foundations for rigid manipulators, their direct application to DCRs remains limited by time-varying kinematics and underactuated deformation dynamics. This paper proposes Jacobian Exploratory Dual-Phase RL (JEDP-RL), a framework that decomposes planning into phased Jacobian estimation and policy execution. During each training step, we first perform small-scale local exploratory actions to estimate the deformation Jacobian matrix, then augment the state representation with Jacobian features to restore approximate Markovianity. Extensive SOFA surgical dynamic simulations demonstrate JEDP-RL's three key advantages over proximal policy optimization (PPO) baselines: 1) Convergence speed: 3.2x faster policy convergence, 2) Navigation efficiency: requires 25% fewer steps to reach the target, and 3) Generalization ability: achieve 92% success rate under material property variations and achieve 83% (33% higher than PPO) success rate in the unseen tissue environment.

Figures

Tables

Table 1. TABLE I : Evaluation results in training environment

Policy	YM	SR [%] $\uparrow$	AE [mm] $\downarrow$	NS $\downarrow$	TL [mm] $\downarrow$
PPO-RL	15000	99	2.20	30.63	19.55
JEDP-RL	15000	100.0	1.83	26.0	21.39

Table 2. TABLE II : Comparative experiment result under material property variations

Policy	YM	SR [%] $\uparrow$	AE [mm] $\downarrow$	NS $\downarrow$	TL [mm] $\downarrow$
PPO-RL	15000	89	2.34	50.93	38.8
JEDP-RL	15000	95	1.89	41.7	41.6
PPO-RL	10000	72	2.45	72.0	52.3
JEDP-RL	10000	98	1.78	37.2	34.0
PPO-RL	25000	87	2.36	50.0	40.1
JEDP-RL	25000	92	2.15	38.9	44.12

Equations

$$\mathbf{v}=\mathbf{J}(\mathbf{q})\dot{\mathbf{q}}$$

$$J_c=\frac{\partial \mathbf{x}}{\partial ({\kappa }_1,{\varphi }_1,...,{\kappa }_m,{\varphi }_m)}\in R^{6\times 2m}$$

$$J(t)=J_0+\Delta J(E(t),{\mathbf{F}}_{ext}(t))$$

$$\Vert \Delta J\Vert \propto \Vert {\mathbf{F}}_{ext}\Vert \cdot \cos \theta$$

$$V^{\pi }(s)={\mathbb{E}}_{\pi }\left[\sum _{k=0}^{\infty }{\gamma }^k{R}_{t+k+1}|S_t=s\right]$$

$$P({\mathbf{s}}_{t+1}|{\mathbf{s}}_t,a_t)\ne P({\mathbf{s}}_{t+1}|{\mathbf{s}}_t,a_t,{\mathbf{s}}_{t-1},...)$$

$$P_t(s^{\prime }|s,a)\ne P_{t+\Delta t}(s^{\prime }|s,a)$$

$$\delta \mathbf{A}=[\delta {\mathbf{a}}_1\cdots \delta {\mathbf{a}}_n]\in {\mathbb{R}}^{d_a\times n}$$

$$\delta \mathbf{O}=[\delta {\mathbf{o}}_1\cdots \delta {\mathbf{o}}_n{]}^T\in {\mathbb{R}}^{n\times d_o}$$

$$\hat{\mathbf{J}}=(\delta \mathbf{A}\delta {\mathbf{A}}^T+\lambda \mathbf{I}{)}^{-1}\delta \mathbf{A}\delta \mathbf{O}$$

$${\mathbf{s}}_t^{\text{aug}}=[{\mathbf{o}}_t;\text{vec}(\hat{\mathbf{J}})]\in {\mathbb{R}}^{d_o+d_a\times d_o}$$

$${\pi }^{\ast }=\arg \underset{\pi }{\max }{\mathbb{E}}_{\pi }\left[\sum _{t=0}^T{\gamma }^t{r}_t\right]\text{s.t.}{D}_{KL}({\pi }_{\text{old}}||\pi )\le \delta$$

Full text

I Introduction

Deformable continuum robots (DCRs) have transformed minimally invasive procedures through their ability to navigate confined endoluminal anatomical spaces [1, 2]. Particularly, in gastrointestinal endoscopic applications, over 85% of the robot’s body consists of passive deformable segments exhibiting nonlinear viscoelastic responses to environmental contacts [3, 4]. This structural compliance fundamentally challenges conventional Jacobian-based control paradigms developed for rigid manipulators, as the time-varying kinematics invalidate static Jacobian assumptions [5, 6].

Traditional Jacobian approaches face three inherent limitations in DCR control: (1) passive deformation segments lack direct actuation mapping due to underactuated configurations [7], (2) viscoelastic creep causes time-dependent Jacobian drift in silicone-based actuators [8], and (3) partial state observability obscures deformation dynamics, with a significant portion of dynamic states remaining unmeasurable[9, 10]. Recent attempts to address these challenges include temporal Jacobian estimation through recurrent neural networks [11] and Kalman filtering [12], but as demonstrated by [13], excessive time intervals between observations lead to large Jacobian estimation errors due to unmodeled transient dynamics. This temporal sensitivity necessitates new formulations that couple real-time Jacobian identification with control policy adaptation [14].

RL offers promising alternatives by learning control policies through environmental interactions [15, 16]. Model-free approaches such as proximal policy optimization (PPO) [15] and soft actor-critic (SAC) [17] have achieved empirical success in soft robot control [18, 19]. However, these methods suffer from three critical shortcomings in DCR applications: (1) unstructured exploration fails to sample informative deformation states with great redundancy [20], (2) policy gradients degrade under partial observability with great variance inflation [9], and (3) ignoring Jacobian features discards crucial mechanical information critical for contact-rich manipulation [13]. The hierarchical framework by [14] demonstrates that decoupling system identification from policy learning can mitigate these issues, but requires adaptation to handle DCRs’ continuous deformation space spanning 8+ dimensions [21].

The proposed Jacobian Exploratory Dual-Phase RL (JEDP-RL) framework addresses these limitations through innovative integration of model-based exploration and learning-based control. Phase 1 performs local small-scale exploration to estimate the deformation Jacobian matrix. Phase 2 utilizes these Jacobian estimates as augmented state features in constrained policy optimization, achieving 3.2× faster convergence than baseline PPO [15]. This dual-phase architecture sequentially performs phased state estimation and policy execution, effectively reconstructing a Markovian representation from partial observations. The deformation Jacobian serves as a physics-informed feature that encodes current configuration, material properties, and contact situation, bridging the gap between partial observability and RL’s Markov requirements.

The remainder of this paper is organized as follows. Section II formalizes the deformation of Jacobians and their challenges in RL integration. Section III details our dual-phase architecture. Section IV presents comparative SOFA [22] results, followed by conclusions in Section V.

II Theoretical Background

II-A Jacobian Matrix

The Jacobian matrix serves as the cornerstone for robotic kinematic analysis, particularly in rigid-body systems. For an $n$-DOF serial manipulator, the Jacobian $\mathbf{J}\in\mathbb{R}^{6\times n}$ establishes the differential relationship between joint space and task space velocities through twist propagation:

[TABLE]

where each column $\mathbf{J}_{i}$ encodes the instantaneous screw motion induced by the joint $i$, with distinct formulations for revolute ($\mathbf{z}_{i}\times(\mathbf{p}_{e}-\mathbf{p}_{i})$) and prismatic ($\mathbf{z}_{i}$) joints. This formulation underpins essential robotic operations, including singularity analysis and resolved-rate motion control.

II-B Jacobian for Deformable Continuum Robot

The fundamental differences necessitate new Jacobian paradigms for continuum robots. The Jacobian matrix formulation for DCRs presents unique computational challenges:

• •

High-Dimensional Nonlinearity: The kinematic model of a DCR with $n$ degrees of freedom requires solving:

[TABLE]

where $\kappa_{i}$ and $\phi_{i}$ represent local curvature and twist, leading to exponentially increasing complexity with $m$ segments.

• •

Time-Varying Material Properties: The effective Young’s modulus $E(t)$ affects Jacobian updates:

[TABLE]

• •

Environmental Interactions: The external contact forces induce unmodeled deformations:

[TABLE]

where $\theta$ is the contact angle relative to the backbone tangent.

The aforementioned Jacobian paradigm was designed for mostly controllable and observable DCRs. It still faces challenges when being applied to objects like gastrointestinal endoscopes.

• •

Undriven Deformation: Over 85% of the endoscope’s distal section lacks direct actuation, making its deformation dependent on contact forces with surrounding tissues rather than commanded joint inputs. This breaks the basic Jacobian premise $J=\partial x/\partial\theta$ since $\theta$ no longer fully parameterizes the configuration space.

• •

Nonlinear Contact Dynamics: Tissue compliance introduces position-dependent stiffness matrices $K(x)$ that modify the effective Jacobian as $J_{eff}=J\cdot(I+K^{-1}J^{T}\partial F/\partial x)$ , where $F$ denotes contact forces. The lack of real-time force sensing in clinical settings makes $J_{eff}$ computationally intractable.

• •

Partial Shape Observability: Current endoscopic imaging provides only lumen surface data, leaving a large part of the DCR’s backbone curvature unobservable. This creates an underdetermined system for Jacobian estimation.

II-C Markov Processes and Reinforcement Learning

II-C1 Foundations of Markov Decision Processes

The Markov property $P(S_{t+1}|S_{t})=P(S_{t+1}|S_{t},S_{t-1},...)$ provides the theoretical bedrock for RL. An MDP quintuple $\langle\mathcal{S},\mathcal{A},P,R,\gamma\rangle$ enables policy optimization through Bellman equations:

[TABLE]

II-C2 Non-Markovian Dynamics in DCRs

DCR systems fundamentally violate Markov assumptions through:

• •

Partial Observability: Measured states $\mathbf{o}_{t}=[\mathbf{x}_{t},\mathbf{F}_{t}]$ capture only a limited subset of backbone curvature states, leaving key dynamic features unobserved.

• •

Historical Dependency: Viscoelastic hysteresis introduces path dependence:

[TABLE]

• •

Non-Stationary Transitions: Tissue remodeling creates time-varying dynamics:

[TABLE]

In summary, to make good use of RL in DCRs, instead of simply inputting observations as state information to the model, more information is needed for state estimation. The strategy to obtain more information is stated in the following section.

III Methodology

The preceding analysis reveals that effective RL planning for DCRs like endoscopic robots requires enhanced state estimation beyond basic observation inputs. While Jacobian matrices provide fundamental motion relationships in robotics, their direct computation becomes intractable for passively deformable structures due to un-actuated segments and limited sensing. Our JEDP-RL framework addresses this through phased Jacobian estimation and state augmentation, as shown in Algorithm 1 and detailed below.

III-A Phase 1: Local Jacobian Estimation

III-A1 Exploratory Action Generation

At each planning step $t$, we apply small-scale orthogonal perturbations $\delta\mathbf{a}_{i}\in\mathbb{R}^{d_{a}}$ across $n$ exploration steps, generating the action matrix:

[TABLE]

In the later experiment of this paper, intuitively, $\delta\mathbf{A}$ is set to $\lambda{I}$ to ensure orthogonality, where $\lambda$ is a small constant. We will explore learnable exploratory action generation in our future work to exploratory strategy maximizes information gain while minimizing robot displacement.

III-A2 Corresponding Observation Recording

Corresponding workspace displacements are recorded through endoscopic vision and force sensing:

[TABLE]

III-A3 Local Jacobian Approximation

We compute the deformation Jacobian via regularized least-squares:

[TABLE]

where $\lambda=0.01$ prevents singularity.

III-B Phase 2: Jacobian-Informed Policy Optimization

III-B1 State Augmentation

The augmented state vector combines instantaneous observations with estimated Jacobian:

[TABLE]

III-B2 Constrained Policy Optimization

We employ proximal policy optimization (PPO) [15] with adaptive KL constraints:

[TABLE]

the KL threshold $\delta$ adapts dynamically based on policy entropy.

Through local exploration, the reverse-engineered Jacobian matrix $\hat{\mathbf{J}}$ encodes static/dynamic, robot-environment interactions, enabling policy generalization capabilities across different objects and environments, adaptability to dynamic conditions, and robustness against disturbances.

IV Experiment

IV-A Simulation Setup

The simulation framework was designed to replicate the intricate physical interactions between a Deformable Continuum Robot (DCR) and dynamic biological environments. This experimental setup aimed to bridge the gap between simulated training environments and real-world clinical scenarios through high-fidelity physical modeling.

All simulation experiments, including DRL training and evaluations in SOFA, were executed on a 12th Gen Intel® Core™ i7-12700K CPU with 20 cores. The SOFA scene operated at 40 Hz, while the simulation training rate was maintained at 4 Hz.

IV-A1 Simulation Environment

The physics-based simulation, implemented within the SOFA framework, incorporated three fundamental components essential for realistic soft tissue interaction. First, the tendon-driven actuation system comprises four independently controlled cables (C1-C4) and one axial displacement mechanism (M). Second, the hyperelastic material model with Young’s modulus (YM) set to 15,000 and Poisson’s ratio of 0.3 provided biologically relevant deformation characteristics. Third, the dynamic contact resolution mechanism processed six collision detection cycles per training iteration to accurately capture progressive tissue deformation.

As depicted in Fig. 2(a), the simulation environment introduced time-varying peristaltic forces through sinusoidal waveforms to emulate natural gastrointestinal motility. These periodic disturbances created dynamic environmental challenges that tested the control policies’ ability to maintain navigation accuracy under continuously changing conditions.

The target selection methodology, illustrated in Fig. 2(b), employed a rigorous randomization protocol to ensure evaluation fairness. During training, destinations were selected from targetlist_A containing 10 predefined positions within the green zone, while evaluation utilized the distinct targetlist_B with 10 previously unseen targets. Both lists were generated using identical randomization seeds to maintain spatial distribution consistency while preventing target position overlap between the training and testing phases. All quantitative metrics demonstrated in the testing phase were computed as averaged values across 10,000-step data acquisition sequences, ensuring the performance measurements’ statistical reliability.

IV-A2 DRL Training Configurations

The deep RL implementation compared two distinct policy architectures: the baseline Proximal Policy Optimization (PPO-RL) and our proposed Jacobian-Enhanced Dual-Phase Reinforcement Learning (JEDP-RL). The baseline PPO-RL policy operated on a 15-dimensional state vector encompassing end-effector position (3D), velocity (3D), previous actuation values (5D), contact force vector (3D), and binary contact status (1D).

JEDP-RL enhanced this state representation through systematic integration of local exploratory actions designed to estimate Jacobians. Exploratory actions $\delta\mathbf{A}$, were set to $0.05I$, enabling the estimation of local deformation gradients without compromising system stability. The action space for both policies was constrained to $[-0.4,0.4]^{5}$ to ensure mechanically safe operation, with the five control parameters governing four tendon actuators and one axial displacement mechanism.

The neural network architecture employed four fully-connected layers with 256, 128, 64, and 32 neurons, respectively, using Tanh activation functions throughout. This configuration balanced representational capacity with computational efficiency, enabling effective policy learning in high-dimensional state spaces while maintaining real-time training feasibility on standard GPU hardware.

IV-B Training Performance

The training performance analysis revealed two significant advantages of the JEDP-RL approach under dynamic deformation conditions. First, as shown in Fig. 3(a), the proposed method achieved policy convergence in 0.36M training steps compared to PPO-RL’s 1.12M training steps, representing a 3.2 times reduction in required training iterations. This accelerated convergence stems from JEDP-RL’s dual-phase exploration strategy, which systematically combines global policy updates with localized Jacobian estimation through local exploratory motions.

Second, post-convergence navigation efficiency metrics demonstrated JEDP-RL’s superior navigation efficiency. As quantified in Fig. 3(b), the average navigation steps (NS) decreased from 32.5 steps for PPO-RL to 26 steps for JEDP-RL, corresponding to a 25% improvement.

The success rate (SR) is defined as the distance between the end-effector of DCR and the target is less than 3 mm. The SR of JEDP-RL achieves 100%(vs. PPO-RL’s 99%). The results of SR reveal the precision in navigation both for PPO-RL and JEDP-RL in training environments.

The proposed methodology introduces a controlled efficiency-effectiveness trade-off: while the integration of exploratory motions elevates per-step computational expenditure, systematic exclusion of collision verification during exploration phases (justified by the inherently low collision likelihood associated with minimal exploratory actions) preserves comparable training efficiency throughput between JEDP-RL and PPO-RL.

Trajectory length (TL) from the initial position to the target during the whole navigation process was adopted as the primary cost metric due to its demonstrated cross-platform consistency, exhibiting small variations between simulated and physical deployments. Comparative analysis reveals that JEDP-RL’s TL of 21.39 mm shows a slight 9.4% increase over PPO-RL’s 19.55 mm. This marginal TL elevation constitutes an acceptable compromise given the significant improvement in convergence speed and environment adaptability, which will be demonstrated in the following part of the experiments. We will conduct further research to develop adaptive exploration scheduling in the future to optimize this trade-off, potentially reducing cost penalties while preserving advantages.

IV-C Generalization Experiments

Generalization capability was assessed through three test scenarios: altered material properties, intensified peristalsis, and cross-anatomy deployment (vascular system).

IV-C1 Generalization to Environmental Dynamic Parameter Variations

The generalization capability evaluation first examined performance under modified environmental dynamics, including doubled peristaltic force amplitude and altered tissue material properties. As shown in Table II, JEDP-RL maintained a 95% success rate compared to PPO-RL’s 89% with 7.2% higher TL cost (41.6 mm vs. 38.8 mm). The average positioning error decreased from 2.34 mm to 1.89 mm, demonstrating improved navigation precision under dynamic environmental conditions.

IV-C2 Generalization to Robot Material Property Variations

Experimental evaluation under varying material properties revealed JEDP-RL’s superior adaptability to mechanical parameter deviations. When tested with substantially softer materials (YM=10,000), JEDP-RL demonstrated exceptional generalization capability, achieving a near-perfect 98% success rate compared to PPO-RL’s 72%. This performance superiority extended across all metrics, showcasing 27.3% lower average positioning error (1.78 mm vs. 2.45 mm), and 35% shorter TL. This experimental validation aligns with theoretical expectations, where softer materials induce heightened structural unpredictability that critically undermines model-free approaches like PPO-RL lacking explicit state estimation mechanisms. By contrast, JEDP-RL’s local exploration protocol effectively acquires Jacobian information, enabling precise deformation prediction and control compensation in compliant tissue environments.

Under stiff material conditions (YM=25,000), JEDP-RL maintained its performance advantage with a 92% success rate versus 87% for PPO-RL, coupled with an 8.9% improvement in positioning accuracy (2.15 mm vs. 2.36 mm). While exhibiting a marginal 10% trajectory cost increase in this regime, the associated 5% success rate enhancement and 8.9% error reduction justify this controlled efficiency trade-off, particularly considering clinical priorities favoring navigation reliability over minimal path optimization.

IV-C3 Generalization to Unseen Environment

The final generalization test evaluated policy performance in a fundamentally different anatomical environment: a simulated blood vessel with distinct biomechanical characteristics. As shown in Fig. 4, this environment introduced pulsatile flow forces and a challenging bifurcation geometry.

Fig. 5 presents the fine-tuning and evaluation results of both policies. As shown in Fig. 5(a), the episode reward mean during fine-tuning indicates a steady improvement for both policies, but JEDP-RL achieves a significantly higher reward compared to PPO-RL. This suggests that JEDP-RL adapts more efficiently to the new environment and learns effective navigation strategies in the presence of dynamic deformations. Fig. 5(b) shows that the best average episode length required for PPO-RL to reach the target is $19\%$ higher than that of JEDP-RL. This indicates that JEDP-RL achieves the target locations with more efficient trajectories, highlighting its advantage in adapting to environmental changes. Additionally, Fig. 5(c) compares the zero-shot performance of both policies before fine-tuning, showing that JEDP-RL (red) achieves $11\%$ higher SR compared to PPO-RL (blue). After fine-tuning, JEDP-RL further outperforms PPO-RL, achieving a maximum SR of $83\%$, which is $33\%$ higher than that of the fine-tuned PPO-RL.

The bifurcation navigation analysis reveals fundamental differences in policy adaptation strategies. While PPO-RL demonstrates branch-specific competence (Upper Branch: 82% SR, Lower Branch: 7% SR), JEDP-RL maintains balanced performance across both anatomical structures (UB: 82% SR, LB: 69% SR). This 9.8:1 success rate ratio disparity between branches for PPO-RL versus JEDP-RL’s 1.2:1 ratio underscores the proposed method’s capacity to develop generalized navigation strategies rather than environment-specific solutions.

V Conclusion

This study establishes the efficacy of Jacobian exploratory dual-phase RL for deformable continuum robot (DCR) navigation in dynamic anatomical environments. The experimental results across stomach and vascular simulations demonstrate three fundamental advancements:

1). Accelerated Policy Convergence: The JEDP-RL framework achieved 3.2× faster convergence than PPO-RL (Fig. 3(a)), validating the effectiveness of local Jacobian exploration. This acceleration stems from the dual-phase architecture that decouples mechanical parameter identification from policy optimization, effectively addressing the time-varying kinematics problem in DCRs.

2). Enhanced Navigation Efficiency: Post-convergence analysis revealed JEDP-RL requires 25% fewer steps than PPO-RL to reach targets (Fig. 3(b)).

3). Superior Generalization Capability: Under material property variations, JEDP-RL achieved over 92% SR. In unseen vascular environments, JEDP-RL achieved an 83% SR after finetuning versus PPO’s 50% (Fig. 5(c)).

While the suggested method yields significance, the increase in trajectory length caused by exploration remains a challenge that requires further attention. For future research, we will further probe into learnable exploration action generation and relevant studies on sim2real algorithms.