Learning Dolly-In Filming From Demonstration Using a Ground-Based Robot

TL;DR

This paper introduces a demonstration-based learning approach using GAIL for autonomous cinematic camera control with a ground robot, outperforming traditional RL methods and enabling real-world deployment without extensive tuning.

Contribution

It presents a novel GAIL-based imitation learning method for robotic filmmaking that transfers from simulation to real-world without fine-tuning, reducing technical effort and enhancing creative control.

Findings

GAIL policy outperforms PPO in simulation with higher rewards and faster convergence.

The learned policy transfers directly to real robots, outperforming previous methods.

The approach enables real-time, stylized camera control for filmmakers with minimal setup.

Abstract

Cinematic camera control demands a balance of precision and artistry - qualities that are difficult to encode through handcrafted reward functions. While reinforcement learning (RL) has been applied to robotic filmmaking, its reliance on bespoke rewards and extensive tuning limits creative usability. We propose a Learning from Demonstration (LfD) approach using Generative Adversarial Imitation Learning (GAIL) to automate dolly-in shots with a free-roaming, ground-based filming robot. Expert trajectories are collected via joystick teleoperation in simulation, capturing smooth, expressive motion without explicit objective design. Trained exclusively on these demonstrations, our GAIL policy outperforms a PPO baseline in simulation, achieving higher rewards, faster convergence, and lower variance. Crucially, it transfers directly to a real-world robot without fine-tuning, achieving more consistent framing and subject alignment than a prior TD3-based method. These results show that LfD offers a robust, reward-free alternative to RL in cinematic domains, enabling real-time deployment with minimal technical effort. Our pipeline brings intuitive, stylized camera control within reach of creative professionals, bridging the gap between artistic intent and robotic autonomy.

Tables

Table 1. TABLE I : Comparison of GAIL and PPO on dolly-in shots. GAIL outperforms PPO in both task settings using only 25 expert demonstrations. TD3 performs best but requires significantly more experience, highlighting the trade-off between sample efficiency and performance.

Method	Base Control	Full Control
GAIL (25 Demos)	-116.3 ± 36.0	-126.3 ± 25.6
PPO Baseline	-127 ± 25.0	-132 ± 24.1
Expert Demonstrations	-142.1 ± 29.5	-118.1 ± 39.8

Table 2. TABLE II : Sim2Real performance comparison between the TD3 baseline  [ 2 ] and our proposed GAIL (PPO) method, across start positions. Metrics include cumulative reward, object area, and X/Y position in both simulation and real-world deployments. The Sim2Real Rank Correlation Coefficient (SRCC) quantifies consistency between simulated and real-world outcomes. SRCC values are colour-coded as follows: bright green (very strong: 0.8–1), light green (strong: 0.6–0.79), light yellow (moderate: 0.4–0.59), light red (weak: 0–0.39), and darker red (negative: ≤ \leq 0) . Higher SRCC indicates better transferability. This table summarises key performance outcomes supported by detailed error breakdowns in Tables  III – V .

		Cum. Reward		Object Area			Object Position (X-axis)			Object Position (Y-axis)
Method	Start Pos.	Sim.	Real.	Sim.	Real.	SRCC	Sim.	Real	SRCC	Sim.	Real	SRCC
TD3 Baseline	Left	-170.77	-176.27	10.12	9.63	0.69	59.42	61.22	0.52	44.0	44.54	0.65
	Right	-166.03	-166.35	10.09	10.38	0.80	59.32	59.08	0.72	44.1	44.23	0.83
	Centre	-129.93	-134.96	10.09	11.09	0.46	59.79	55.67	0.56	44.0	46.51	0.69
GAIL (PPO)	Left	-145.19	-143.07	10.23	10.20	0.97	59.82	59.59	0.72	38.92	37.22	0.58
	Right	-130.95	-132.16	10.20	10.0	0.98	59.78	59.88	0.76	38.96	37.26	0.61
	Centre	-117.60	-116.05	10.16	9.84	0.99	59.76	59.51	0.87	38.96	36.46	0.67

Table 3. TABLE III : Object area error across start positions. GAIL (PPO) achieves lower error in all cases, including a perfect match at the right position. Values show absolute deviation from the 10-unit framing target, with percentages indicating relative error. Colours indicate severity: green (low), yellow (moderate), red (high).

Start Pos.	Object Area Error
	TD3	GAIL (PPO)	% Improv.
Left	0.37 (3.7%)	0.20 (2.0%)	+45.9%
Right	0.38 (3.8%)	0.00 (0.0%)	+100.0%
Centre	1.09 (10.9%)	0.16 (1.6%)	+85.3%

Table 4. TABLE IV : Horizontal (X-axis) alignment error across start positions. GAIL (PPO) outperforms the TD3 baseline, reducing error by over 88% in the most challenging scenario. Errors are shown relative to the target screen centre (60).

Start Pos.	X Position Error
	TD3	GAIL (PPO)	% Improv.
Left	1.22 (2.03%)	0.41 (0.68%)	+66.4%
Right	0.92 (1.53%)	0.12 (0.20%)	+87.0%
Centre	4.33 (7.22%)	0.49 (0.82%)	+88.7%

Table 5. TABLE V : Vertical (Y-axis) alignment error from the target position (40) across start positions. GAIL (PPO) substantially improves vertical centring compared to TD3, especially from the centre start. Error values reflect absolute deviation, with colour highlighting the severity.

Start Pos.	Y Position Error
	TD3	GAIL (PPO)	% Improv.
Left	4.54 (11.35%)	2.78 (6.95%)	+38.8%
Right	4.23 (10.58%)	2.74 (6.85%)	+35.2%
Centre	6.51 (16.28%)	3.54 (8.85%)	+45.6%

Equations

$$L^{\text{CLIP}}(\theta )={\hat{\mathbb{E}}}_t\left[\min \left(r_t(\theta ){\hat{A}}_t,\text{clip}(r_t(\theta ),1-ϵ,1+ϵ){\hat{A}}_t\right)\right],$$

$$\underset{{\pi }_{\theta }}{\min }\underset{D_{\varphi }}{\max }{\mathbb{E}}_{{\pi }_E}[\log D_{\varphi }(s,a)]+{\mathbb{E}}_{{\pi }_{\theta }}[\log (1-D_{\varphi }(s,a))].$$

$$a=\{\begin{array}{ll}[\text{throttle},\text{steering}]& \text{(Base)}\\ [\text{throttle},\text{steering},\text{pan},\text{tilt}]& \text{(Full)}\end{array}$$

$$r_t={\lambda }_{\text{area}}\cdot \Delta A_t-{\lambda }_{\text{steer}}\cdot \Delta {\dot{\theta }}_t^2$$

$$r_t={\lambda }_{\text{area}}\cdot \Delta A_t-{\lambda }_{\text{steer}}\cdot \Delta {\dot{\theta }}_t^2-{\lambda }_{\text{cam}}\cdot (\Delta {\dot{\varphi }}_t^2+\Delta {\dot{\psi }}_t^2)$$

Full text

Trained exclusively on these demonstrations, our GAIL policy outperforms a PPO baseline in simulation, achieving higher rewards, faster convergence, and lower variance. Crucially, it transfers directly to a real-world robot without fine-tuning—achieving more consistent framing and subject alignment than a prior TD3-based method. These results show that LfD offers a robust, reward-free alternative to RL in cinematic domains, enabling real-time deployment with minimal technical effort. Our pipeline brings intuitive, stylised camera control within reach of creative professionals—bridging the gap between artistic intent and robotic autonomy.

I Introduction

Filmmaking demands camera motion that is both precise and expressive. Automating this with ground-based robots introduces challenges that span both technical accuracy and creative sensitivity. When successful, such systems enable consistent, repeatable shots—freeing filmmakers to focus on dynamic and artistically demanding scenes.

Data-driven methods offer promising solutions by sidestepping explicit environmental modeling [1]. Among them, reinforcement learning (RL) has shown the ability to learn camera behaviors through trial and error. Recent work has demonstrated automated dolly-in shots with reliable zero-shot sim-to-real transfer on wheeled robots [2]. Yet despite these advances, ground-based robotic cinematography remains underexplored compared to its aerial counterpart.

However, RL’s practical limitations—handcrafted reward functions, long training times, and high computational cost—present barriers to adoption. Designing aesthetic rewards is difficult, making RL workflows poorly aligned with intuitive, creative filmmaking.

Learning from Demonstration (LfD) learns control policies from human camera trajectories, capturing artistic intent without reward engineering. Timing, framing, and composition are encoded in the demos, making LfD effective for cinematography.

While LfD has seen broad application in manipulation and navigation tasks [3], its use in ground-based filmmaking remains limited. Prior work has primarily focused on drones [4, 5], while wheeled robots offer unique advantages: stability, precision, and repeatability—key attributes for stylised camera work.

In this paper, we extend the zero-shot sim-to-real framework introduced in [2], replacing their RL-based method with an LfD-based pipeline. Our approach removes the need for handcrafted rewards, accelerates training, and better aligns with creative workflows. Policies trained from expert joystick demonstrations in simulation transfer directly to real-world deployment, achieving consistent, expert-like dolly-in shots without fine-tuning.

While our method builds on established imitation learning techniques, our contribution lies in demonstrating that LfD—when applied within this framework—enables robust, accessible cinematographic automation with minimal tuning. Compared to RL, our approach lowers the barrier to entry for both filmmakers and robotics practitioners.

Our key contributions are:

A complete LfD pipeline for robotic cinematography, from expert data collection to policy training and real-world deployment. 2. 2.

Quantitative comparison of LfD and RL performance in simulation, relative to expert trajectories. 3. 3.

Real-world validation demonstrating zero-shot sim-to-real transfer with consistent cinematic behaviour.

The remainder of this paper details our problem formulation (Section II), data collection and LfD training pipeline (Sections III–IV), experimental setup and results (Section V), and concludes with discussion and future directions (Sections VI–VII).

II Problem Formulation

The dolly-in shot is a classic cinematographic technique where the camera moves smoothly toward a subject while maintaining precise centring and gradual scaling within the frame. The shot concludes once the subject reaches a desired size. Coordinating motion and framing is a creative task that can be hard to express algorithmically.

Reinforcement learning (RL) has been applied to this problem [2], but its success depends on carefully engineered reward functions that promote smooth motion, framing stability, and aesthetic composition. These reward signals are often hard to define, especially when the goal is perceptual or stylistic. As a result, RL-based pipelines typically require extensive tuning and domain knowledge, limiting accessibility for filmmakers or creative practitioners.

We propose an alternative: Learning from Demonstration (LfD). Rather than explicitly defining success with handcrafted objectives, LfD learns policies directly from expert camera trajectories. These demonstrations naturally encode timing, composition, and stylistic nuance, allowing the robot to reproduce expert-like behaviour without handcrafted rewards.

Our method follows a three-stage Learning from Demonstration pipeline: expert demonstration, policy training, and deployment. This is illustrated in Figure 1, which outlines the full flow from joystick-operated data collection to simulation and real-world evaluation. To enable direct comparison with prior work, we adopt the same simulation environment, robot models, and camera setup as in [2], but replace the RL core with an LfD-based approach.

III Demonstration Collection

To train our Learning from Demonstration (LfD) pipeline, we collected expert camera trajectories in simulation via joystick teleoperation. A single operator controlled a ground robot with a virtual camera using an Xbox controller, interfaced through Pygame [6] in a PyBullet-based simulation environment [7, 8]. This setup enabled expressive, real-time control with minimal training overhead, and reflects standard practice in LfD, where teleoperation is commonly used to capture expert behaviour [3, 9] (Figure 2).

Each demonstration recorded a complete dolly-in trajectory as a sequence of state-action pairs. We collected 25 demonstrations per task (Base and Full Control), varying the robot’s starting position, orientation, and lighting to promote generalisation. Data were logged using the TrajectoryAccumulator from the Imitation Library [10], producing standardised datasets of observations, actions, and transitions.

To ensure consistency, a single expert operator performed all demonstrations. The spatial distribution of initial positions is shown in Figure 3, with a representative view of the simulation environment in Figure 4.

IV Learning from Demonstration (LfD)

Robotic cinematography presents a challenge: artistic intent is difficult to encode explicitly. While reinforcement learning (RL) has been applied to automate camera motion, it depends on handcrafted rewards that are often subjective and costly to design. In contrast, Learning from Demonstration (LfD) enables policies to learn directly from expert trajectories—capturing framing, timing, and style without manual reward design.

We model the task as a Markov Decision Process (MDP) with states $S$, actions $A$, transitions $T$, and discount factor $\gamma$, where the reward $R$ is unknown or hard to define. Instead, we assume access to expert trajectories $\tau=\{(s_{0},a_{0}),\dots,(s_{n},a_{n})\}$, from which the goal is to recover a policy $\pi(a|s)$ that mimics expert behaviour.

We evaluate two learning strategies: Proximal Policy Optimisation (PPO), a reinforcement learning baseline trained with handcrafted rewards, and Generative Adversarial Imitation Learning (GAIL), our primary LfD method, trained solely on expert demonstrations without access to rewards.

IV-A PPO Baseline

Proximal Policy Optimisation (PPO) [11] is an on-policy RL algorithm optimising the clipped objective:

[TABLE]

where $r_{t}(\theta)$ is the policy ratio and $\hat{A}_{t}$ the advantage. We use the Stable Baselines 3 implementation [12] with the same reward structure as in. [2] (see Section V-A). While TD3 performed well in previous work, we use PPO as our RL baseline due to its simplicity, widespread use, and compatibility with the on-policy imitation framework provided by the imitation library [10].

IV-B GAIL for Cinematic Imitation

Generative Adversarial Imitation Learning (GAIL) [13] casts imitation as a two-player game:

A discriminator $D_{\phi}(s,a)$ distinguishes expert from agent behaviour. 2. 2.

A policy $\pi_{\theta}(a|s)$ learns to fool the discriminator.

The training objective is:

[TABLE]

We implement GAIL in PyTorch using the imitation library [10], with learning rate $1\times 10^{-4}$ and batch size 64. The learned reward signal allows the agent to align with expert behaviour without explicit design.

Although we evaluated newer methods (e.g., LS-IQ, IQ-Learn), they proved unstable under our constraints. GAIL offered greater robustness and convergence, making it the most suitable option.

Prior work has validated GAIL in manipulation and construction [14, 15]; we extend it to camera control, which requires smooth, perceptual, and temporally consistent outputs—properties well suited to GAIL’s formulation.

V Experimental Results and Analysis

This section details the experimental setup and outcomes, validating the effectiveness of RL and LfD agents in performing autonomous dolly-in shot tasks.

V-A Experimental Setup

We evaluate our Learning from Demonstration (LfD) pipeline in the high-fidelity PyBullet simulation environment introduced in [2], which simulates a ground-based filming robot with a controllable camera. To ensure direct comparability, we adopt the same robot model, environment, and reward structure used in PPO training. Our contribution is the integration of an LfD pipeline based on expert joystick demonstrations.

Task Description

The target behaviour is a dolly-in shot: the robot moves toward a static subject while keeping it centered and smoothly scaled within the frame. This requires coordinated locomotion and camera control.

We evaluate two task variants:

Base Control: Controls throttle and steering. 2. 2.

Full Control: Adds pan and tilt for active framing.

Action Space

Depending on the control mode, the policy outputs either:

[TABLE]

All actions are scaled to respect physical limits and passed through tanh activations for smooth transitions.

PPO Reward Function

The PPO baseline is trained using a scalar reward adapted from  [2], combining two terms:

• •

Framing Progress: Rewards forward motion that increases the subject’s size in the frame while keeping it centered.

• •

Motion Smoothness: Penalises abrupt changes in control inputs to promote cinematic stability.

For the Base Control task (throttle and steering), the reward at each timestep is:

[TABLE]

where $\Delta A_{t}$ is the change in object area (subject scale) and $\Delta\dot{\theta}_{t}$ is the change in steering rate.

For the Full Control task (adds pan and tilt), the smoothness term is extended to include camera motion:

[TABLE]

where $\Delta\dot{\phi}_{t}$ and $\Delta\dot{\psi}_{t}$ are changes in pan and tilt rates, respectively.

Hyperparameters $\lambda_{\text{area}},\lambda_{\text{steer}},\lambda_{\text{cam}}$ are tuned via grid search to balance framing accuracy and motion stability.

Training Protocol

Each agent is trained for 1 million timesteps using 1500-step episodes. We run three random seeds per configuration to assess variance. Evaluation is based on final episodic reward (100 trials), convergence speed, and stability.

V-B Simulation Experiments

We compare Generative Adversarial Imitation Learning (GAIL) and Proximal Policy Optimisation (PPO) on the dolly-in cinematography task across two settings: Base Control (throttle and steering) and Full Control (adds pan and tilt). GAIL is trained on 25 joystick-operated demonstrations, while PPO learns from scratch using the handcrafted reward defined in [2]. For additional context, we include the TD3 results from the same work as a high-performing, reward-engineered baseline requiring substantially more training.

Overall Performance.

As shown in Table I, GAIL consistently outperforms PPO across both task settings. In Base Control, GAIL achieves an average reward improvement of 8.4%, and in Full Control, a 4.3% gain. GAIL also exhibits faster convergence and lower variance across three training seeds. Although TD3 achieves the highest reward, it is considerably less sample efficient.

Impact of Demonstration Diversity.

To assess how the diversity of demonstrations affects policy learning, we compare GAIL agents trained on 25 demonstrations sampled from low (1), moderate (3), and high (5) distinct starting positions. Figure 5 presents the resulting learning curves. Greater start position diversity leads to improved generalisation and more stable learning. In both tasks, high-diversity GAIL matches or outperforms PPO, underscoring that not only the quantity but the distribution of demonstrations is crucial for robust behavior.

V-C Real-world Experiments

To validate zero-shot Sim2Real transfer, we deploy the GAIL (PPO) policy, trained entirely in simulation using 25 diverse demonstrations, onto a physical ground robot with no fine-tuning. This tests the system’s ability to generalise cinematic behaviour under real-world conditions, mirroring the simulation setup in Experiment 1.

Following  [2], we evaluate each method from three canonical starting positions (left, centre, right), recording cumulative reward and camera framing metrics (object area and X/Y position). As in simulation, rewards are computed per step, and framing targets remain aligned with expert intent. Sim2Real fidelity is quantified using the Sim2Real Correlation Coefficient (SRCC, [16]), measuring alignment between simulated and real-world outcomes.

Results (Table II) show that GAIL (PPO) consistently outperforms the TD3 baseline across all start positions. GAIL achieves higher cumulative rewards and significantly stronger SRCC scores, including near-perfect correlation on object area ($\geq$0.97). Additional error breakdowns (Tables III–V) show improvements of up to 100% in object area accuracy and over 88% in horizontal centering.

To test generalisation beyond seen configurations, we conduct 75 additional trials from randomised positions. The robot successfully completes the dolly-in in all 75 trials (100%) where the subject is visible, demonstrating robust generalisation and cinematic framing under variation.

Figure 7 visualises typical real-world trajectories. Subfigure (a) shows one trajectory with heading vectors overlaid, while subfigure (b) overlays several runs from a separate start location, illustrating consistency across executions. Camera views confirm that the subject remains centred and appropriately scaled, validating cinematic quality.

These results validate that cinematic policies learned via imitation in simulation can reliably generalise to real-world deployment, reducing the need for domain-specific tuning or reward design.

VI Discussion and Limitations

Our results show that Learning from Demonstration (LfD), implemented via GAIL, offers a practical alternative to reinforcement learning (RL) for robotic cinematography. GAIL-trained policies, using only 25 demonstrations, outperform PPO in simulation and transfer directly to real hardware without fine-tuning—removing the need for reward design or extensive tuning. This lowers the barrier for creative users and enables rapid deployment of stylised robotic behaviour.

In simulation, GAIL outperforms PPO, faster convergence, higher rewards, lower variance, especially with diverse demos. Caveats: single-expert data and a simplified simulator lacking noise, occlusions, and dynamics; next steps include richer environments or style-conditioned policies

Real-world tests confirm zero-shot transfer under controlled conditions: flat ground, static subject, and consistent lighting. Performance in more dynamic scenarios (e.g., subject re-identification, occlusions) remains untested. Despite low-cost sensors and actuators, GAIL handled real-world noise well—suggesting robustness—but higher-end hardware could unlock more advanced behaviours like adaptive framing or camera motion.

While GAIL avoids reward tuning, it introduces adversarial training challenges. Compared to behavioural cloning (BC), it requires more hyperparameter tuning and stable demonstrations. For longer or more complex tasks, data collection may become a bottleneck. Tooling for streamlined demonstration capture or hybrid LfD–RL pipelines could improve scalability and creative flexibility.

VII Conclusions and Future work

e presented a data-driven pipeline for robotic cinematography using Learning from Demonstration (LfD), enabling ground-based robots to perform expert-level dolly-in shots without handcrafted rewards. Using joystick-operated demonstrations, we trained GAIL policies in simulation that transferred successfully to real-world deployment in a zero-shot manner—achieving consistent, cinematic behavior with no fine-tuning.

Compared to reinforcement learning, our approach improves learning efficiency, reduces engineering effort, and aligns better with creative workflows. Real-world results confirm strong performance and high simulation-to-deployment fidelity, supporting its practical utility.

Future work could explore multi-operator datasets for stylistic diversity, extend the system to dynamic subjects or complex camera motions (e.g., arcs, tracking shots), and incorporate semantic understanding of scene context or composition rules. These additions would expand applicability while preserving artistic intent.

Overall, our results show that LfD offers a robust foundation for creative robotics—bridging intuitive human demonstrations with real-world autonomy in service of cinematic storytelling.