Wireless Software Synchronization of Multiple Distributed Cameras

TL;DR

This paper introduces a software-based method for precisely synchronizing multiple smartphones over WiFi for multi-camera imaging, achieving microsecond accuracy without special hardware.

Contribution

It presents a novel two-stage algorithm for clock synchronization and image alignment across multiple commodity smartphones, enabling improved multi-view applications.

Findings

Achieves less than 250 microseconds synchronization accuracy.

Validates effectiveness with multi-camera rig experiments.

Enhances multi-view stereo and scene stitching quality.

Abstract

We present a method for precisely time-synchronizing the capture of image sequences from a collection of smartphone cameras connected over WiFi. Our method is entirely software-based, has only modest hardware requirements, and achieves an accuracy of less than 250 microseconds on unmodified commodity hardware. It does not use image content and synchronizes cameras prior to capture. The algorithm operates in two stages. In the first stage, we designate one device as the leader and synchronize each client device's clock to it by estimating network delay. Once clocks are synchronized, the second stage initiates continuous image streaming, estimates the relative phase of image timestamps between each client and the leader, and shifts the streams into alignment. We quantitatively validate our results on a multi-camera rig imaging a high-precision LED array and qualitatively demonstrate significant improvements to multi-view stereo depth estimation and stitching of dynamic scenes. We release as open source 'libsoftwaresync', an Android implementation of our system, to inspire new types of collective capture applications.

Figures

Tables

Table 1. TABLE I: Synchronization Accuracy of Naive vs Our Method

Method	Max error	Mean Abs Error	Stdev
Naive Wired	$180\mathrm{ms}$	$103\mathrm{ms}$	$50\mathrm{ms}$
Naive Bluetooth	$200\mathrm{ms}$	$69\mathrm{ms}$	$65\mathrm{ms}$
Naive WiFi	$677\mathrm{ms}$	$123\mathrm{ms}$	$84\mathrm{ms}$
Our Method	$0.121\mathrm{ms}$	$0.032\mathrm{ms}$	$0.025\mathrm{ms}$

Table 2. TABLE II: Distribution of Errors by Component

	Max error	Mean Abs Error	Stdev
Clock	$\le 121\mu s$	$\le 32\mu s$	$\le 25\mu s$
Phase	$21\mu s$	$7\mu s$	$5\mu s$
Total	$121\mu s$	$32\mu s$	$25\mu s$

Table 3. TABLE III: Network latency symmetry and bias: Mean vs. Min

	Mean	Min
Leader to client	$1,133\mu s$	$517\mu s$
Client to leader	$1,878\mu s$	$479\mu s$
Abs. latency difference	$746\mu s$	$38\mu s$
Round-trip latency	$3,012\mu s$	$996\mu s$
NTP bias	$372\mu s$	$19\mu s$

Equations

$$\theta$$

$$U(0,L)+S(\mathrm{mod}T)\approx U(0,L)(\mathrm{mod}T)\approx U(0,T),$$

$$R=\frac{\log 0.05}{\log (1-ϵ/T)}.$$

$${ϵ}_{total}={ϵ}_{clock}+{ϵ}_{phase}.$$

Full text

I Introduction

Many computer vision algorithms require multiple images from different viewpoints as input. These algorithms can fail spectacularly when applied to dynamic scenes if the images are not captured at the same time. Depth from stereo, view interpolation, and view supervision losses are just a few examples of such algorithms. We provide a solution to temporally synchronize the cameras of multiple inexpensive smartphones, so that they can capture images simultaneously (Fig. 1).

Our solution is software-based, does not require any additional hardware, and scales up to multiple devices. Furthermore, the devices are synchronized prior to capture and can be located in any physical arrangement as long as they can communicate over a wireless network.

Traditionally, synchronizing cameras in advance of capture is solved by using specialized hardware such as analog video gen-lock or the IEEE 1394 isochronous interface. While effective and reliable, hardware solutions require cumbersome wires, limiting their portability, and more importantly, are unavailable on the vast majority of cameras. For non-specialized cameras, this limits usage to nearly static scenes where several frames of timing error is acceptable.

Another option is to capture image sequences and align the frames after capture. A clapperboard or similar “oracle” device that is visible to all cameras can assist in aligning the frames. However, accuracy is limited to half a frame duration. Post-processing methods to obtain sub-frame alignment exist [1, 2, 3, 4, 5], but can break down for low-texture or noisy scenes where finding correspondences is hard. Even if these algorithms yield perfect sub-frame alignment, it is necessary to temporally interpolate the frames to the aligned sub-frame time [3], which is another challenging problem. In addition, such methods may require capturing much more data than necessary to ensure that the same instant in time is captured by all cameras.

In contrast, our system is capable of achieving a synchronization accuracy of less than $250\text{\,}\mathrm{\SIUnitSymbolMicro s}$ before capture, allowing for the multi-view simultaneous capture of highly dynamic phenomena such as sports action shots, birds in flight, and splashing liquids (Fig. 2). To acquire synchronized images, users are only required to install our software, position the phones, and press the shutter button on one of the phones. Our system does not look at image content; instead, it uses only image timestamps and wireless messages for synchronization. This means the cameras can have non-overlapping fields of view and can be placed in any configuration. For example, a hand-held, inexpensive light field capture rig can be created by placing the cameras in a square array. Or the cameras can be placed in a linear array, so that a view interpolation algorithm can create a “bullet time” effect where the camera moves around an apparently frozen dynamic subject [6, 7].

Our system is especially useful for collecting data for deep learning algorithms that require synchronized images for training [8, 9, 10] and also need to be deployed on a smartphone. Were such data collected with hardware-synchronized non-smartphone cameras, biases introduced by differences in the sensor or lens could ruin the algorithm’s performance. A portable rig containing smartphones synchronized using our system provides an inexpensive method to collect such datasets in the wild.

Our system operates in two stages. First, the clocks of all the phones are synchronized to the clock of a leader device using a variant of the Network Time Protocol (NTP). This synchronization is only enough to achieve an accuracy of half a frame duration (e.g., $16\text{\,}\mathrm{ms}$). In the second stage, the cameras are instructed to capture a continuous stream of images. We then shift the phase of all client streams to that of the leader and achieve an accuracy better than $250\text{\,}\mathrm{\SIUnitSymbolMicro s}$.

In summary, we contribute the following:

• •

Two algorithms for efficiently phase aligning camera image streams to arbitrary precision.

• •

A system combining phase alignment with a variant of NTP that achieves an accuracy better than $250\text{\,}\mathrm{\SIUnitSymbolMicro s}$ on unmodified consumer smartphones over WiFi.

• •

A theoretical analysis and empirical confirmation of the accuracy and limits of our system.

We release as open source libsoftwaresync, an Android implementation of our system.

II Related Work

Precision time synchronization in a distributed system is a classically hard problem. The fact that each device is physically independent with its own hardware clock makes synchronization necessary even if networking were reliable with low communications latency. When applied to the camera synchronization problem, misalignment is primarily due to two factors. First, each device has an independent notion of the current time, which may drift and is at best only loosely aligned to an external source (e.g., GPS or cell network). Second, consumer cameras typically do not feature a mechanism by which image capture can be triggered to occur at a specific time. Instead, there is a large firmware and software stack between the camera hardware and the application that requests an image, which introduces additional variable latency. Without physical wires to synchronize all clocks and cameras in the network, it is challenging to trigger all cameras at the same time.

A number of systems focus on aligning video sequences and correcting for rolling shutter after they have been captured. Some systems use active illumination to tag frames from which an offset may be inferred [1]. Others rely on the scene content itself, by detecting abrupt lighting changes [2], tracking continuous periodic motion [3], or requiring motion be simple enough that the spatiotemporal transformation can be represented by a low-dimensional model [4, 5]. As the source imagery is fundamentally unsynchronized, these post-capture techniques all ultimately rely on optical flow and interpolation to bring images into sub-frame alignment. Our goal is to synchronize cameras before capture, ensuring they expose the same instant in time and sidestepping the frame interpolation problem.

Gu et al. [11] describe a simple scheme for projector-camera synchronization by forcing the camera to wait until the projector has displayed an image before triggering capture. This technique is straightforward to implement but cannot capture moving subjects, can introduce unacceptably long shutter lag, and is impractical for video sequences. Litos et al. [12] describe a system that uses the Network Time Protocol (NTP) [13] to align device clocks but neglects the variable latency between when software sends a shutter command and when the camera ultimately captures a frame. Petkovć et al. [14] describe a clever system for synchronizing a projector and a camera by using the camera itself to measure the signal propagation delay. But this requires a projector and also does not account for the variable latency in triggering a camera.

The system by Ahrenberg et al. [15] is closest in spirit to ours. Like our system, they rely on NTP to synchronize device clocks before sending a single recording command. Their system relies on the IEEE 1394 hardware trigger to keep pace, but like the other systems above, assumes that the delay between the software trigger command and image capture to be the same among all cameras. Our system does not require a hardware trigger and explicitly models the variable latency between the software trigger command and image exposure.

III Overview

We give a high level overview of our system. We first list our assumptions of the underlying system, which justify our algorithm and where it may be deployed.

III-A Setting

Our system consists of a collection of $N$ devices, each of which is equipped with a camera and a network card. In practice, these are smartphones with an integrated camera and WiFi. At startup, the devices connect over WiFi, synchronize their clocks, and puts their cameras into continuous streaming mode (Sec. IV). After a second stage where camera streams are phase aligned (Sec. V), initialization is complete. At any time after, when a capture request containing a timestamp arrives, all devices simply save the appropriate images from a ring buffer to disk. Note that this requested timestamp can be in the past, enabling zero shutter lag captures. Our goal is to do the above with reasonable latency and as such, rely on a few assumptions of the underlying hardware:

Modest variance in network latency. The first stage of our algorithm brings the devices’ local clocks into alignment. Theoretically, our system can tolerate arbitrary latency variance, although it may take an unacceptable amount of time to converge. To support arbitrary devices without hardware timestamping of network packets, we include in our latency model variable delays induced by the operating system’s networking stack. 2. 2.

Hardware camera timestamps. We require that the camera hardware tag images with timestamps in the domain of the device’s local clock. A key component of our algorithm is accurately modeling the time between requesting an image and when it is captured. If images were not timestamped in the same clock domain, this modeling would not be possible. iPhones and numerous Android devices support this feature. 3. 3.

Hardware image streaming We require that the camera hardware or firmware be able to latch capture settings such that it can stream frames indefinitely with low timing variance between frames. For example, if we request images at ISO 100, exposure time of $10\text{\,}\mathrm{ms}$, and a frame duration (time between frames) of $33\text{\,}\mathrm{ms}$, we should expect a stream of images with timestamps close to $33\text{\,}\mathrm{ms}$ apart. Most commercially available cameras and smartphones support this feature.

IV Synchronizing Device Clocks

Like previous work[12, 15], we use a variant of the Network Time Protocol (NTP) [13] to synchronize device clocks, for which we give a simplified overview. This step is unnecessary if all devices had support for the Precision Time Protocol (PTP) [16], which provides sub-microsecond hardware synchronization. However, PTP hardware is typically not available on consumer smartphones. In our system, the user designates one device as the leader and creates a WiFi hotspot. The remaining $N-1$ clients connect to the leader and estimate their clock offsets. Typically, device clock synchronization is only performed once at the beginning of a capture session. However, depending on the application (e.g., very long sessions such as time lapses), one may want to periodically re-synchronize clocks to account for clock drift or exploit better WiFi conditions. In our experiments, clocks remained stable over the course of one hour.

A single message in the synchronization routine consists of the following handshake (Fig. 3) between the leader and one of the client devices (multiple clients are handled independently):

At time $t_{0}$ in the leader’s clock domain the leader sends the message “$t_{0}$”. 2. 2.

At time $t_{1}$ in the client’s clock domain, it receives the message “$t_{0}$”. 3. 3.

At time $t_{2}>t_{1}$ in the client’s clock domain, it sends the message “$(t_{0},t_{1},t_{2})$”. 4. 4.

At time $t_{3}>t_{0}$ in the leader’s clock domain, it receives the message “$(t_{0},t_{1},t_{2})$”.

We then estimate clock offset $\theta$ and round-trip delay $\phi$:

[TABLE]

Note that the computation of $\theta$ assumes that communications latency is symmetric, as is typical in networking. If not, there will be a systematic bias which limits our overall synchronization accuracy. Since individual clock readings and network latency measurements may be noisy (due to e.g., buffering), we collect multiple samples and filter them. To synchronize $N$ devices, each of the $N-1$ client devices exchanges $K$ messages with the leader in a round robin fashion. Following the best practices suggested by NTP, we determine $K$ using the mean and min filters.

IV-A Mean Filter

The mean filter assumes that communications latency is normally distributed. This lets us compute the sample mean and variance after rejecting outliers where the round-trip delay is greater than some threshold (e.g., due to buffering or interference). Since variance decreases as $1/K$, we can determine the number of messages $K$ required to reach a desired accuracy. However, in our experiments, we found that the mean filter converges to a systematic bias of $\approx$0.4\text{,}\mathrm{ms}$$ when compared to an external reference clock. To ameliorate this bias, we use the min filter.

IV-B Min Filter

The min filter assumes that the sample with the minimum latency is the most reliable. This heuristic, also used by the NTP standard, is based on the hypothesis that the message with the shortest delay experienced the least amount of processing and therefore is the most symmetric. We confirm this hypothesis quantitatively and compare the mean and min filters in Sec. VI. In our implementation, we can either exchange a fixed number of messages and take the minimum, or iterate until we observe a sample with latency below some threshold. To guarantee a predictable startup time of less than 10 seconds on our rig of 5 smartphones with a mean latency of $4\text{\,}\mathrm{ms}$, we used a fixed $K=300$ samples.

V Phase-based image stream alignment

Once all device clocks are synchronized, we can assume that all images are timestamped in the leader’s clock domain (by adding the appropriate offset). However, to actually capture images simultaneously on all devices, we need to account for the latency between when software on a device requests a frame and when the sensor is actually exposed. This latency can be highly variable and is hard to measure accurately. Instead, we exploit hardware image streaming: the ability to indefinitely capture a sequence of images at regular intervals. We seek to shift the phases of each stream, so that all cameras are exposing at the same time. Then, all that remains is to select from the image ring buffer which frames to save to disk.

Suppose that the leader camera and a client camera start streaming at times $u_{goal}$ and $u$ respectively and repeatedly capture images with period $T$. We seek to reduce the difference in their phases $\delta=u-u_{goal}\ (\mathrm{mod}\ T)$ to a small value $\epsilon$ by shifting the phase of the client camera (Fig. 4a). As opposed to trying to align $u$ and $u_{goal}$ directly, phase shifting a frame stream can be done efficiently and accurately.

We describe two methods to phase shift a frame stream, reset sampling, a slower to converge method that is applicable to many devices, and frame injection, a faster to converge method that requires device-specific modeling.

V-A Reset Sampling

In reset sampling, we restart the camera until its phase falls within a tolerance $\epsilon$ of the goal phase (the gray region in Fig. 4b). There is significant variance in how long it takes a camera to reset (600\text{\,}\mathrm{-}$$800\text{\,}\mathrm{ms} in our experiments). Furthermore, this distribution can vary from camera to camera and in the worst case can result in phases always falling outside the tolerance. Therefore, we sleep for a random amount of time $U(0,L)$ after stopping the camera and before restarting it to make the overall latency distribution more uniform. If the latency of starting the camera is described by the random variable $S$, then, the total phase offset can be modeled as:

[TABLE]

where the approximation holds as long as $L$ is large relative to $T$ and the range of $S$. We found a value of $L=1$ second to be reasonable. Since we are able to shift the phase by a random amount in each iteration, we effectively sample phase offsets from a uniform distribution and reject those that are outside the gray region in Fig. 4b. There is an $\epsilon/T$ chance a sample will be accepted on any one iteration. This means to have a 95% chance of converging within a tolerance of $\epsilon$, we need to reset the camera $R$ times:

[TABLE]

For $\epsilon=$1\text{,}\mathrm{ms} and $T=$33\text{\,}\mathrm{ms}, $R=98$ iterations are required.

V-B Frame Injection

Some camera APIs (e.g., Android’s Camera2 API [17]) allow a high priority capture request to be injected into a normal priority continuous image stream. Typically, the continuous stream is used for the viewfinder, while the high priority request is used to capture a still image with different exposure or gain settings. By injecting a frame with exposure longer than the duration $T$, we can shift the phase of a stream in one iteration. In an ideal camera, if the phase offset between two image streams is $\delta$, we can align the streams by injecting a frame with exposure time $T+\delta$. Real cameras can behave differently than this ideal, but as long as the camera is deterministic, we can fit a function mapping a desired phase offset $\delta$ to an exposure that will shift the camera’s stream by $kT+\delta$ for some integer $k$. For example, on one specific phone model, we empirically found that exposures of length $T+\delta/2$ would shift the phase by $2T+\delta$. Some cameras require exposure times to be an integer multiple of a scanline’s readout time. While frame injection is deterministic, we model this quantization as Gaussian noise with standard deviation $\sigma$ and measured $\sigma$ to be on the order of tens of microseconds. Under this assumption, this method will converge in $4\sigma^{2}/\epsilon^{2}$ iterations with 95% probability. If $\epsilon=$1\text{,}\mathrm{ms}$$, frame injection succeeds in one iteration.

V-C Error Detection and Recovery

Since all devices are in the same clock domain, we can detect phase misalignment directly from frame timestamps. This lets us detect frame drops or deviations in the stream period and notify the application to realign automatically. One class of errors we did not encounter in practice but would like to handle gracefully in the future is low-frequency clock drift. We can detect clock drift by periodically exchanging NTP messages, and then rerun phase alignment on the subset of devices that have drifted.

VI Results

We quantitatively verify the accuracy of our method and show that it outperforms several common naive approaches by three orders of magnitude. We also demonstrate how our method qualitatively improves the quality of multi-view stereo and panorama stitching. Finally, we show simultaneous multi-view captures of several highly dynamic scenes.

VI-A Measuring Overall Synchronization Error

To quantitatively validate our method, we use an LED panel as an external high-precision frequency reference (Fig. 5a). The panel contains a fast-changing 10$\times$10 array and a slow-changing 10$\times$1 array. In the fast-changing array, a column is lit for a fixed time period $\tau$, then the next column is lit for $\tau$, and so forth until the last column is reached and the pattern repeats. In the slow-changing array, each LED stays on for $10\tau$. The two arrays allow us to measure synchronization error with resolution on the order of $\tau$. Since the entire panel repeats with a period of $100\tau$, it is possible though unlikely for the measured synchronization to be off by exactly $100k\tau$ where $k$ is an integer. We can eliminate this possibility by using multiple values of $\tau$.

In our first experiment, we placed two Google Pixel 3 smartphones on a tripod and captured $100\text{\,}\mathrm{\SIUnitSymbolMicro s}$ exposures of the LED panel with $\tau$ set to $200\text{\,}\mathrm{\SIUnitSymbolMicro s}$ (Fig.  5a). The frame duration $T$ was $33\text{\,}\mathrm{ms}$. To avoid confounding synchronization error with misalignment caused by mismatched rolling shutters, we oriented the phones so that their center scanlines both capture the fifth row of the fast moving array of the LED panel.

We measured the synchronization error of our min filter and frame injection methods by conducting 239 trials. In each trial, we first reset the synchronization process to undo clock sync and added a random length long exposure frame to undo the phase alignment. We then asked our system to synchronize to a tolerance of $\epsilon=$20\text{,}\mathrm{\SIUnitSymbolMicro s}$$. Example captures from the two phones are shown in Fig. 5(b-c). The skewed pattern of LEDs is due to rolling shutter.

In all pairs of images, there is no more than 1 LED difference between the pairs, showing that the total synchronization error is less than $200\text{\,}\mathrm{\SIUnitSymbolMicro s}$. We also ran experiments with $\tau$ set to $10\text{\,}\mathrm{ms}$ to verify that our results were not off by an integer multiple of $20\text{\,}\mathrm{ms}$. To get more fine-grained error measurements, we exploit the fact that short exposures and rolling shutter means some LEDs will only be partially captured. This allows us to estimate the error between the two phones with temporal resolution better than $\tau$. We use the mean pixel intensity in each grid cell in the fifth row of Fig. 5(b-c) to estimate the sub-grid position of the LED lights.

For two devices, we found the maximum error between them to be $121\text{\,}\mathrm{\SIUnitSymbolMicro s}$ (Fig. 6c and Table. I). With more than two devices, any pair can have this maximum error but in opposite directions implying a worst case error of $242\text{\,}\mathrm{\SIUnitSymbolMicro s}$.

We also compared to three naive synchronization methods: wired, Bluetooth, and WiFi. In the naive wired method, we used a selfie stick connected via an audio splitter to two Pixel 3 smartphones, such that a single button press would trigger both phones. We used the same setup as in Fig. 5(a), except we set $\tau$ to $20\text{\,}\mathrm{ms}$. In our naive Bluetooth method, we connected wireless Bluetooth photo capture triggers to each device and used a mechanical button to press both triggers at the same time. In our naive WiFi method, we connected one phone to the other’s WiFi hotspot. On shutter press, the phone with the hotspot then instructed itself and the client to take photos immediately. We conducted 20 trials for each of these methods. In the wired and Bluetooth methods, 2 and 1 shot failed to capture respectively due to issues with the extra hardware. These naive methods have errors over 1000 times our method and occasionally fail to work (Table I). In addition, the wired and Bluetooth solutions require extra hardware making them cumbersome to use and difficult to scale to more devices.

VI-B Measuring Clock and Phase Accuracy

The overall accuracy of our system, $\epsilon_{total}$, depends on many factors. The two main components are due to errors in clock estimation ($\epsilon_{clock}$) and errors in image stream phase alignment ($\epsilon_{phase}$). There are also other sources which we do not account for, such as imperfect hardware timestamps and variances in the readout period $T$. Ignoring error from other sources, we model the total error as:

[TABLE]

In the previous section, we directly measured $\epsilon_{total}$ using an external reference. We can also directly measure $\epsilon_{phase}$ by comparing an image’s hardware timestamps to what was requested. $\epsilon_{clock}$ can then be estimated as $\epsilon_{total}-\epsilon_{phase}$.

We first measure phase error over the same 239 trials as in the previous section. We found that the phase error is always within our requested tolerance of $20\text{\,}\mathrm{\SIUnitSymbolMicro s}$ and that the distribution of differences has zero mean (Fig. 6a). The clock error is not zero mean and has a slight negative bias (Fig. 6b). We analyze this further in the next section.

Note that both the maximum and mean absolute error are comfortably less than $1\text{\,}\mathrm{ms}$. Table II also shows that the dominant component of the error in our system is due to clock error, which, as noted earlier, can be reduced to less than 1 microsecond by using PTP hardware.

VI-C Analysis of Clock Filters

We analyze the behavior and effectiveness of using the mean and min clock filters. With the same setup, the two phones exchange 10,000 NTP messages, and capture one image pair of our LED panel. From this image pair, we can estimate the true clock offset, from which we can derive the one-way latency of each message. Like our other experiments, images are hardware timestamped and messages are software timestamped, both in each device’s local clock domain. Each message takes roughly $3\text{\,}\mathrm{ms}$ round-trip and the entire procedure takes 34.3 seconds. Fig. 7 shows the difference in one-way latency distributions over 10,000 messages after removing 64 outliers where the latency exceeds $10\text{\,}\mathrm{ms}$.

Mean Filter - NTP’s clock offset accuracy relies on network symmetry. Fig. 7 and Table III shows that this is not the case in our setup and that computing the mean offset over many messages converges to a bias that is approximately half the absolute latency difference. The exact values will vary based on network configuration and hardware used. The supplement contains additional details on how we model the relationship between round-trip latency and bias.

Min Filter - The min filter approach uses the message with the smallest round-trip latency. In Table III, NTP bias is approximately half the difference in one-way latencies. The min filter, by choosing the sample with lowest latency, is subject to significantly less latency asymmetry and therefore achieves a much lower bias in its clock offset. We conducted 3 quantitative experiments with sample sizes $K=300,1000,10000$ and found that on average, the shortest round-trip latency to be $1.87\text{\,}\mathrm{ms}$, $1.02\text{\,}\mathrm{ms}$, and $0.996\text{\,}\mathrm{ms}$, respectively with low variance. In practice, we use 300 samples which takes under 1 second per client and still provides $<$1\text{,}\mathrm{ms}$$ total synchronization accuracy.

Errors under $200\text{\,}\mathrm{\SIUnitSymbolMicro s}$ are likely to be confounded by measurement noise such as misaligned scanlines between phones, rolling shutter skew, and our method of using an LED panel running at $200\text{\,}\mathrm{\SIUnitSymbolMicro s}$. Phase error is under $21\text{\,}\mathrm{\SIUnitSymbolMicro s}$, which is minor compared to compared to the total error (Table II).

VI-D Phase Alignment Convergence Time

We measure the number of iterations and total wall time required for phase alignment using both the frame injection and reset sampling methods. Note that once a client has synchronized its clock with the leader, phase alignment can proceed independent of other clients. Phase aligning $N$ devices takes only as long as the slowest device.

For the frame injection method, we used the same dataset of 239 trials as in earlier experiments. Aligning to $\epsilon=$20\text{,}\mathrm{\SIUnitSymbolMicro s}$$ took between 4-7 injected frames, with a mean of 5.5 frames and $\sigma=0.97$ frames. On Pixel 3, each injected frame takes $300\text{\,}\mathrm{ms}$ so phase alignment converges in less than 3 seconds.

To measure reset sampling, we ran 10 additional trials with a tolerance of $\epsilon=$1\text{,}\mathrm{ms}$$. As expected, reset sampling took significantly longer than frame injection, taking on average 28.7 iterations before convergence with a standard deviation of 25.2 iterations. Because reset sampling requires restarting the camera and sleeping for a random amount ([0, 1] seconds), each iteration took on average 1.23 seconds ($\sigma=0.27$ seconds). According to Eq. 3, it will take 98 iterations or 120 seconds to achieve a 95% probability of alignment. Although reset sampling is significantly more expensive than frame injection, it does not require any modeling of camera request latency and is deployable on a wider variety of devices.

VI-E Applications

While our system has many applications, we focus on two here – stereo reconstruction and image compositing of dynamic scenes. In addition, we use our method to capture multiple views of highly dynamic scenes (Fig. 10 and the supplement). Our system has also been used for training a machine learning algorithm shipping with a commercial smartphone [18].

Stereo reconstruction of dynamic scenes - Stereo techniques reconstruct 3D geometry from two or more photos of the same scene from different viewpoints [19]. Assuming known camera poses, these techniques typically first identify a scene point’s projections in multiple photos and then estimates its depth by triangulation. A fundamental assumption is that the scene point is stationary across photos. Therefore, we need synchronized capture to apply these techniques to dynamic scenes.

To qualitatively demonstrate the efficacy of our synchronization for depth estimation of dynamic scenes, we capture a pair of stereo images and compute depth using COLMAP [20], a state-of-art Structure-from-Motion [21] and Multi-View Stereo system [22] (Fig. 8). We compare it to naive synchronization, which we simulate by selecting frames that are $66\text{\,}\mathrm{ms}$ apart, an amount that is comparable to the smallest average error among the different naive synchronization methods listed in Table I. Clearly, improved synchronization results in higher quality depth. Moreover, since our approach scales to multiple cameras, we can further increase the accuracy of depth, especially around the occluded regions, by adding more synchronized cameras (see the supplement).

Image compositing for dynamic scenes - Image composites stitch multiple images of the same scene from different viewpoints into a single image[23, 24] that may otherwise be difficult or impossible to capture using a single camera, e.g., an extremely wide angle image or a multi-perspective image[25, 26]. A typical approach consists of aligning the images, projecting them onto a suitable compositing surface, and blending them to minimize alignment errors and account for differences in exposure, color balance, etc. While a composite may be constructed from images captured by a single camera, a dynamic scene requires synchronized captures from multiple cameras to minimize stitching artifacts.

Fig. 9 demonstrates that our system can be used to avoid stitching artifacts due to motion. Our system can also synchronize exposure and white-balance across devices making it easier to blend between images in the composite.

VII Conclusion and Future Work

We presented an entirely software-based system for capturing time-synchronized image sequences on a collection of smartphones. Our method is wireless, runs on commodity hardware, and is able to achieve an accuracy of better than $250\text{\,}\mathrm{\SIUnitSymbolMicro s}$, which we verified using a precision chronometer to be significantly better than naive methods that synchronize cameras prior to capture.

We empirically confirmed that the accuracy of our method largely depends on NTP clock bias, which can be minimized following the existing best practice of min filtering. We also analyzed and verified the convergence behavior of two phase alignment approaches: reset sampling, which is slow but widely applicable, and frame injection, which converges rapidly but has a minor hardware requirement.

As future work, we would like to scale up our system, i.e., to thousands of different devices distributed over a large geographical area maintaining synchronization over extended periods. Our system has only been tested on Google Pixel 1, 2, and 3 smartphones. The network is also restricted to $11$ devices (a limit imposed by the Android OS) and all clients must currently communicate directly with a designated leader device. A peer-to-peer model can address both these limitations and adapt to both clock drift and motion. We open-source libsoftwaresync in hopes of inspiring new possibilities in social, collective photography applications. We envision such a system being used to capture large-scale dynamic events such as concerts and sports matches.

Acknowledgment

We thank Sam Hasinoff, Jon Barron, Roman Lewkow and Ryan Geiss for their helpful comments and suggestions, as well as Nikhil Karnard, Tim Brooks, Orly Liba, and David Jacobs for their help with collecting qualitative results.