RipViz: Finding Rip Currents by Learning Pathline Behavior

TL;DR

RipViz is a novel method combining flow analysis and machine learning to automatically detect and visualize dangerous rip currents in stationary videos, aiding public safety and awareness.

Contribution

It introduces a hybrid approach using optical flow, pathline analysis, and an LSTM autoencoder to identify rip currents as flow anomalies without user input.

Findings

Successfully detects rip currents in stationary videos.

Provides clear visualization of rip locations.

Automated and easy to interpret for lay audiences.

Abstract

We present a hybrid machine learning and flow analysis feature detection method, RipViz, to extract rip currents from stationary videos. Rip currents are dangerous strong currents that can drag beachgoers out to sea. Most people are either unaware of them or do not know what they look like. In some instances, even trained personnel such as lifeguards have difficulty identifying them. RipViz produces a simple, easy to understand visualization of rip location overlaid on the source video. With RipViz, we first obtain an unsteady 2D vector field from the stationary video using optical flow. Movement at each pixel is analyzed over time. At each seed point, sequences of short pathlines, rather a single long pathline, are traced across the frames of the video to better capture the quasi-periodic flow behavior of wave activity. Because of the motion on the beach, the surf zone, and the surrounding areas, these pathlines may still appear very cluttered and incomprehensible. Furthermore, lay audiences are not familiar with pathlines and may not know how to interpret them. To address this, we treat rip currents as a flow anomaly in an otherwise normal flow. To learn about the normal flow behavior, we train an LSTM autoencoder with pathline sequences from normal ocean, foreground, and background movements. During test time, we use the trained LSTM autoencoder to detect anomalous pathlines (i.e., those in the rip zone). The origination points of such anomalous pathlines, over the course of the video, are then presented as points within the rip zone. RipViz is fully automated and does not require user input. Feedback from domain expert suggests that RipViz has the potential for wider use.

Tables

Table 1. Table 1: Results summary: Notice that RipViz has the highest F 1 subscript 𝐹 1 F_{1} score, and does not rely on user input at run time.

Method	Automated?	F1 Score
Object Detector [11]	yes	0.43
Timelines [30]	no	$0.72$
Filtered color map [30]	yes	0.28
Filtered arrow glyph [30]	yes	0.16
FlowNet[17]	no	0.16
modified FlowNet [single long pathlines]	no	0.19
modified FlowNet [single short pathlines]	no	0.32
RipViz [single long pathlines]	yes	0.08
RipViz [sequence of short pathlines]	yes	0.85

Equations

$$\mathcal{L}=-\frac{1}{N}\sum _{i=1}^N[w_1\cdot p_i\log \widehat{p_i}+w_0\cdot (1-p_i)\log (1-\widehat{p_i})]$$

$$F_1=\frac{2}{\frac{1}{recall}+\frac{1}{precision}}=\frac{2}{\frac{FNs+TPs}{TPs}+\frac{FPs+TPs}{TPs}}$$

Full text

\CCScatlist\CCScat

K.6.1Management of Computing and Information SystemsProject and People ManagementLife Cycle; \CCScatK.7.mThe Computing ProfessionMiscellaneousEthics

\teaser

Rip currents are deadly but remain invisible to many: We propose a feature detection method, RipViz, to make these invisible rip currents visible. RipViz highlights locations of rip currents as a red region. This region is determined by finding seed points that produce pathline sequences deviating from normal ocean flow. Non-experts and experts alike can use RipViz to visualize rip currents.

\vgtcinsertpkg

Introduction

Rip currents are powerful, narrow channels of fast-moving water flowing towards the sea from the nearshore [3, 6, 20, 27]. The speed of seaward rips can be very strong, reaching two meters per second, faster than an Olympic swimmer. They are a dangerous beach hazard that most people do not recognize. As a result, there are thousands of drownings each year due to rip currents globally [10, 25]. The goal of this work is to improve public safety by helping the beachgoers see the rip currents when they are present.

The mechanism for rip currents is an increase in the mean water level, referred to as setup, which occurs when waves break against the shore. This setup can vary along a shoreline depending on the amount of water or height of breaking waves. Rip currents form as water tends to flow from regions of high setup (larger waves) to regions of lower setup (smaller waves), where currents converge to form a seaward flowing rip.

Detecting rip currents with machine learning (ML) is challenging because there are different types of rips, each with a different appearance. The three major factors that lead to different types of rips are the shape of the shoreline, the bathymetry, and hydrodynamic factors (e.g., wave height and direction, tides). The combination of these lead to rip currents with different visual signatures. Rips may also either be transient or persistent in space and time. Detecting rip currents pose unique challenges compared to detecting other objects such as cars, or people, etc. Rips are amorphous without a well defined shape or boundary, and are ephemeral without well defined temporal bounds.

Using an object detector, such as those included in a recent survey [22], would require a substantial training dataset for each type of rip current. To date, there are only two publicly available training datasets for rip detection: individually labeled frames [11] and time averaged images [29], both of the same type of rip current. There are no existing training dataset for other types of rip.

By nature, rips always eventually flow seaward regardless of their visual appearance. Therefore, we propose a novel hybrid approach that incorporates flow analysis with ML for rip detection, rather than relying on image colors. We also propose encoding rips as locations with anomalous flow behavior, transforming the detection task into differentiating normal from anomalous behavior. This greatly simplifies the collection of training data since only a single dataset is needed.

There are two existing approaches to detecting rip currents from stationary videos. The first approach uses the appearance of rip currents to detect them. This includes human experts analyzing time-averaged (Timex) video or running automated object detectors [11, 29, 35, 36] to detect rip currents. The second approach relies on flow analysis, where the flow behavior of ocean waves is analyzed to detect rip currents. Direction-based clustering of flow vectors [34, 30] and timelines [30] placed parallel to the beach are used in this approach. While able to detect weaker rip currents or those where the appearance is not obvious, timelines require user input to specify their initial placement.

In this paper, we introduce RipViz, a fully automated, hybrid of deep learning and flow analysis, feature detection method to find rip currents, as shown in Figure RipViz: Finding Rip Currents by Learning Pathline Behavior. We first obtain the time varying flow field from a stationary video using optical flow.

We use pathline sequences to capture the flow behavior. One could simply seed pathlines at every point and trace each of them for the duration of the video. However, due to the quasi-periodic nature of wave activity, this simple approach produces too much clutter to be of much use as shown in Figure 1. Furthermore, pathlines integrated over the full length of the video accumulate more error, especially in noisy real world datasets. Instead, we generate sequence of shorter pathlines for each seed point. By staggering the initiation of pathlines, our expectation is that pathline sequences outside the rip zone will behave differently. For example, pathlines seeded in the surf zone where the waves are breaking will have large variations in their trajectories. Pathlines seeded further out to sea, on the beach, and sky would be fairly still. Pathlines seeded within the rip zone will have less variations in their trajectories yet will be different from the stationary ones.

In RipViz, we frame detecting rip currents as a flow anomaly detection problem. An LSTM autoencoder with a custom weighted loss function is used to learn the spatiotemporal features of pathline sequences for normal ocean flow (i.e. not rip currents). The trained LSTM autoencoder can predict anomalous pathline sequences (i.e. rip currents) during test time. The origination points of anomalous pathlines are identified and highlighted as a means of visualizing the rip zone. Our target users are general public who are not familiar with rip current dynamics nor visual analytic systems. Hence, our design for the visualization output is to make it as simple and unambiguous as possible.

The main contribution of this paper is:

• •

A hybrid feature detection method that combines machine learning and flow analysis techniques to automatically find and visualize dangerous rip currents.

In order to realize this, the following innovations are necessary:

• •

For flow fields with a quasi-periodic behavior, such as ocean flow, working with a sequence of shorter pathlines is better than a single long pathline.

• •

A weighted binary cross entropy, unlike the unweighted binary cross entropy used in previous works, is more effective when learning from sparsely distributed pathlines generated from ocean flow.

1 Related Work

**Streamline Selection: ** Streamline selection and seed placement are well studied in flow visualization. Sane et al. [39] provide a survey of the body of work over the last two decades. These works share overlapping goals with research on streamline clustering [5], and flow simplification [23]. Each aims to produce an uncluttered presentation of the flow field while still capturing the essential flow features and behaviors. For example, Marchesin et al. [28] proposed dynamically selecting a set of streamlines that leads to intelligible and uncluttered streamline selection. Ma et al. [26] used an importance-driven approach to view-dependent streamline selection that guarantees coherent streamline update when the view changes gradually. Yu et al. [44] proposed hierarchical streamline bundles by producing streamlines near critical points without enforcing dense seeding throughout the volume. They grouped the streamlines to form a hierarchy from which they extracted streamline bundles at different levels of detail. Tao et al.[41] proposed two interrelated channels between candidate streamlines and sample viewpoints. They selected streamlines by taking into account their contribution to all sample viewpoints. However, the methods mentioned above use handcrafted features to define the feature representation of streamlines. Handcrafting features to represent pathlines in complex unsteady flow fields, such as ocean flow fields, is a challenging task. In contrast, the method presented in this paper learns complex feature representations without the need for handcrafted features.

**Deep Learning for Flow Visualization: ** In recent years, the visualization community has worked with ML in two ways: visualization to understand the ML model, and use of ML in visualization tasks. On the latter, particularly for flow visualization tasks, Berenjkoub et al.[2] used U-net, a deep learning neural network, to identify vortex boundaries. Kim and Günther [19] used a neural network to extract a steady reference frame from an unsteady vector field. Han et al. [17] used an autoencoder-based deep learning model, FlowNet, to learn feature representations of streamlines in 3D steady flow fields which are then used to cluster the streamlines. They used longer streamlines that were integrated through the entire extent of the data. Furthermore, they used one streamline per seed point.

The work presented in this paper uses a sequence of pathlines per seed point to learn the flow behavior from noisy unsteady 2D flow fields. This work uses an LSTM autoencoder to detect anomalous pathline sequences from noisy unsteady 2D flow fields.

**Rip Current Detection: ** Traditional rip current detection generally involves in-situ instrumentation such as GPS-equipped drifters and current meters [21, 27]. Remote sensing of rip currents is also possible with aerial imaging of the spread of fluorescein dye in the rip zone captured by drones, and the use of marine radar [16, 21]. These require expensive equipment or a team of observers which makes them impractical for rip current monitoring or detection purposes. However, with simple optical video capture such as from surf webcams, it is possible to detect certain rip currents using a time-space (timestack) display or a simulated long time exposure images (timex) display [18]. Timex images reveal rip locations as darker regions in the image that correspond to deeper channels in the bathymetry where water may flow seaward. In contrast, incoming water associated with the breaking waves in the surf zone appear much brighter in timex images. This type of rip is referred to as bathymetry controlled rips and are characterized by a quiet region in the rip channel that is flanked by breaking waves on either side. Interpreting timex images require some domain expertise. Maryan et al. [29] used a Viola-Jones framework to train their model on timex images, and indicate the location of the rips via bounding boxes. Likewise, Rashid et al. [35, 36] also used timex images but utilized a modified version of the Tiny-Yolo V3 architecture. Similarly, Ellis and McGill [12] used timex images in conjunction with environmental information such as tides, wave height, and period to cluster offshore movements to rip currents. However, their method produced false positives in situations where non-water objects such as surfers, paddle boarders, etc., are also moving offshore. Rather than working with timex images, de Silva et al. [11] trained a Faster R-CNN model with a accumulation buffer to detect bathymetry rips using individual frames of the video. All of these methods detect bathymetry rips based on the appearance of the sea state. However, in instances where appearance is different or weak, these methods fail to detect rip currents.

An alternative approach is to detect rips based on the observed behavior. Philip and Pang [34] obtained an unsteady flow field from the video and hypothesized that the rip current is directly opposite the dominant flow in the vector field due to the incoming wave motion. They grouped vectors based on the direction and magnitude to visualize the rip current. Mori et al.[30] used a similar approach to group vectors and improved the visualization of the rip current by mapping direction to color and magnitude to hue. In the same work, Mori et al.[30] also used timelines to visualize rip currents. They placed timelines parallel to the beach and observed its shape as it gets dragged by the rip current. The work presented in this paper also uses optical video of the sea state to detect rip currents. The underlying approach is also based on the flow behavior, and hence not constrained to detecting bathymetry rips. However, the presented methodology in this paper is the first to propose combining ML and flow analysis to detect different types of rip currents. Because detection is based on treating flow behavior in rip currents as anomalous, the task of collecting and labeling training data for an ML model is unified and simplified. In contrast, a standard ML rip detection model would require a training data set for each type of rip current – a costly and time consuming process.

**Autoencoders: ** Autoencoders are a type of neural network that can learn object representations without any supervision [45, 7]. Autoencoders are trained with a loss function that compares the input and reconstructed output by using a reconstruction error. Long short term memory (LSTM) autoencoders are a specialized type of autoencoder that can learn from sequential data. These types of autoencoders are equipped with LSTM layers that can learn how data is temporally related. Furthermore, autoencoders are also used as anomaly detectors[38]. For anomaly detection, autoencoders are first trained with normal data. When presented with anomalous data, the trained autoencoder will produce a high reconstruction error. We exploit this property to detect anomalous flow behavior.

The flow visualization community has used autoencoders to find feature representations of objects that then can be used for clustering. In their work, Han et al. [17] proposed to learn feature representation of streamlines and stream surfaces of 3D steady flow fields by using an autoencoder with a binary cross-entropy function. They then further reduced the dimensionality of the features and used those to generate clusters. However, their method does not learn how pathlines are temporally related. Additionally, their unweighted binary cross entropy loss function does not account for sparsely distributed pathlines. In our work, we use an LSTM autoencoder with a custom weighted binary cross-entropy loss function, to learn spatiotemporal behavior of sparsely distributed pathline sequences from ocean scenes.

2 RipViz

Identifying rip currents in complex and chaotic ocean flow is challenging. We first reconstructed a 2D unsteady flow field using optical flow from ocean videos. Our method captured the ocean flow characteristics by using short pathline sequences which are then represented as stacks of binary images. We used an LSTM autoencoder to learn spatiotemporal features of these pathline sequences. We trained our model with pathline sequences of normal ocean flow using a custom weighted binary cross-entropy function, suited for learning pathline sequences of ocean flow. We then used this trained LSTM autoencoder to identify pathline sequences of abnormal ocean flow or rip currents. We visualized the rip currents by projecting the seed points of these abnormal pathline sequences back onto the video frame and growing a transparent red region around these points.

2.1 Flow Field Reconstruction

An unsteady 2D flow field is obtained from the stationary video using optical flow. Many optical flow algorithms use the relative motion of neighboring pixels between consecutive frames in the video to calculate the local flow. We use Lucas-Kanade [24] sparse optical flow function in the OpenCV library [14] to trace pathlines. We verified each integration step by comparing the forward and backward integration of the flow field.

2.2 Sequence of Pathlines

Each pathline is represented by a 1D vector, $\textbf{p}=\{x_{1},y_{1},\cdots,x_{n},y_{n}\}$, where $(x_{i},y_{i})$ is a point in the coordinate system of the video frame and $n$ is the length of the pathline. For the LSTM autoencoder to learn about the pathlines, we transformed these pathlines into their own 2D pathline-centric coordinate system. To do this, each pathline is represented by an $L\times L$ binary image I, where each point in p is translated to center the seed point in the binary image. For longer pathlines that extend beyond $L\times L$, we increased the size of the binary image to accommodate these longer pathlines. We then resized all of these larger binary images down to $L\times L$. The reason why all pathlines were not simply centered then resized to $L\times L$ is that we need to differentiate pathlines that barely moved from their initial seed position versus those that actually travelled beyond $L\times L$. Also note that by centering, the 2D representation of the pathlines are now location agnostic. This combination allows us to compare pathlines from different parts of the video to identify those with similar behavior. As noted earlier, tracing pathlines over the entire length of a video of quasi-periodic motion derived from noisy optical flow calculations result in unusable cluttered flow representations. Instead, we reseeded pathlines at the same seed point over regular time intervals to generate pathline series of length $S$. We represented each pathline sequence as a stack of binary images of size $S\times L\times L$ associated with each seed point.

2.3 LSTM Autoencoder

We used an LSTM autoencoder to learn from the pathline sequences. The LSTM autoencoder consisted of two components, an encoder, and a decoder, as shown in Figure 2. The encoder learns the spatiotemporal features of the pathline sequence. The decoder reconstructs the pathline sequence by using the learned spatiotemporal feature representation.

We specify the input layer of the LSTM autoencoder to expect pathline sequences of shape $S\times L\times L$. The first and second layers of the encoder are time-distributed 2D convolutional layers. These layers consist of $128$ and $64$ convolutional filters, respectively. These layers process each pathline of the sequence separately and learn the spatial features of each pathline. The third and fourth layers are 2D convolutional LSTM layers. Each layer consists of $64$ and $32$ convolutional filters, respectively. These convolutional LSTM layers process the sequence of pathlines together and learn the temporal features of the pathline sequences. The first layer of the decoder is a convolutional LSTM layer with $64$ convolutional filters. The second and third layers of the decoder are time-distributed deconvolutional layers with $64$, and $128$ respectively. The last layer of the LSTM autoencoder is a time-distributed convolutional layer with $1$ convolutional filter. Each layer, except for the input and the output layers, is followed by a normalization layer [1]. The input volume has no padding; therefore, we set the stride of all layers to $1$. The autoencoder outputs a sequence of pathlines of shape $S\times L\times L$. For the hidden layers, we use the ReLU activation [31]. For the output layer, we use the sigmoid activation.

In comparison to recent autoencoder based stream line selection methods, our method not only learns the spatial features of each pathline but also learns how each pathline is temporally related to other pathlines in the same pathline sequence. Learning these spatio-temporal features allows our method to better learn about the quasi-periodic nature of the chaotic ocean flow.

2.4 Weighted Loss Function

We trained the LSTM autoencoder by minimizing the difference between each training sample against the corresponding model prediction. The loss function used to compute this difference is a weighted binary cross-entropy loss function. Since each training sample is a pathline sequence represented as a stack of binary images of size $S\times L\times L$, we treat each sample as a binary volume where $p_{i}$ is $1$ when the pathline crosses that voxel and [math] otherwise. $\hat{p_{i}}$ is the corresponding value from the predicted volume. The loss is calculated over all voxels (i.e., $N=S\times L\times L$ ).

[TABLE]

$w_{1}$ is weight assigned to voxels when $p_{i}=1$. $w_{0}$ is weight assigned to voxels when $p_{i}=0$. In contrast, previous autoencoder based neural network architectures used in flow visualization tasks used an unweighted version of the same loss function (i.e., $w_{1}=w_{0}=1$ ). We found that a weighted binary cross-entropy function is better suited for our application domain for reasons discussed in Section 3.2.

2.5 Detecting and Visualizing Rip Currents

During inference/test time, we calculated the reconstruction error for each pathline sequence by using the loss function defined earlier. If the error is larger than threshold $T$, we labeled those pathline sequences as anomalous (i.e., rip currents). Otherwise, we labeled the sequences as belonging to normal flow. We discuss how $T$ was selected in Section 3.2. Once the anomalous pathline sequences were found, the corresponding seed points were connected by using a region growth algorithm to generate a region. Isolated seeds without neighbors are discarded. To visualize the rip zone, We projected this region back onto the frames in the video as shown in Figure RipViz: Finding Rip Currents by Learning Pathline Behavior.

2.6 Network Training

We implemented our neural network in Tensorflow by using the Keras application programming interface (API)[9]. We trained the network using an NVIDIA Tesla A100 graphical processing unit (GPU). In the training process, we initialized parameters in all layers of the neural network using a normal distribution $\mathcal{N}(\mu,\sigma^{2})$, where mean $\mu=0$ and variance $\sigma^{2}=0.01$. We applied the Adam optimizer [38] to update the parameters with a learning rate of $10^{-6}$. We used the minibatch size of $10$ and trained our model with $100$ epochs.

3 Results and Discussion

We describe our dataset, provide an analysis of the components and parameters used in RipViz, and present comparisons with other methods.

3.1 Dataset

Our dataset consists of $55$ stationary videos. Each video is $1.5$ to $4$ minutes long and of size $1920\times 1080$ pixels. We collected the data from Salinas, Marina, and Davenport beaches in California in winter of 2022. We used either a tripod-mounted Canon EOS Rebel T7 DSLR camera or a Samsung Android phone to acquire the data. We chose $4$ videos to generate $16000$ non-rip pathline sequences for training our model by using the seeding strategy discussed in Section 3.2. Our validation dataset consists of 12000 pathlines generated from three videos. The remaining videos were used for testing. Note that our training data does not include examples of each of the many types of rip currents, only examples of normal ocean flow. This greatly simplifies collection of training data. Each video was labeled under the guidance of the rip current expert. Pathlines seeded in the rip current area were labeled as “rip” and others as “non-rip”.

The framing of these videos, which includes the distance of camera to the water and the focal length or zoom factor of the lens, is meant to be representative of surf webcams (e.g. surfline.com) that might be useful for rip current detection. Parameters such as integration length $n$ and binary image size $L$ are based on such framing. Sensitivity of these parameters on different framings are discussed in Section 3.2.

Note that we assume a stable video source and hence did not perform any video stabilization. Wind can cause slight movements of the camera, but also movements of grass, clouds, etc. Also, video processing seldom work with raw video but rather on compressed video. Deriving the optical flow field on compressed video may also introduce some motion artifacts especially in highly compressed regions such as the sky or empty sandy beaches. RipViz handles these type of motion artifacts better than existing methods for our dataset.

3.2 Analysis of RipViz components

The RipViz method contains two important changes from prior methods: the use of a weighted loss function, and the use of a sequences of pathlines rather than a single pathline. Without these changes we found the ML fails to learn ocean flow. In addition, the method has parameters like detection threshold, $T$, and binary image size $L$ which are likely dependent on our specific application domain. In this section we analyze each of these factors, showing that our changes are necessary, and providing the method by which we determined parameters.

Threshold $T$: Threshold $T$ is used to filter anomalous pathlines (i.e, rip currents) based on the reconstruction error as discussed in Section 2.5. In order to find the optimal threshold $T$ we calculated $F_{1}$ score at varying threshold values in a subset of our data. $F_{1}$ score is defined as,

[TABLE]

If a detected pathline falls within the expert annotated boundary of the rip current, then it’s counted as a TPs (true positive). Otherwise, it’s considered an FP (false positive). Suppose a pathline originating within the rip current is not detected, then it’s an FN (false negative). The range of the $F_{1}$ score is between [math] and $1$. If most pathlines fall within the rip current boundary, the score will be closer to $1$, otherwise closer to [math]. We found that threshold, $T=1.0$ produced the highest $F_{1}$ score , and was used as $T$ for the experiments in this paper.

**Weights of the loss function $w_{0}$ and $w_{1}$: ** We reconstructed the flow field from videos using optical flow. As discussed in Section 2.2, all pathlines are represented in a binary image I. We observed that some pathlines are short and do not extend far from the seed point, while other pathlines are longer and travel farther from the seed point, as shown in Figure 3. We needed the model to learn the distribution of these shorter and longer pathlines because it allowed the model to learn the normal flow behavior in an ocean scene. However, this leads to an imbalance in the number of voxels with 1s and 0s, especially for shorter pathlines. For the deep learning model to learn effectively about these small features/pathlines, we needed to increase the weight of the loss function when the target probability ($p_{i}$) is $1$. The loss function, as described in Section 2.4, penalizes the model more for making mistakes when predicting pixels with the target probability $1$ in the training process. Using the weighted binary cross-entropy function made the model better learn the distribution of shorter from longer pathlines. In contrast, in FlowNet [17], the streamlines were well spread out across their input volumes, making the use of a weighted loss function unnecessary.

To find the optimal $w_{0}$ and $w_{1}$ for our application domain, we randomly choose a small subset of pathline sequences, some short and some long, to train models at different combinations of $w_{0}$ and $w_{1}$. We trained each model for $100$ epochs, using the same training conditions specified in Section 2.6. For each trained model, we observed how many voxels were correctly recreated. If the predicted probability for a voxel with target probability of $1$ is greater than $0.7$, we marked that voxel as TP; otherwise, it is considered to be FN. If the model predicted pathline voxels in areas with no pathline, we marked that as FP. We calculated the $F_{1}$ score for each model with different weight combinations using equation 2. We observed that non-uniform weighting is necessary, with the highest $F_{1}$ score generated when $w_{0}=1$ and $w_{1}=40$ as shown in Figure 4. Note that using unweighted cross-entropy, as in previous work, is the equivalent of setting $w_{0}=w_{1}=1$. In this condition we found that the model completely fails to learn ocean flow.

In some instances where the model did not converge, TPs were [math], which resulted in an undefined $F_{1}$ score. In such instances, we followed the default reporting convention of Scikit-learn python library [33] and reported those $F_{1}s$ as $0.00$.

**Pathline Length $n$ and Sequence Length $S$: ** In RipViz, we use pathline sequences to learn flow behavior. In order to find the optimal pathline length, $n$, and sequence length, $S$, for our application domain, we trained multiple models while changing $n$ and $S$, and observed the $F_{1}$ score as shown in Figure 5. We noticed two patterns from this experiment. First, increasing the pathline length resulted in a higher accuracy up to an optimal length; beyond this, the accuracy decreased. Second, using a sequence of pathlines instead of a single pathline per seed point produces more accurate results. We found that $n=900$ and $S=100$ were optimal for our data set.

**Size of binary image $I=L\times L$: ** We represented each pathline in a binary image as discussed in 2.2. We trained several models with our training dataset where binary images were of different sizes as shown in Figure 6. We compared each model by calculating the $F_{1}$ score, similar to how $F_{1}$ score was calculated for tuning Threshold $T$. Our experiments indicated that small binary images ($L\times L=5\times 5$) cannot sufficiently capture the variations of pathline movements, resulting in a lower $F_{1}$ score. We also found that larger binary images ($L\times L=128\times 128$) also resulted in a slightly lower $F_{1}$ score. Furthermore, larger binary images resulted in a high memory usage and a longer training time. We found that $L\times L=64\times 64$ is sufficient at capturing the variations of pathlines resulting in a high $F_{1}$ score. In our experiments we used $L\times L=64\times 64$ as the size of the binary image $I$.

**Seeding strategy: ** Various seeding strategies attempt to optimize different goals such as aesthetics, clutter reduction, capturing flow features, etc [39]. In this paper, the primary consideration for pathline seeding take into account where one can have the highest return in informational value regarding the water movement. Because most of our training data and random beach images that one can find on the web are taken in landscape mode, seeds are distributed uniformly along the horizontal axis. Conversely, the water body is usually in the middle, with the sky on top and the beach on the bottom halves of the framing. We therefore use a Gaussian distribution of seeds along the vertical axis. This is a form of importance sampling [4]. During the training process, we use importance seeding to maximize pathlines that track water movement, and reduce irrelevant information such as the beach or cloud movement in the sky. During the testing process, we use regular sampling since there is no guarantee that the test data will also be in landscape mode.

3.3 Comparison with FlowNet using Single Long and Sequences of Short Pathlines

We compared RipViz with FlowNet [17], a deep learning based streamline clustering method. In FlowNet, $3D$ streamline features were learned by an autoencoder. These 3D streamlines were traced across the full extent of the data. Additionally, they traced one streamline per seed point. The trained autoencoder was used to generate 1D representations of 3D streamlines. The dimensionality of these $1D$ representations is further reduced by t-distributed stochastic neighbor embedding (t-SNE) [43]. The resulting low dimensional vectors are then clustered by using DBSCAN [13]. An interactive user interface was used to find the desired clustering by changing the maximum distance between two feature descriptors ($eps$) and the minimum number of samples in each cluster ($ms$).The autoencoder in FlowNet is trained with a unweighted binary cross-entropy function (i.e., $w_{0}=w_{1}=1$).

In order to compare with FlowNet, we transformed our long $2D$ pathlines to $3D$ by adding a time axis as the $z$ axis [42]. However, since our pathlines are sparsely distributed in the $3D$ binary volume, we found that the FlowNet model with the unweighted binary cross entropy did not learn to differentiate the pathlines. Resulting in a single cluster for all the pathlines. We show the clustering result in Figure 8. Additionally, the training time was relatively long. A proper comparison requires modification of FlowNet to work with our application domain. The modifications are discussed in Appendix A.

We trained the modified FlowNet model with long pathlines, integrated across the entirety of the video, and with single short pathlines ($n=900$). The output of the modified FlowNet is shown in Figure 8. The first column shows a representative frame of each video. We see that without modifications, FlowNet lumps all the pathlines into one cluster. Both the modified FlowNet and RipViz, when fed with single pathlines that run the entirety of the video, also fail to identify the rips. When fed with shorter pathlines ($n=900$), modified FlowNet start to show signs of rip currents but includes significant FPs as well. Only, when sequence of pathlines as used in RipViz do we see the rip currents clearly.

In order to obtain a quantitative comparison between modified FlowNet with shorter pathlines and RipViz, we calculated their $F_{1}$ scores.

If a seed point is flagged as as a rip by either method is within the expert annotated rip current boundary then we mark it as TP, otherwise we mark it as FP. If seed points within the rip are not selected then we mark those points FN. We found that the $F_{1}$ score for modified FlowNet was $0.32$ compared to the $F_{1}$ score of $0.85$ for RipViz as shown in Table 1. We hypothesize the low $F_{1}$ score for the modified FlowNet was due to its high FP rate.

Furthermore, we found tuning the two hyperparamters $eps$ and $ms$ for the clustering step was crucial in finding the appropriate clusters. We noticed having some domain knowledge was helpful for the user when exploring these two hyperparameters to find the ideal clustering. In contrast, RipViz is automated and does not require any user input during run time.

Both modified FlowNet and RipViz are based on autoencoders. However, RipViz learns how each pathline of the same seed point was related in time by learning spatiotemporal features of pathline sequences. This allows RipViz to learn ocean flow behavior more effectively. On the other hand, modified FlowNet does not learn how pathlines are related in time. Therefore, we hypothesize that using pathline sequences is better for learning quasi-periodic behavior, such as in ocean flow.

3.4 Comparison with existing rip detection methods

We compared RipViz with existing rip current detection methods as shown in Figure 9. The first and second columns of Figure 9 show a representative frame of the video and the expert drawn ground truth. The remaining columns show the output of the object detector [11], RipViz, filtered arrow glyphs [30], filtered color maps [30], and timelines [30], respectively.

3.4.1 Comparison with behavior based methods

**Timelines: ** Mori et al. [30] proposed to use timelines to detect and visualize rip currents. They placed timelines parallel to the beach and traced the points on the timeline using optical flow to update its position. They observed the shape of the timeline as it gets dragged by the rip current. Timelines get deformed and extend to the rip channel above its initial position, as shown in Figure 9.

However, the initial placement of the timeline is a major contributing factor for timelines to be successful. All the timelines indicated in Figure 9 are placed optimally. To compare the timeline method with RipViz, we calculated the $F_{1}$ score. If the timeline was able to visualize the rip current in a video, then we counted it as TP, otherwise it is counted as FN. We found that properly placed timelines can visualize rip currents in most cases. However, the user has to specify the good initial placement for each video. In contrast, RipViz is automated, and no user input was needed during run time, as shown in Table 1.

**Direction based clustering: ** Philip and Pang [34] hypothesized that the rip current is directly opposite the dominant flow in the vector field, which corresponds to the incoming wave direction. Places where the vectors were opposing the majority flow are potential rip zones. Areas with sufficiently large clusters of such vectors and large enough magnitude were highlighted as rip zones.

Mori et al. [30] used a similar approach and mapped direction to color and magnitude to hue for better visualization. As shown Filtered Color Map column of Figure 9, the method can detect the location of the rip current. However, this method also highlighted the swash zone, the shallow part of the beach, as evident in examples 1-3. We found that while the rip current was highlighted in most cases, the number of false positive pixels tend to be high. In order to compare with RipViz, we calculated the $F_{1}$ score. As shown in Table 1 filtered color map resulted in a $F_{1}$ score of 0.28, due to its high false positive rate.

Mori et al. [30] also proposed arrow glyphs to visualize rip currents. The output of this method is shown in Arrow Glyph column of Figure 9. We noticed that the arrow glyph method is more susceptible to noise in the flow field compared to other methods and RipViz. This is evident by the arrows projected on the sky and the beach although there is a little movement on those parts of the video. Although arrow glyph method highlighted the rip current in the majority of the videos, there were also a large number of false positive detections. Similarly we calculated the $F_{1}$ score. As indicated in Table 1, filtered arrow glyph had a $F_{1}$ score of 0.16 due is relatively high false positive rate.

3.4.2 Comparison with appearance based methods

**Object Detectors: ** de Silva et al. [11] used object detectors to detect rip currents. They trained a deep learning based object detector, Faster R-CNN [37], with images of rip currents. The images they used had a clear visual signature for bathymetry controlled rip currents with a darker region between breaking waves. Then they used the model to predict the location of rip currents, by overlaying a bounding box on the rip currents on videos. The output of the method is shown in the Object Detector column of Figure 9.

We noticed that the object detector can detect parts of rip currents as shown in examples 2-5 in Figure 9. However, rip currents have different appearances [6]. As evident in example 1, it is possible for an objector detector to miss the actual rip current location. In contrast, RipViz is trained on pathlines that capture the behavior rather than appearance of rip currents. Hence, it was able to detect the rip current in all the examples.

Furthermore, we noticed that bounding boxes only detects the general area of the rip current. Sometimes the bounding boxes can cover parts of the beach or miss parts of the rip especially with long elongated rips.

We also noticed that the information such as the curvature of the rip current cannot be gleaned from bounding boxes alone.

In order to compare with RipViz, we calculated the $F_{1}$ score. We counted how many pixels were covered by the bounding box. Out of those, rip pixels are counted as TP, the remaining pixels were counted as FP. If the object detector does not predict a bounding box, then we counted the pixels within the boundary as FN. As shown in Table 1, the bounding boxes resulted in a $F_{1}$ score of $0.43$ compared to $0.85$ for RipViz. We attribute this lower $F_{1}$ score to the large number of false positives and false negatives generated respectively by the non-rip areas of the bounding box and by part of the rip current not being covered by the bounding box.

Additionally, we tested RipViz on other types of rip currents where currently there are no published object detector available. In Figure 10, we show a few examples of such rip current types. Experts categorize the rip currents in the first and second row as sediment and structural rips. However, as shown in Figure 10 RipViz could detect these rip currents regardless of their appearance. More importantly, no additional training data were needed for sediment and structural rips. Examples 3 and 4 illustrate two rip currents where object detectors failed due to a lack of expected visual features. In these two examples, RipViz detected the rip currents because it uses behavior rather than appearance to detect rip currents.

3.5 Discoveries made by Experts using RipViz

Rip current researchers use their expert knowledge to identify the boundary of rip currents by observing the appearance of visual features such as gaps in breaking waves or sediment plumes. However, sometimes parts of the rip current lack these visual features, making it challenging for the expert to identify the entire extent of the rip current.

In contrast to appearance-based identification, RipViz uses the behavior of rip currents, not appearance, to detect rip currents. Experts can use RipViz as a visualization tool to determine the entire boundary of the rip current when visual features are lacking. In Figure 11, we demonstrate a few use cases where the rip current expert’s original boundary estimate was updated after examining the visualization provided by RipViz. The updated boundaries now include feeder currents near shore and what appears to be a circulating pattern farther offshore in Discovery 1 and Discovery 3, as well as a weaker neighboring rip that merged with the dominant rip in Discovery 2. Incidentally, Discovery 1 and 3 both show circulating rips where the guidance from beach signage to swim parallel to shore may not always work [27]. We offer the complete table of discoveries made on our test data in the supplementary materials.

3.6 User Study

We conducted a user study to better understand how non-experts can use RipViz as a tool to become more aware of rip currents. We grouped $400$ non-experts into two groups; one group was trained only with beach warning signs of rip currents, the other group was trained with RipViz. We then showed each participant a randomly picked video of a rip current and asked them if there was a rip current present. In the group trained with beach signs, $24\%$ failed to recognize the existence of the rip current in the video. In the group trained with RipViz, only $14\%$ were unable to recognize the presence of the rip current in the video.

We performed a second experiment to better understand if the participants could locate the rip current within the video. Both groups were given multiple choices of rip current boundary estimates and were asked to pick the correct one. In the group trained with beach signs, only $34\%$ could choose the correct boundary. In the group trained with RipViz, $78\%$ could select the correct boundary. The non-experts were acquired using Mechanical Turk [32, 8], with basic screening for reliable workers, and paid $0.10-$0.20 per task.

3.7 Sensitivity to video framing

Some of the parameters are dependent on the camera framing which would include distance of camera to the water and focal length or zoom factor of the lens. Alternatively, we can think about the width of the beach that is visible in the frame. We base our framing by examining a number of existing surf webcams (e.g. webcoos.com) and planned webcam installations. We experimented on how sensitive our parameters are to changes in camera framing. In the Figure 12, we show effects of varying the framing while keeping our current set of parameters. Recall that during testing, we trace the same number of seed points that are distributed in a regular grid. Based on the results shown in Figure 12, we believe the parameters are still valid within 20-30% change in camera framing. We found the $F_{1}$ score to be $0.82\pm 0.3$ for all the examples, without much deviation.

4 Summary and Remarks

As Ben Schneiderman succinctly captured in his quote: “The purpose of visualization is insight, not pictures”, our goal is to make apparent what may not be visible to the untrained eyes. The focus of this work is to first find the feature of interest (rip) and then present them in a simple, easy to understand manner by highlighting their location directly on the video. This capabality is encapsulated in RipViz, a hybrid feature detection method that combined ML and flow analysis to extract rip currents from stationary videos. We used shorter pathline sequences to capture the flow behavior in a noisy quasi-periodic flow field. Then we used an LSTM autoencoder to learn the behavior of normal ocean pathline sequences. The trained model allowed us to label pathline sequences belonging to rip currents as anomalous by comparing the reconstruction error. By framing the rip detection problem as an anomalous flow detection problem, the onerous task of finding and labeling training datasets for each type of rip current is also greatly reduced.

The Visualization community has used deep learning methods to learn flow behavior from pathlines. In particular, existing literature uses autoencoders to learn from flow data. The authors use single long pathlines/streamlines per seed point as input to their deep learning models. However, straight-forward learning based on traditional pathlines did not work for our application due to the quasi-periodic flow fields such as those found near the surf zone. Therefore, we adapted and innovated on the existing deep learning methods to use a sequence of pathlines per seed point instead

The main contribution of this paper is:

• •

A hybrid feature detection method that combines machine learning and flow analysis techniques to automatically find and visualize dangerous rip currents.

In order to realize this, the following innovations are necessary:

• •

For flow fields with a quasi-periodic behavior, such as ocean flow, working with a sequence of shorter pathlines is better than a single long pathline.

• •

A weighted binary cross entropy, unlike the unweighted binary cross entropy used in previous works, is more effective when learning from sparsely distributed pathlines generated from ocean scenes.

A key assumption about the stationary videos is that the camera is sufficiently close to the water in order for the optical flow algorithm to pick up measurable velocities. For similar reasons, we assume that the camera is pointed mostly seaward and not parallel to the beach. Pointing the camera down a long stretch of beach will create an optical flow that is not representative especially for points farther away from the camera. Rectifying the frames prior to optical flow calculations may extend the usable range a bit farther but does not justify the extra computational cost. For these reasons, RipViz works best when the camera is close to the water and pointed mostly seaward.

Our training data were videos from mostly uncrowded beaches.

We did study seed points along the path of a jogger. Such seed points were not marked as anomalous. This is likely due to the fact that the subsequent pathlines from the sequence marked the initial seed point as mostly normal. We believe that seed points that are placed on surfers or birds or other objects in the scene will likely be marked as normal as well. We plan to study this further with more testing data.

*Extensions and Future Works: * Some of the contributions listed above are not specific to rip current detection. We plan to investigate how pathline sequences of unsteady flow fields coupled with the weighted cross-entropy loss function can be used to distinguish between laminar and non-laminar flows, and possibly train a model to detect vortices as anomalous behavior.

We surmise that analysis using pathline sequences may also benefit certain classes of flow data aside from those in this paper. In fact, we are exploring that avenue and plan to report on the results in a separate paper. In short, the particular needs of rip current detection led us to develop the approach presented in this paper, and which also provides another potential tool for the visualization community to study certain classes of flow fields.

Appendix A: Modified FlowNet

Han et al. [17] proposed, FlowNet, a deep learning based method to cluster and select streamlines. They first transformed each streamline into a 1D feature vector of length $1024$. Then they used the t-distributed stochastic neighbor embedding (t-SNE) [43] to reduce the dimensionality of the 1D vector from $1024$ to $2$. The resulting vectors are then clustered using DBSCAN[13], a density-based spatial clustering method for applications with noise. The authors provided a user interface to vary the clustering by changing the maximum distance between two feature descriptors ($eps$) and the minimum number of samples in each cluster ($ms$) in the DBSCAN algorithm.

We started with the FlowNet architecture for this work, but modified it according to the requirements of our application. Their primary focus was to cluster streamlines from 3D steady flow fields. Therefore, the authors designed the deep neural network architecture to take in a 3D streamline as input. However, in our application domain, the pathlines are 2D. In order to adapt FlowNet to our application domain, we changed the architecture of FlowNet to take in a 2D pathlines as input. Likewise, we changed the original neural network architecture by replacing 3D convolutional layers and 3D batch normalization layers, respectively, with 2D convolutional layers and 2D batch normalization layers to accommodate 2D pathlines. We also updated the input size and output size of the fully connected layers accordingly. We kept the same number of layers, same activation functions, and the same number of convolutional filters as defined in the FlowNet paper. We refer to the new neural network architecture as modified FlowNet.

We also found that the unweighted binary cross-entropy function used in the FlowNet paper was insufficient to learn the variations of pathlines of the ocean flow for reasons discussed in Section 3.2. Therefore, we used the weighted binary cross-entropy loss function defined in Section 2.4 when training the modified FlowNet. We trained the modified FlowNet using the same approach as discussed in the FlowNet paper[17].

We used the trained modified FlowNet to find 1D representations of the pathlines by extracting the latent feature descriptor from the autoencoder. We reduced the dimensionality of the 1D vector by using t-SNE to a vector with a length of $3$. Then we clustered the resulting vectors using DBSCAN, which requires the user to set the maximum distance between two feature descriptors ($eps$) and the minimum number of samples in each cluster ($ms$) before clustering. The cluster with the largest overlap with the ground truth was selected.

Appendix B: Making the Case for Shorter Pathlines in Noisy Datasets.

We estimated the flow field from videos using optical flow. We assumed a small error/noise associated with the optical flow estimate. Numerical integration, even well designed higher order methods, are subject to accumulation of error. This problem is aggravated when one is working with noisy datasets. For the case of pathline integration, we hypothesized that shorter sequence of pathlines will be more accurate in capturing the flow behavior in noisy flow fields than a single longer pathline.

We used two synthetic datasets to indirectly verify our hypothesis: the 2D unsteady Double Gyre dataset [40] and 2D Unsteady Four Rotating Centers dataset [15].

We added noise at $5\%$, $10\%$, and $20\%$ levels to each vector component of each dataset. We compared the similarity of pathlines originating from the same seed point. We observed that shorter pathlines were more similar to those without any noise. To quantify our observations, we randomly seeded $200$ pathlines in each dataset and traced those pathlines. We calculated mean squared error (MSE) for pathlines at varying lengths as shown in Figure 13. We observed that longer pathlines have higher MSE compared to shorter pathlines. Therefore, we anticipate that shorter pathlines will capture a more accurate representation of the flow behavior in reconstructed noisy flow fields such as ours.

Additionally, we visualized how pathline sequences accumulate errors due to noise over longer integration times as shown in Figure 14. Here, we show two pathline sequences, one seeded in the sky and the other on the beach. We expected these two pathline sequences to be very short and remain closer to the seed point due to the lack of motion around those seed points. However, as we integrate beyond $900$ time steps, we notice that the pathlines seem to travel far from the seed point even though we don’t see any visible motion on the video. We attribute this to the accumulation of noise/error when integrating over a more extended time. The noise can come from optical flow estimation or video compression loss.

Acknowledgements.

This report was prepared in part as a result of work sponsored by the Southeast Coastal Ocean Observing Regional Association (SECOORA) with NOAA financial assistance award number NA20NOS0120220. The statements, findings, conclusions, and recommendations are those of the author(s) and do not necessarily reflect the views of SECOORA or NOAA.