Mass Estimation from Images using Deep Neural Network and Sparse Ground Truth

TL;DR

This paper introduces a semi-supervised deep learning approach that accurately estimates mass from image sequences with sparse ground truth, outperforming traditional volumetric methods in agricultural settings.

Contribution

It presents a novel semi-supervised deep neural network architecture with physics-informed penalties for mass estimation from images, handling complex data and limited ground truth.

Findings

Accurately predicts mass from images using sparse ground truth data.

Surpasses older volumetric-based mass prediction methods.

Demonstrates faster learning and improved stability with physics-informed architecture.

Abstract

Supervised learning is the workhorse for regression and classification tasks, but the standard approach presumes ground truth for every measurement. In real world applications, limitations due to expense or general in-feasibility due to the specific application are common. In the context of agriculture applications, yield monitoring is one such example where simple-physics based measurements such as volume or force-impact have been used to quantify mass flow, which incur error due to sensor calibration. By utilizing semi-supervised deep learning with gradient aggregation and a sequence of images, in this work we can accurately estimate a physical quantity (mass) with complex data structures and sparse ground truth. Using a vision system capturing images of a sugarcane elevator and running bamboo under controlled testing as a surrogate material to harvesting sugarcane, mass is accurately predicted from images by training a DNN using only final load weights. The DNN succeeds in capturing the complex density physics of random stacking of slender rods internally as part of the mass prediction model, and surpasses older volumetric-based methods for mass prediction. Furthermore, by incorporating knowledge about the system physics through the DNN architecture and penalty terms, improvements in prediction accuracy and stability, as well as faster learning are obtained. It is shown that the classic nonlinear regression optimization can be reformulated with an aggregation term with some independence assumptions to achieve this feat. Since the number of images for any given run are too large to fit on typical GPU vRAM, an implementation is shown that compensates for the limited memory but still achieve fast training times. The same approach presented herein could be applied to other applications like yield monitoring on grain combines or other harvesters using vision or other instrumentation.

Tables

Table 1. Table 1 : Summary of developed architectures and their sizes

Architecture	Number of parameters	Number of layers
Res-16E	959489	16
Res-9E	154113	9
Res-9ER	45921	9

Table 2. Table 2 : Summary of architectures accuracies

	Test-Set Accuracy
Architecture	W/O Temporal Smoothness	W/ Temporal Smoothness
Res-16E	94.8%	95.3%
Res-9E	94.6%	94.9%
Res-9ER	94.5% ($\pm$0.18)	95.5% ($\pm$0.07)

Equations

$$L(y;\hat{y})=\sum_{i=1}^{k}\frac{1}{n}\bigg{(}y_{i}-\hat{y}_{i}\bigg{)}^{2}$$

$$Mass=f(max(V-\beta ,0);\theta )\times max(V-\beta ,0)\times v_{elev}\times t$$

$$L_{i}(x,y;w)=\frac{1}{n_{i}}\big{\{}y_{i}-\sum_{j=1}^{n_{i}}(f(x_{ij};w)\times v_{ij}\times t)\big{\}}^{2}+\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}{\frac{\lambda}{n_{i}}\sum_{j=1}^{n_{i}}\big{\{}f(x_{ij};w)-f(x_{i{(j-1)}};w)\big{\}}^{2}}\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\vspace{-2mm}$$

$$\frac{\partial{L_{i}}}{\partial{w}}\leftarrow-\frac{2}{n_{i}}\big{[}y_{i}-\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\sum_{j=1}^{n_{i}}\hat{y}_{ij}\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\big{]}\times\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\sum_{j=1}^{n_{i}}\frac{\partial{\hat{y}_{ij}}}{\partial{w}}\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}+\frac{2\lambda}{n_{i}}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\sum_{j=1}^{n_{i}}{\bigg{\{}\big{[}\breve{y}_{ij}-\breve{y}_{i(j-1)}\big{]}\times\big{[}\frac{\partial{\breve{y}_{ij}}}{\partial{w}}-\frac{\partial{\breve{y}_{i(j-1)}}}{\partial{w}}\big{]}\bigg{\}}}\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}$$

Full text

1 Introduction

Advances over the past couple of years in machine learning architectures and training methods, along with supporting open source frameworks such as Tensorflow, has made leveraging complex high-dimension datasets more accessible to scientists and engineers in various domains than ever before. Along with this, various methods in the domain of machine learning have been developed to accommodate the variability, structure, and complexity of different research problems. Supervised learning has been most widely used but this method requires vast amount of labeled training data. The process of labeling data can be expensive, difficult, and time consuming when dealing with very large datasets. In many real world scenarios it is common not to have ground truth for every data point in the dataset, which poses a challenge to supervised learning. Unsupervised learning methods aiming to learn from data without labels have shown potential on simplistic data sets but have their drawbacks. In the presence of sufficient labeled data, unsupervised learning cannot achieve the accuracy levels of supervised learning, since there is no guarantee that the patterns the algorithm finds correlate directly with the ground truth or characteristics of interest lison2015introduction ; marsland2011machine . The work presented herein represents a middle-ground between the two aforementioned approaches, employing a version of semi-supervised learning by utilizing sparsely labeled data (i.e. data where not every measurement is labeled) for training algorithms. Semi-supervised learning has been leveraged in many applications such as pedestrian countingtan2011semi , weed mapping in sunflower cropsperez2015semi , monocular depth map prediction kuznietsov2017semi , and object detectors from videosmisra2015watch . We leverage an agriculture application to prove the utility of this method by showing that predicting the mass from images of a complex material flowing on a sugarcane elevator, an otherwise difficult scenario to obtain ground truth for each image, is both rendered feasible and can replace older methods which use 3d point cloud information to estimate volume and subsequently convert to mass by (sometimes frequent) calibration while obtaining better performance. Some existing work to measure mass flow in similar context employs the use of LiDAR sensors jadhav2014volumetric , and spectral analysisshin2012mass . Accurate prediction of mass flow by images in this context enables precision agriculture through optimizing machine productivity and efficiency, and mitigating risk (e.g. by avoiding overfilling trucks) while making a more accessible low-cost solution since it only requires 2d camera and inference via a relatively light-weight neural network. The image in Figure 1 is from the stereo camera mounted on the elevator and used in this work for volume estimation, and mass estimation by deep learning directly on the image. Mass estimation is of great importance to several industries and can be quite challenging obtain robust and accurate predictions depending on the material and application. It can be a critical parameter to industries like gas and oil and can be an optimizing factor for industries like mineral processing vayrynen2013mass .The presented methods are by no means limited to image-based applications or agriculture industry and are readily transferable to other mass flow sensing applications from grain harvesters to mining, can work with other data sources besides images, and can be extended to any application where it is possible and desirable to utilize sparse ground truth under some basic assumptions like those presented here.

In this work, we propose an approach to learn complex physics relationships between the bulk density, quantity, and location of material in images to accurately estimate mass, with only sparse ground truth. Develop a regularization term that enforces temporal smoothness to achieve more stable prediction signals and better accuracy on data-point level. Propose a training procedure that handles gradient computation in batch mode, since the number of images for any given run is too large to fit in vRAM on a typical GPU. Develop and implement a lightweight DNN architecture that can run efficiently on low performance and memory bound devices. The proposed mass flow estimation method is shown to outperform previous methods that estimate volume and hinge on a calibrated density to transform the volume estimate to a mass estimate, while being less sensitive to light and offering a significantly more cost-effective solution.

2 Background

2.1 Modified Nonlinear Regression Loss for Sparse Ground Truth

Self-training, mixture models, co-training, multiview learning, and 3S-vector machines are common semi-supervised learning techniqueszhu2009introduction . Generally, these techniques mostly rely on input data when correlating with the expected response. In our proposed method we focus on the response itself, i.e. to take advantage of measurements with sparse ground truth to train a predictive model, a tractable relationship must be asserted and hold between the measurements and labels. As long as that relationship can be formed into a loss function that can then be optimized, a model can be trained. Reinforcement learning with a Markov process assumption is one such example of sparse rewards and a tractable relationship where methods such as policy gradientssutton2000policy can be employed. In a simpler case like this one, measurements can all be treated independently, and when predictions from these measurements are aggregated with a summation to produce a total (accumulated) mass, a loss function for the predictive model can be realized. Thus, the typical nonlinear regression formulation using mean squared error (MSE) loss can be slightly modified to incorporate an additional aggregation term over the predictions for each run in $k$ runs as shown below in Equation 1 .The only difference being that the predictions of image "$x_{j}$" over a given run are summed and then compared to the ground truth and scaled by length "n" of the run. This type of simple modification to nonlinear regression opens up a major opportunity to bypass more costly or infeasible situations to obtain labels or ground truth in order to train a predictive model. The application presented in this work is one such situation where it is much easier to measure an accumulated mass as opposed to trying to obtain an individual mass for each measurement, which would be highly infeasible in any practical sense.

[TABLE]

2.2 Data Summary

To support development of a mass flow sensor based on stereo-camera volume measurements in previous unpublished work of our own, a design of experiment was conducted that took into account various factors that were projected to impact the response of the measurements and simulate environmental variables of operating conditions in a commercial application. These included factors such as variable illumination, mass flows, and elevator speeds. Six lighting levels were used that span from <0.7k lux up to $\sim$ 6.7k lux as measured by a light meter located on the elevator in front of the camera. Other factors are summarized by histograms of the average values per run as shown in Figure 2 . Although some variation exists in the total mass of material (bamboo) used in each test run, this factor was held more consistent to focus on observing the impact of changing the other factors. A total of 239 runs were split into 60/20/20 - train/validation/test sets, which includes 8 empty runs (zero mass) running for $\sim$ 60 seconds with the elevator moving. The data from these tests is utilized for the 2d vision-based solution and compared to the results from volumetric-based mass prediction.

2.3 Mass Flow by Volume

The baseline comparison for our implementation is a volume estimate obtained from a stereo camera at 7.5Hz. Previous work in sugar beet yield monitoring has used similar concepts with different instruments to obtain a volume measurement from which mass flow can be estimated konstantinovic2007soil . The stereo camera produced a point cloud referenced to the plane of the elevator. Volume was calculated from this by binning the point cloud into squares in the $<x,y>$ plane and taking the median per bin to remain insensitive to outliers. Bins with a low number of points or standard deviation above a specific threshold were deemed unreliable and excluded from the volume estimation. Note that the calculated volume units from the stereo camera were scaled for convenience, and so the units used in this analysis are not labeled with physical units (nor do they matter for this application since it must be calibrated anyways). Likewise, the density is the total mass of the material divided by the total volume as calculated from the point cloud, and is shown as a unit-less number in the plots shown in Figure  3 . It was found during testing that, so long as lighting was kept at a certain minimum level (in this case > 2.7k Lux), significant nonlinearities with the point cloud estimation were avoided. These nonlinearities in low lighting manifested as increased number of points with low elevation (or “z”) values in each bin which dragged down the volume estimates. Since the shutter speed had to be fast due to the quick movement of the slats (and therefore material), increasing the exposure time was not an option. However as long as certain minimum amount of lighting is provided, a decreasing density still persists but is not as extreme. This remaining decreasing density trend is hypothesized due to a packing effect where the bulk density of the material decreases as the total volume of material increases. Work investigating the densities of disordered packing of slender cylinders supports this liu2018evolutions ; zhang2006experimental . Drawing inspiration from the trends in the above figures showing density decreasing with higher volumes, an algorithm was devised and fit that explicitly uses the stereo volume to predict the density, and then use it as a multiplier on the volume as is shown in Equation 2 . This has some similarities in concept to an attention mechanism used in NLP semantic analysisbahdanau2014neural , but in this case the weighted multiplier (density) for a given volume estimate is unbounded.

[TABLE]

Where "f" is a 2-layer neural network parameterized by “$\theta$” that outputs a prediction of density based on the volume (V), scaled by elevator speed ($v_{elev}$) and capture time ($t$), with tanh activation and 3-hidden units. The $\beta$-parameter is also fit to account for any positive bias present since the stereo volume calculation was designed towards not missing any volume. This was also a meaningful formulation since if the $\beta$ parameter turned out to be negative then it would indicate a volume estimation in need of refinement, since it would be compensating for volume not detected. This was fit using the same aggregation technique as described in Equation  1 using a 2nd order optimizer

3 Deep Learning with Sparse Ground Truth

3.1 Learning Mass from Images

Learning mass from images is possible in this problem context due to the constraints imposed on the system and the ability to learn complex patterns via end-to-end training with deep learning. More specifically, the camera has a fixed sampling frequency that is fast enough to measure all the material passing by, it is mounted at a fixed distance from the elevator, and the only additional factor needed is slat velocity to scale the accumulated mass. Intuitively, the velocity is scaling the mass to produce a mass flow, since if the mass was sitting still it would not be accumulated. Alternatively it can be thought of as a way to account for frame overlap. Even though complex nonlinearities exist with the density and the material location in the image (e.g. the same material further away is smaller in the image), these patterns can be learned by optimizing a DNN due to fixed locations between the camera and the elevator. Understanding this reasoning about the system is crucial to designing and training a successful model in this and similar applications, since (e.g.) naively assuming counting pixels with material will correlate well with mass would lead to wasted efforts considering only a single factor such as image deformation. The value of using images of volumetric methods will potentially be enhanced further when observing harvested materials of sugarcane or grains, since in the case of sugarcane the constituents such as billets and leaves (and therefore density) vary considerably more . The constituents and their sizes are observable in images though and therefore patterns more than likely learnable by the same sparse ground truth methods. In grains the sizes of the kernels can change with moisture, among other things, and may exhibit other visual signs that correlate with density that an image-based algorithm could identify and learn to account for.

3.2 Temporal Smoothing

Utilizing prior knowledge about the problem allows more custom formulation of the model, loss function, and training procedure to improve accuracy and stability. In this application, it is clear that there is temporal correlation and images near in time should have more similarity in mass than images further away in time. To account for this during training, a regularizing term was added to the cost function with an associated hyper-parameter that allows penalizing a 1st-order lagged difference in the predicted mass values. Equation  3 shows full loss function for a runi that includes prediction error and the additional temporal smoothing regularization term in red.

[TABLE]

Prediction is corrected by the elevator speed $v$ and capture time $t$ (constant $7.5Hz$) to account of frame overlap and the loss is normalized by the number of images “n” in each run to equally weight the gradient update from each run. The penalty strength is controlled by a hyper-parameter $\lambda$ (chosen empirically $0.05$). To see this, note that the gradient of the loss function has a sum of gradients in it, which if left un-normalized will amount to larger gradient updates for longer runs (runs with more images) even though the runs mostly contain the same total mass.

3.3 Gradient Update

To perform a gradient update, we need to compare our prediction, $\hat{y}_{i}$ with ground truth $y_{i}$, but our prediction and regularization terms contain sums, which means the gradient of the loss in Equation  3 does as well, as can be seen by Equation  4 . Since the number of images is too large to fit on a single GPU (a problem for very sparse ground truth), we keep a running sum of the gradients, predictions, and regularizing term from each batch to greatly reduce memory requirements. The full gradient can then be calculated and applied at the end of the run using the accumulated terms from the gradient along with the ground truth, run length, and regularizing hyper-parameter. Equation  4 describes gradient of the full form of the loss equation associated with one run or run “i”. Terms in red are accumulated from each batch of a run and the full gradient calculated and applied when the end of a run is reached during the training loop.

[TABLE]

Given the large and variable size of runs, we compute gradients and predictions in batches and accumulate them over the course of their respective runs as shown in the Pseudo-code in algorithm 1 .

3.4 Model Architecture and Training Procedure

We consider complexity and size as essential factors in the developed architecture, where eventual application goes on embedded hardware at mass scale and saving every bit of computation to a minimum is highly desirable to save costs. To assess the complexity of our dataset we adopt a residual like stylehe2016identity and follow a systematic approach in DNN design, starting shallow then reproducing accordingly. A 9-layer DNN with ELU units and residual connections, defined as – “RES9-ER” shown in Figure  4, was found to have good predictive accuracy as well as fast training and inference. Similar models with ReLU activation and 16 layers were also investigated before converging on this final architecture. ELU activation were considered in our DNN because it showed better noise dampening and more stable signal as well as helped converge faster than ReLUs as shown in Figure  5(a) . The 16-layer network – “RES-16E”, barely out-performed the 9-layer network – “ RES-9E”. For this reason, we investigated the

learned features in every layer of Res-9E and observed redundant features, hence we reduced the number of filters in Res-9E to introduce reduced Res-9, the best performing architecture defined as – “RES-9ER”. Our observation of the redundant features is empirical and validated using this work resemblejs .Figure  5(b) shows a selected feature 64 against a set of other similar features 1, 14, 62, whereas regions of different pixel information are highlighted in pink in the similarity map.Table 1 shows the sizes of RES-9E, RES-16, and RES-9ER.

4 Results & Analysis

4.1 Visualization Explanation of DNN Functioning

To get an intuition on what the network actually learns and what parts of the image contribute the most to the outcome, we follow a generalized gradient-based CNN visualization approach to visualize the learned mass in images. We adapt the grad-cam method proposed by this work gradcam2017 and develop an implementation that is compatible with eager mode in Tensorflow eager . Shown in Figure  6 , three different images and their corresponding grad-cam heat-maps. As shown in Figure  6(a), the network learns when there is not bamboo material present in the image as well as the network learns when there is various amounts of bamboo in the image as show by the grad-cam in Figure  6(b) 6(c).

4.2 Temporal Smoothness Effect

Training with temporal smoothness for the different developed architectures increased the overall accuracy by 0.6% on average. Table 2 lists the accuracies obtained for RES-9E, RES-16E, and RES-9ER with and without temporal smoothness. Improvements in the signal shape are observed as pointed out in Figure 7(c) when compared to Figure 7(b), especially the circled regions in red.

Thus far, even without temporal smoothing the signal had very good stability which is much better than the volume-based signal. Figure 7 shows the predict signal under different scenarios of a run (RunX) trained with RES-9ER with a ground truth of 261.3KG. Using temporal smoothness resulted in a more smoothed signal and improved accuracy by 6.6% and 2.8% compared to the volume-based and ELU only signals respectively.

4.3 Comparison to Volume-Based Predictions

We compare the mass predicted signal using Res-9ER against volume-based mass predicted signal using stereo camera in the following scenarios: 1) Incremental and decremental variable flow - Figure 8(a), 2) Intermittent flow - Figure 8(b), and 3) Poor lighting conditions - Figure 8(c). Further, Figure 8 shows overlays of predicted signals for a number of runs that exhibit the stated characteristics. The DNN-based prediction outperforms the volume-based prediction in each of the scenarios which shows the robustness of the algorithm when posed to variable environmental conditions as opposed to the volume-based signal. We would like to note that the run selected for the poor lighting conditions is one of the runs identified to have too low lighting to use volume estimates on.

4.4 Error Distribution and Outliers

Studying error distribution of the dataset helped us understand where the DNN algorithm made poor predictions as well as how to improve generalization. No extreme outliers were found in the validation or test sets, but the training set contained a single outlying run circled in red in Figure 9(b). The total mass prediction of the outlying run was too low ($\sim 40\%$ of what it should have been), but yet the DNN mass prediction per image before scaling with elevator speeds and capture time was very high, which was also confirmed by a grad-cam analysis as seen in Figure 9(a), where even the sidewalls are being incorrectly incorporated into the prediction of mass. Therefore, we investigated the speed signal and it was found that the outlying run had a very low elevator speed (average of 0.1 m/s) with large variations in speed. It is likely that the optimization process tried to use the reflections of bamboo on the side of elevator to compensate for the low total predicted mass (which was actually a problem of low fidelity from the speed sensor at low elevator speeds). Since this outlier was an extreme corner condition of very low elevator speed that is not at all likely in real applications, thus it could be removed and the network retrained to likely achieve better overall results. Similarly, a corrective action could entail using a speed sensor with higher resolution and fidelity over a broad speed range. Figure 9(b) shows histograms of error distribution from the DNN and volume based predictions across the entire dataset. The DNN-based method can be seen to perform much better even though we included a large outlier due to elevator speed abnormalities, and also gave the volume estimates an unfair advantage by excluding low-light runs for which it faired very poorly.

4.5 Conclusion

In this work we proposed a semi-supervised algorithm that makes inference on mass flow of material from images by training a DNN with sparse ground truth, and showed improvements over older and more expensive methods that must first acquire a volumetric estimate of the material and then calibrate the density. Future opportunities are considerable. The same methods can be readily applied in other applications where sparse ground truth with some basic assumptions on aggregating the values is possible. A natural extension of this work is to predict mass flow of field harvested sugarcane and grain crops as measured on machine, which experiences many other complicating factors affecting density and visual appearance of the material. Likewise, applying unsupervised learning methods to train disentangled representations of features seems highly feasible and a logical next step.