Object tracking in video signals using Compressive Sensing

TL;DR

This paper explores reducing pixel count in video signals via Compressive Sensing to efficiently track moving objects, demonstrating that lower pixel counts still preserve object trajectories.

Contribution

It introduces a method for object tracking in videos with significantly reduced pixel data using Compressive Sensing, maintaining trajectory accuracy.

Findings

Object trajectories are preserved even at 1% pixel sampling.

Reconstructed videos with fewer pixels still allow accurate object tracking.

Trajectory graphs show minimal deviation from original videos.

Abstract

Reducing the number of pixels in video signals while maintaining quality needed for recovering the trace of an object using Compressive Sensing is main subject of this work. Quality of frames, from video that contains moving object, are gradually reduced by keeping different number of pixels in each iteration, going from 45% all the way to 1%. Using algorithm for tracing object, results were satisfactory and showed mere changes in trajectory graphs, obtained from original and reconstructed videos.

Equations

$$b=A\bar{x}$$

$$\min \{\Vert x{\Vert }_0:Ax=b\}$$

$$\min \{\Vert x{\Vert }_1:Ax=b\}$$

$$x=\sum _{j=1}^N{\alpha }_j{\psi }_j$$

Full text

• Compressive sensing; Video signals; Object tracking*

I Introduction

I-A Compressive sensing

Thanks to the sparse property, that portrays considerable number of signals that surround us, the engineers are able to use compressive sensing algorithm and so make much better use of electronics today. In order to understand Compressive Sensing [1]-[15] one must first comprehend well famous theorem dating back to 1949. That’s when Claude Shannon, widely known as ”the father of information theory”, stated: “If a function $f(t)$ contains no frequencies higher than $W$ hertz, it is completely determined by giving its ordinates at a series of points spaced $1/(2W)$ seconds apart”. Even though this is a fundamental principle in the field of information theory the whole premise of CS lies on avoiding it. Shannon’s theorem states that the resolution of an image is proportional to the number of measurements. Doubling the resolution calls for doubling the measurements.

But thanks to the development of CS, the reconstruction of super-resolved images and signals from far fewer measurements than deemed necessary is possible. Not only that it also offers hope for directly acquiring a compressed digital representation of a signal without first sampling that signal.

What makes this theorem so applicable is it’s not only restricted to noiseless signals but ,with few or none alternations to algorithm, it also covers the real-world signals which by default include noise.

The system inspected here is an under-determined system. An under-determined system is a system in which the measurements taken are less than the number of unknown signal values which results in a system with infinite number of solutions. Fortunately, this problem is solvable if the signal $x$ is compressible. In order to sense a sparse objects, by taking as few measurements as possible, the best approach is measuring at randomly selected frequencies.

[TABLE]

$\bar{x}$- discrete signal

b – vector of linear measurements formed by taking inner products of $\bar{x}$ with a set of linearly independent vectors $a_{i}$, $i=1,2,3…m$.

A – has rows $a_{i}^{T}$

Encoding is the process of obtaining b from $\bar{x}$ and decoding is the process of recovering $\bar{x}$

Presuming that b is derived from a highly sparse signal, meaning it has very few non-zero elements. The best decoding approach is too look for the sparsest signal of all those that take part in producing the measurement b

[TABLE]

$\lVert x\rVert_{0}$-the number of non-zeros in $x$.

Due to the complexity of solving this problem by enumeration the ”$l_{0}$ -norm” is replaced by the ”$l_{1}$ -norm”

[TABLE]

If we are given a sparse vector $x$ we can write it down as a weighted linear combination.

[TABLE]

$\psi_{j}$ – column vectors

$\alpha_{j}$– Fourier coefficients

but only $S$ of these $\alpha_{j}$ are going to be zero the rest are not going to be zero. $S<<N$, $N$ is the number of pixels. If the signal $\bar{x}$ is sparse enough and with the right matrix A than $\bar{x}$ can solve both (2) and (3) for $b=A\bar{x}$. This is called recoverability.

Again, here we are putting an emphasis on actual implementation of CS algorithm in video recordings. Dividing frames from the original video and using CS on every frame independently is in most cases time consuming. Some of the neater solutions can be seen in project regarding this subject [1]. Key idea is to exploit the intra-frame correlation. Using modification of the original Kalman Filter so that the motion prediction is incorporated and as such proceeds to the CS step more beneficially.

I-B Object tracking

Object tracking is the act of locating a moving object or multiple moving objects over time and reconstructing the trajectory that the objects took [16]-[23].The simplest known method is the block matching technique. While observing the current frame we take a block of pixels into consideration. The idea is to estimate the position by comparing the blocks within a region containing the targeted object with the reference frame [2].The results would be better if the image region occupied by the target in the current frame is visibly different from the other parts of the frame. Problems that occur during object tracking are caused by several factors such as slow frame rate when the object is moving fast, interference caused by a cluttered background etc.[3].

In our study of movement, we got the position, velocity and acceleration of a ball, from a video file recorded with a fixed camera. The detection of this is based on frame subtraction, a common background is subtracted from each frame and then treated. When program works in 2D, we assume that all the movement, velocity and acceleration is contained in a vertical plane perpendicular to the camera. In 3D, the ball is free moving.

II CS and object tracking

Here CS represents a helping hand for managing videos on which OT will be performed. How low can we go on memory storage and still get a satisfying trace that’s actually useful. Problems that are still rising questions are complex background, brightness changes but one of the most challenging are the ones including sudden and/or short ‘disappearance’ of the actual object, i.e. occlusion occurs, in region where installed cameras have no information the cause usually being the object “hiding” behind another. This requires reconstruction of path by assumptions based on its positions before and after vanishing and also on speed and acceleration information.

The higher the frequency of frames the better results can be expected. Characterization of an object is an extremely important component for any type of object tracking algorithms. In order to examine all this, we used CS on two different types of signals one that could be described as considerably sparser than the other.

III Experimental Results

Experiments were conducted using two different videos of a bouncing ball. The Compressive Sensing algorithm used in this experiment requires the video to be converted to grayscale. Then the video is transformed into image frames. Compressive sensing is done on to each frame. The compressed frames are then pieced together to form a new video. Finally the trace of the ball in the video is reconstructed and plotted in 2D.

Experiment I

The first video is much more simple, meaning, it is far sparser than the other one. Due to the small number of nonzero coefficients, it serves as a good example of compressive sensing. Figure 3 shows few still frames from the original video.

Duration of this video is 2.5 seconds, frame rate is 24.32 frames per second. Unlike the experiment with the other video, which will be discussed later, the background of this video is blank. This is a huge advantage because the background will not cause any interference. As a result of the simplicity of the video, video frames are well reconstructed using 45%, 30%, 20%,10%,5% and even 1% of available pixels. Figure 3 and 3 present few reconstructed frames from 1% and 5% available pixels respectively and figure 4 presents the difference between the trace of the object when the video is not compressed (red dots) and when only 1% (left) and 5% (right) of the pixels were taken (blue dots).

As we can see with 5% we have complete overlapping hence showing results with the higher percentage of conserved pixels is unnecessary.

Experiment II

The second experiment is conducted using a much more complicated video than the previous one. The motion of the ball is more complex, the background is diverse and the object has a shadow (figure 10).

Duration of the video is 2.3667 seconds, size of the video frame is 512x512 and frame rate is 30 frames per second.

In figures 13 to 13 we can see that with the decrease of percentage of pixels retained,the deviation is bigger. With only 45% of retained pixels complete match is achieved while with 1% of retained pixels the object can be tracked with decent precision, but the trace is shifted from the original.

Experiments were done using three different computers with different specifications. Table provided in fig 10 shows the elapsed time of the compressive sensing algorithm for different percentages.

In addition we are presenting PSNR (Peak signal-to-noise ratio) results in figures 15 and 15. As the figures show, we will get a high PSNR when compressive sensing is introduced to the first video. That means that the video will retain its quality even when 1% of pixels are sensed. That is expected when taken the ‘simplicity’ of the first video into consideration. But when it comes to the second experiment the results are quite different. Given the complexity of the second video, when 1% of pixels are used more errors are introduced. The first video has a much higher PSNR than the second video hence the quality of the first video is better. When 45% of pixels are contained, the first video has a PSNR of about 80dB and the second video has a PSNR of about 30dB.

IV Conclusion

In this paper, we preformed compressive sensing and object tracking on two videos . The second video was less sparse than the first one. The first video has less details, a blank background and smoother movement. All that led to the conclusion that even with 1% of pixels retained the video closely resembled the original. And the object tracking was still pretty accurate. Due to the complexity of the second video, the results were not as satisfactory as they were with the first one. Even with 10% of pixels conserved a clear decline in quality can be seen. This results in shifted reconstructed path, of the targeted object, when compared to the original. Nevertheless, if this algorithm is used when rigorous accuracy isn’t the main goal, the results are still adequate.