Can We See Photosynthesis? Magnifying the Tiny Color Changes of Plant Green Leaves Using Eulerian Video Magnification

TL;DR

This paper demonstrates a method to visually detect photosynthesis in green leaves by magnifying tiny color changes using Eulerian video magnification, enabling non-invasive observation with standard cameras.

Contribution

It introduces a novel approach to observe plant biological activity through magnified color changes without specialized instruments.

Findings

Live leaves show detectable color changes unlike plastic leaves.

Magnified color variations correlate with photosynthesis activity.

Method uses standard digital cameras and controlled environment.

Abstract

Plant aliveness is proven through laboratory experiments and special scientific instruments. In this paper, we aim to detect the degree of animation of plants based on the magnification of the small color changes in the plant's green leaves using the Eulerian video magnification. Capturing the video under a controlled environment, e.g., using a tripod and direct current (DC) light sources, reduces camera movements and minimizes light fluctuations; we aim to reduce the external factors as much as possible. The acquired video is then stabilized and a proposed algorithm used to reduce the illumination variations. Lastly, the Euler magnification is utilized to magnify the color changes on the light invariant video. The proposed system does not require any special purpose instruments as it uses a digital camera with a regular frame rate. The results of magnified color changes on both natural and plastic leaves show that the live green leaves have color changes in contrast to the plastic leaves. Hence, we can argue that the color changes of the leaves are due to biological operations, such as photosynthesis. To date, this is possibly the first work that focuses on interpreting visually, some biological operations of plants without any special purpose instruments.

Equations

$$\vec{v}=⟨M(I_{nor{m}_1}),M(I_{nor{m}_2}),...,M(I_{nor{m}_{L-1}})⟩,$$

$$M(I_{nor{m}_i})=\frac{\Vert I_{nor{m}_i}-I_{nor{m}_{i-1}}{\Vert }_1}{(n\times m)},$$

$$S_i=⟨F_i-{\vec{v}}_{(i)}⟩,$$

Full text

1 Introduction

Chlorophyll is a term used for green pigments found in plants. It has an important role in the photosynthesis process. When a molecule of chlorophyll absorbs light, it is promoted from its ground state to the excited state. The molecule in the excited state contains energy that is either:

Passed to another chlorophyll molecule where energy is used in a two-step photosynthesis, photosystem I and photosystem II (i.e. photochemical quenching) 2. 2.

Returned to the ground state by emitting the energy as heat dissipation (i.e.non-photochemical quenching) 3. 3.

Returned to the ground state by emitting a photon (i.e. fluorescence)

Light absorption by photosynthetic pigments is extremely fast. The transition from the electronic ground state to a hyperactive state and the decays of the overactive state to the first singlet excited state occur within femtoseconds ($10^{-13}-10^{-15}$ seconds) rabinowitch1969photosynthesis . To use measurements of chlorophyll fluorescence in analyzing photosynthesis, researchers must distinguish between photochemical quenching and non-photochemical quenching. This is achieved by stopping photochemistry (i.e., photochemical quenching), which allows researchers to measure fluorescence with the sole presence of non-photochemical quenching. To reduce photochemical quenching to negligible levels, a high-intensity, short flash of light is applied to the green leaf. The high-intensity short flash can be produced mostly by laser beams or other means. This is the method of measuring fluorescence in natural or dead leaves. Existing technologies and methodologies use special instruments, such as Fluorimeter, to measure fluorescence cendrero2015dynamic .

Eulerian Video Magnification is a recent technique for magnifying the tiny changes in colors that are not observable by the naked eyesWu:2012:EVM:2185520.2185561 . This technique can be used to magnify the changes in the colors of a plant leaf to visualize the effect of chlorophyll fluorescence to the leaf colors. Because this technique requires no additional instruments and more cheap than the existing methodologies, it may replace the existing techniques and special purpose instruments. Such technique can help researchers and farmers in monitoring the status of plants without the need for bringing a sample to a laboratory and without any cost except a traditional camera.

The main contribution of this work is to visualize, without using special purpose instruments, the tiny changes of colors on the green leaf of the plant. The tiny changes in color observed are the by-product of the excited state process (i.e., photochemical quenching, non-photochemical quenching and fluorescence imitation). Chlorophyll fluorescence imaging is a useful non-destructive method to monitor the health status of plants. It shows how the color changes of live plants differ from the small changes of inanimate objects, see Fig. 1.

The rest of the paper is organized as follows. In Section 2, we present in details the proposed system. The experimental results are presented in Section 3, followed by the conclusion in Section 4.

2 Methodology

The proposed system aims to use a regular digital camera to visualize the changes of the live leaf’s colors that represent the Chlorophyll fluorescence without using any special purpose instruments. In this section, we present two main steps that are used to overcome the external factors, namely stabilizing and reduction of illumination variation. Lastly, we present the color magnification step.

2.1 Stabilizing

The objective of this stage is to eliminate the remaining undesirable shakes or movements of both camera and plant leaves. The corner points of the reference object are extracted using minimum eigenvalue algorithm eigenvalue , then the Kanade-Lucas-Tomasi (KLT) algorithm tomasi1991detection is utilized using forward-backward error thresholding bidirectional . For each frame, the transformation matrix between the corner points of the current frame and the corner points of the previous frame is estimated using M-estimator sample consensus algorithm MSAC . Consequently, we use the inverse of the transformation matrix to reduce the movements of the reference object.

The recorded video is stabilized in two phases. The first phase is to reduce the camera shaking, while the second phase is to eliminate undesirable movements of the Region of Interest (ROI) of the plant. For the first phase of stabilization, a static rigid body at the background is determined by the experimenter. This rigid body is used as a reference object in order to eliminate camera vibrations. For the second phase, in order to generate a video of an immobilized green leaf, the experimenter re-applies the stabilization process to the (ROI) of the green leaf that is determined by the user as well.

2.2 Reduction of Illumination Variations

We minimize the changes of the illumination to emphasize that color changes are due to changes in the plant leaf color and not caused by any other reason. The background is separated by removing the leaves from the input video. We use the traditional chroma keying technique to exclude green pixels $Pg$ and generate a video without green pixels $V{-P_{g}}$. Then, we generate an approximate illumination invariant video (normalized video). Given an input video, we generate a video $V{-P_{g}}$ without green pixels. For each $n\times m$ frame in $V{-P_{g}}$, we construct a 1D vector (1$\times(n\times m$)) $I$ that represents the 1D representation of the frame. A normalized 1D vector $I_{norm}$ is generated using homomorphic filtering-based normalization technique HOMO .

For each color frame $I_{norm_{i}}$, we calculate a vector $\vec{v_{i}}$ that represents the changes of lighting with time as shown in the following equations

[TABLE]

[TABLE]

where $M(I_{norm_{i}})$ is a kernel function that calculates the mean of the first order derivative of $I_{norm_{i}}$ w.r.t. time, $i$ is the number of the current frame in the video, and $L$ represents the number of frames in the given video. We generate a signal $S$ that represents the normalized video, i.e., an estimated lighting invariant version, using the following equation

[TABLE]

where $F_{i}$ is the frame number $i$ of the original video.

Although we attempt to reduce both lighting changes and camera movements in a reasonable manner, high illumination variations, e.g. outdoor lighting, and camera movements lead to possibly inaccurate results. Thus, we suggest conducting the experiments under controlled environment (i.e., indoor lighting conditions and a semi-static or static camera).

2.3 Color Magnification

To magnify the changes of the (ROI) colors, we use Euler magnification technique Wu:2012:EVM:2185520.2185561 . The given video is decomposed into different spatial frequency bands using a spatial pyramid. Temporal filtering is applied to all bands to extract the frequency band of interest. The magnified video is reconstructed using the amplified signals of those frequencies which are added back to the original signal. The selected band of temporal frequencies varies based on the case. According to the extremely high rate of the fluorescence, we use large values of the amplification factor $\alpha$ and the spatial wavelength $\lambda$ to magnify the changes of colors as much as possible. In addition, we cut off the amplification factor of the blue channel to reduce the generated noise; where the blue channel is the color channel that is most susceptible to the noise multiresolution .

3 Experimental Results

We have applied the proposed system to HD video sequences at 30 frames per second (fps). We have used $\alpha=120$ and $\lambda=1000$ for magnifying the color changes. The results show that the pixel intensities of the (ROI) for the live plant oscillate in a manner worth noting, compared to an inanimate object.

Fig. 1 shows two case studies. The upper part of the figure represents a live green leaf that was captured under outdoor lighting conditions. The middle part shows the 3D visualization of the color vibrations of the (ROI). The left part represents the frequency of color changes with time. The lower part of the figure shows the result of magnifying the color changes of a plastic green leaf, a live plant green leaf, and a segment of the background. As shown, there are small changes in the both plastic green leaf color and the background. However, the live green leaf has noticeable color changes as shown in the 3D visualization and frequency of the magnified changes of colors. Fig. 2-a enlarges the mean of color changes of a segment of the background, a plastic green leaf, and a live green leaf that were used in another experiment under controlled indoor lighting conditions.

Proof of Concept: We have recorded a video for a live green leaf, and a plastic green leaf under the same indoor lighting conditions. We study the frequency cross-correlation of the following:

A patch of the live green leaf and the background 2. 2.

A patch of the plastic green leaf and the background 3. 3.

A patch of the live and plastic leaves

As shown in Fig. 2-b, the cross-correlation between the live and plastic leaves has the maximum amplitude because both of them have similar green color. This similarity of colors has its effect on reflecting the incoming light to be captured in the video. Nevertheless, the cross-correlation between the live green leaf and the background is smaller than the cross-correlation between the plastic green leaf and the background. As shown in Figure 3, another experiment was performed using a dead green leaf (boiled at 100 Degree Celsius for 10 minutes) and a live green leaf. As illustrated in the figure, the cross-correlation between the color changes on the live green leaf and a segment of the background is smaller than the cross-correlation between the color changes on the dead green leaf and the patch of the background. This indicates that the live green leaf undergoes noticeable tiny color changes compared to the plastic green leaf. As aforementioned, that tiny changes in color is the by-product of the excited state process (i.e., photochemical quenching, non-photochemical quenching and fluorescence imitation) that happen due to some green leaf physiological activities (i.e., photosynthesis) that makes the color of its leaves oscillate as shown in Fig. 1.

3.1 Comparisons

In order to get a quantitative comparison between the color changes of the live and dead leaves, we have recorded 32 videos for eight leaves in two cases: alive and dead, under both DC and AC lighting 111The videos are available in the following link: https://goo.gl/ezEPzn. Then, we have applied the proposed system to get 2,500 samples of the magnified colors of the ROI for each leaf (total number of samples are 40,000 samples per lighting condition). Each sample is represented by the first order derivative of the magnified color w.r.t. time of a single pixel in the ROI. Finally, we have trained a radial basis function SVM classifier using these samples to get an approximated discrimination level of the color changes of the live and dead leaves. In the end, we have reported the mean accuracy of the 5-fold cross validation. The classification accuracy is 64.28% and 62.57% using videos illuminated by a DC light source and a regular light source (AC), respectively. Consequently, we can say that the color changes of the live leaves have something different, compared with dead leaves.

4 Conclusion

In this work, we have presented a system to visualize the subtle color changes of a live plant green leaf, which is a by-product of the photosynthesis process, without using any special purpose instruments. Firstly, the acquired video is stabilized to eliminate, as much as possible, undesirable movements of both plant leaf samples and camera. The illumination variations are reduced to emphasize the color changes that are produced only by the plant leaves. The lighting invariant version of the stabilized video is used to magnify the subtle changes of colors in the region of interest using Eulerian video magnification technique. The mean of color changes of the live plant leaf differs from that of inanimate objects. We have demonstrated that by using a regular digital camera, the aliveness of green leaves could be detected by studying the frequencies of color changes which is due to biological operations (i.e., photosynthesis).