FPCD: An Open Aerial VHR Dataset for Farm Pond Change Detection
Chintan Tundia1, Rajiv Kumar1, Om Damani 1 and G. Sivakumar 1
1**Indian Institute of Technology Bombay, Mumbai, INDIA
{chintan, rajiv, damani}@cse.iitb.ac.in, [email protected]
Abstract
Change detection for aerial imagery involves locating and identifying changes associated with the areas of interest between co-registered bi-temporal or multi-temporal images of a geographical location.
Farm ponds are man-made structures belonging to the category of minor irrigation structures used to collect surface run-off water for future irrigation purposes. Detection of farm ponds from aerial imagery and their evolution over time helps in land surveying to
analyze the agricultural shifts, policy implementation, seasonal effects and climate changes. In this paper, we introduce a publicly available object detection and instance segmentation (OD/IS) dataset for localizing farm ponds from aerial imagery. We also collected and annotated the bi-temporal data over a time-span of 14 years across 17 villages, resulting in a binary change detection dataset called Farm Pond Change Detection Dataset (FPCD).
We have benchmarked and analyzed the performance of various object detection and instance segmentation methods on our OD/IS dataset and the change detection methods over the FPCD dataset. The datasets are publicly accessible at this page:
\urlhttps://huggingface.co/datasets/ctundia/FPCD.
1 INTRODUCTION
Accurate and timely detection of geographical changes on earth’s surface gives extensive information about the various activities and phenomena happening on earth. Change detection task helps in analyzing and understanding co-registered images for change information. A change instance between two images refers to the semantic level differences in the appearance between the two images in association to the regions of interest captured at different points in time. On geographical images, it helps to keep track of the changes to analyze evolution of land geography or land objects and to mitigate hazards at local and global scales. The availability of high-resolution aerial imagery has enabled land use and land cover monitoring to detect objects such as wells, farm ponds, check dams, etc. at instance level.
111Authors a,b made equal contribution
Change detection can be bi-temporal when two points in time are compared or multi-temporal when multiple points in time are compared. When multi-temporal data is captured by satellites, drones or aerial vehicles, they are constrained by spatial, spectral and temporal elements, in addition to atmospheric conditions, resolution, etc. The cloud cover, shadows and seasonal changes become visible in many aerial images that affect the overall appearance of the images resulting in the co-registered images to appear from different domains. Moreover, acquiring and annotating images from satellite images to build change detection datasets is a costly process that involves many underlying tasks. The absence of paired images of the same location captured at different times makes it difficult to obtain a useful change detection dataset. Also when paired bi-temporal images are available, there might not be changes present, or the bi-temporal images are captured in very short intervals.
Farm ponds have become popular as private irrigation sources in developing countries like India over the last two decades. A farm pond is an artificial dug out structure having an inlet and outlet for collecting the surface runoff water flowing from the farm area [Tundia et al., 2020]. They are used to collect and store rainwater so as to provide irrigation to crops during periods of water scarcity. It is one of the many minor irrigation structures [Tundia. et al., 2022] with a cultivable command area of up to 2000 hectares. Farm ponds can be classified into two categories based on the presence or absence of water in it: wet farm ponds and dry farm ponds. Also, farm ponds can be either lined or unlined depending on the use of plastic lining to prevent water seepage into groundwater. Overall, farm ponds can be classified into 4 sub-categories (See Fig 1): lined wet farm pond, unlined wet farm pond, lined dry farm pond and unlined dry farm pond based on their structure and the presence of water.
Though efforts have been put up to promote farm ponds, serious concerns have been raised over their implementation and their usage. The purpose of building farm ponds has long drifted from their original objective of storing rainwater for protective irrigation to being used as storage tanks for pumped-out groundwater exposing the underground water to evaporation losses. In the due time, farm ponds have accelerated the rate of groundwater exploitation to many folds [Prasad et al., 2022].
The objective of our work is to develop change detection models to detect and visualize changes associated with farm ponds and to compute percentage differences in the presence/absence of wet/dry farm ponds. This can help in making policy decisions and in analyzing impacts of farm ponds on aspects like shifting of agricultural practice, policies being implemented, seasonal effects and changes, etc.
1.1 Contributions
Our contributions in this paper are:
-
Farm pond change detection dataset (FPCD): A publicly available dataset for change detection tasks on farm pond categories for different purposes and stakeholders.
2. 2.
A small-scale public dataset for object detection and instance segmentation of four farm pond categories.
The paper is organized as follows: section 1.3 covers the related work,
section 2 covers the details of the proposed dataset,
section 3 covers the experiments, section 4 covers the results and observation, and finally conclusion in section 5.
1.2 Problem Formulation
Generally, change detection tasks involve an input set of multi-temporal images and the corresponding ground truth mask, with most change detection datasets having bi-temporal images mapped to binary mask labels. A general assumption in most change detection tasks is that there is a pixel-to-pixel correspondence between the two images and these correspondences are registered to the same point on a geographical area. Based on the correspondences, each pixel in the change mask can be assigned a label indicating whether there is a change or not. In other words, pixels belonging to the change mask are assigned a change label if the corresponding area of interest has geographical changes with respect to each other and those are not assigned a change label when there are no changes. In a binary change detection setting, the paired input images are T0∈RCxHxW and T1∈RCxHxW with a size of CxHxW, where H and W are the spatial dimensions and C=3 is the input image channel dimensions. The ground truth mask can be represented as pixel based mask label, M∈RCxHxW for the bi-temporal input images of the change detection task.
1.3 Related Work
1.3.1 Datasets
Change detection datasets generally use RGB images [Chen and Shi, 2020] or hyperspectral images [Daudt et al., 2018] with some datasets [Van Etten et al., 2021] having change instances up to 11 million. The image resolution of CD datasets range from a few centimeters [Ji et al., 2019], [Shao et al., 2021], [Tian et al., 2020] to 10 meters [Van Etten et al., 2021], [Daudt et al., 2018]. Some of these datasets have image pairs ranging from a few hundred [Liu et al., 2022] to tens of thousands [Shi et al., 2022] and some have very few input image pairs of very high resolution. While most datasets have fixed image sizes. Most of the CD datasets have modest image sizes except for the exception of a few [Wu et al., 2017], [Ji et al., 2019]. We summarize the different aspects of various change detection datasets along with their details in Table 1.
1.3.2 Change Detection Techniques
Deep learning based change detection methods can be classified into feature-based, patch-based and image based deep learning change detection.
Earlier CNN architectures used siamese with triplet loss [Zhang et al., 2019] and weighted contrastive loss [Zhan et al., 2017] to learn discriminative features between change and no-change images, and used Euclidean distances between the images features to generate the difference images. Recent developments in deep learning have led to the usage of attention mechanisms and feature fusion at various scales improving feature extraction capabilities. We compare and list some of the existing encoder-decoder based models used for change detection below.
Deeplabv3+ [Chen et al., 2018] uses spatial pyramid pooling module to encode multi-scale context at multiple effective fields-of-view and encode-decoder structure to capture sharper object boundaries. It improves upon DeepLabv3 [Chen et al., 2017] by applying depthwise separable convolution to both Atrous Spatial Pyramid Pooling and decoder modules. Pyramid scene parsing network (PSPNet)[Zhao et al., 2017] along with pyramid pooling module applies different-region-based context aggregation to produce global prior representation for pixel-level prediction tasks. Unified Perceptual Parsing Net (UPerNet)[Xiao et al., 2018] is a multi-task framework that can recognize visual concepts from a given image using a training strategy developed to learn from heterogeneous image annotations. Multi-scale Attention Net (MA-Net) [Fan et al., 2020] uses multi-scale feature fusion to improve the segmentation performance by introducing self-attention mechanism to integrate local features with their global dependencies. It uses Position-wise Attention Block (PAB) to model the feature inter-dependencies in spatial dimensions and Multi-scale Fusion Attention Block (MFAB) to capture the channel dependencies between any feature map by multi-scale semantic feature fusion.
LinkNet [Chaurasia and Culurciello, 2017] proposes a novel deep neural network architecture that uses only 11.5 million parameters learning without any significant increase in number of parameters. Pyramid Attention Network (PAN) [Li et al., 2018] uses Feature Pyramid Attention module and Global Attention Upsample module to combine attention mechanism and spatial pyramid to extract precise dense features for pixel labeling. Unet++ [Zhou et al., 2018] improves on Unet [Ronneberger et al., 2015] with new skip pathways and is based on the idea that optimizer can learning easily with semantically similar feature maps by reducing the gap between the feature maps of the encoder and decoder sub-networks. Bitemporal image transformer (BiT) [Hao Chen and Shi, 2021] models contexts within the spatial-temporal domain in a deep feature differencing-based CD framework. The bi-temporal images are encoded as tokens and a transformer encoder models the contexts in the compact token-based space-time. A transformer decoder then refines the original features from the learned context-rich tokens back to the pixel-space.
1.3.3 Object Detection and Instance Segmentation Techniques
Object detection is a computer vision task that localizes and classifies objects of interest in an image. Instance segmentation on the other hand provides a detailed inference for every single pixel in input image.
Since the advent of deep learning[Kumar et al., 2021], object detection has vastly benefited from sophisticated architectures and image representations.
Over the years, deep learning based detectors have evolved into one-stage and two-stage detectors. In one-stage detection, the input image is divided into regions simultaneously with the probabilistic prediction of objects, while in two-stage object detection the object proposals are classified in the second stage from a sparse set of candidate object proposals generated in the first stage. A few of the well-known two-stage detectors include the FasterRCNN [Ren et al., 2017], GridRCNN [Lu et al., 2019], while the YOLOv3 [Redmon and Farhadi, 2018], Generalized Focal Loss [Li et al., 2020], Gradient Harmonized SSD [Li et al., 2019] are one-stage detectors. Detecting objects from aerial imagery is affected by adversities like viewpoint variation, illumination, occlusion, etc. and becomes difficult due to objects being small, sparse and non-uniform in high-resolution aerial images.
2 Farm pond Change Detection (FPCD) Dataset
FPCD is a novel publicly available change detection dataset that focuses on changes associated with irrigation structures from India.
The details of the multi-temporal images including the district, village name and the time-interval over which the images were collected are summarized in Table 5.
2.1 Dataset Collection Technique
The images were collected by following the steps given below:-
-
Area and Timestamp Selection: The ground truth information about the farm pond locations and the list of villages were collected from different sources like news articles, reports and web-pages along with Jalyukt Shivar data[Maharashtra Remote Sensing Application Centre (MRSAC),
2015]. The location of the Farm ponds would span across the different districts of Maharashtra, India. The villages were then rigorously filtered and chosen based on the presence or absence of farm ponds, and bi-temporal pairs were decided by visual inspection of various timestamps.
2. 2.
Grid Formation: For collecting the images of a fixed size for each village, the grids of latitude and longitude at a fixed zoom level of Google Maps were needed. We selected zoom level as 18 which provides resolution of sub-meter level. Grids of size 1024x768 pixels were created by using the village boundaries provided by Indian Village Boundaries Project [Data{Meet}, 2019] in the geo-location dimensions i.e. latitudes and longitudes.
3. 3.
Image Collection: Google Earth Pro desktop software was used for collecting historical imagery. For each grid cell, the map was set such that it would cover the grid cell boundaries with the view. Then the bi-temporal images were saved by setting the timestamp one at a time using the Google Earth Historical Imagery tool.
4. 4.
Object & Instance Annotation: Once we have collected all the image pairs for each village, we annotate the objects individually in each image. Depending on the types of farm ponds, there are four main classes as given in Table 4. We use the annotation tool, LabelMe[Russell et al., 2008] to annotate and label the above mentioned classes. And further we convert them into CoCo[Lin et al., 2014] format required for the object detection and instance segmentation tasks thus forming the Farm pond OD/IS dataset.
5. 5.
Change Mask Generation:
In this step, we take bi-temporal pair of images and corresponding farm pond annotations. Based on the location and farm pond category we generate different types of change masks with change classes as given in Table 2. More details on the type of changes are given in subsection 2.2.
2.2 Dataset Details
A total of 694 images of size 1024 x 768 pixels at zoom level 18 were collected from Google Earth images using the technique described in subsection 2.1. The regions of Maharashtra in India were chosen, since it is largely a groundwater dependent region in western India. The images collected at zoom level 18 are at a very high resolution scale, up to 1 meter. The details of villages collected and their respective timestamps are given in Table 5. Most of the villages have timestamps during the months of Jan-April and the minimum year difference between bi-temporal images is 2 years and the maximum year difference is 9 years, earliest being 2007 and latest being 2021. The FPCD dataset consists of image pairs, change masks and object annotations of farm ponds as polygons set in COCO [Lin et al., 2014] format.
For farm pond change detection, we identify four change classes i.e Farm pond constructed, Farm pond demolished, Farm pond dried and Farm pond wetted.
Let’s consider the bi-temporal pair as T0-T1, T0 being the image captured with the old timestamp and T1 being the image captured with new timestamp image.
We identify a binary change as FP constructed, when there is no farm pond in the T0 image, but a farm pond is observed at the same location in the T1 image. Likewise, we identify a binary change as FP demolished, when a farm pond existed at any location in the T0 image and there is no farm pond at the same location or replaced by different terrain in the T1 image. We identify a binary change as farm pond dried when there existed a wet farm pond in the T0 image and the farm pond at the same location in T1 image becomes dry due to absence of water or presence of low water level. We identify a binary change as farm pond wetted, when there existed a dry farm pond in the T0 image and the farm pond at the same location in T1 image is filled with water or the water reaches the surface level.
Some farmers may pump groundwater and fill the farm ponds instead of using surface-water runoff due to rain leading to water loss by further evaporation. This leads to depletion of groundwater and the practice of farm pond based irrigation becoming unsustainable [Prasad et al., 2022]. Thus grouping farm pond constructed and farm pond demolished class helps in identifying the increase/decrease in farm ponds. This helps stakeholders like researchers and policy makers to monitor the impacts due to increase/decrease in farm ponds like change in agricultural patterns, impact on ground water, etc. We classify this task as Task-1.
In certain dry and semi-arid regions, due to uncertain climatic conditions, yearly and seasonal rainfall is erratic. Agricultural officers and researchers often correlate climatic conditions and ground water levels with such changes to infer further observations eventually leading to necessary interventions needed for agricultural sustainability. Thus we group Farm pond dried and Farm pond wetted into Task-2. We also combined the above grouped changes so as to also provide overall statistics. We call this as Task-3. We further explain experimental details of Task-1, Task-2 and Task-3 in Section 3. The details of change class distribution are given in the Table 3.
3 EXPERIMENTS AND EVALUATION
Tasks like instance segmentation [Lin et al., 2014] and image segmentation [Lin et al., 2014] involves a similar pipeline like that of change detection. We used the change detection framework [Kaiyu Li, 2021] for conducting most of our CD-based experiments. The pytorch based change detection framework supports many models, encoders and deep architectures. In this change detection pipeline, the bi-temporal images are encoded into feature vectors using two encoders. The encoders can be either siamese or non-siamese type. The input images are encoded into encoded feature vectors, which are fused either by concatenating, summing, subtracting or finding the absolute difference of the vectors. The resulting feature vector are decoded as output which is compared to the ground truth mask. We apply simple transforms like flipping, scaling and cropping for augmentations of the dataset images at the global image level. Unlike other tasks, for change detection the same transform has to be applied to the multi-temporal images.
3.1 Evaluation Criteria
We report the precision, recall and F-score values as the metric to compare the performance in different change detection tasks under various encoder backbones settings which is defined as below:
[TABLE]
[TABLE]
[TABLE]
where, TP is the number of true positives, FP is the number of false positives, TN is the number of true negatives and FN is the number of false negatives. We report the bounding box mean average precision (bbox mAP)
and segmentation mean average precision (segm mAP) as the performance metric to compare the various object detection and instance segmentation methods under different backbone settings in the benchmark. Intersection Over Union (IOU) is a measure that evaluates the overlap between ground truth and predicted bounding boxes and helps to determine if a detection is True Positive.
Average Precision (AP) is obtained by interpolating the precision at each recall level r, taking the maximum precision whose recall value is greater than or equal to r+1. Finally the mean of AP of all classes gives mean Average Precision (mAP), the metric being used to compare different detectors.
3.2 Experiments
We benchmarked various object detection (OD) and instance segmentation (IS) methods on the farm pond OD/IS dataset. We use an existing encoder-decoder based change detection framework [Kaiyu Li, 2021] for the binary change detection tasks corresponding to Farm pond construction/ demolition (Task 1) and Farm pond dried/ wetted (Task 2). The encoder belongs to siamese type and the features from each branch are fused by concatenation. A few experiments were conducted to analyze the effect of various components of the change detection pipeline. We combined the masks from Task 1 and Task 2 to understand the impact of performance on multiple change classes. Combining the tasks lead to masks having more change objects than either of the tasks (Refer Fig 2). The train-validation split for each Task has been mentioned in Table 6 . We split the farm ponds dataset for OD/IS task in a 80/20 train/test proportion. We used the MMDetection [Chen et al., 2019] framework for implementing the various methods in the OD/IS benchmark. We used the pretrained backbones of ResNet-50, ResNet-101 and MobileNetv2 primarily trained on Imagenet, COCO and cityscapes for the object detection experiments and Resnet-50, Resnet-101 and Swin Transformer pretrained backbones for instance segmentation experiments.
4 RESULTS AND OBSERVATIONS
The results of change detection on Farm pond construction/demolition (Task-1) taking ResNet-18, ResNet-50 and ResNet-101 as encoders are given in Table 7.
BiT model despite not having best precision or recall values performs the best in terms of F-score with the score of 0.8704.
MANet despite performing highest in precision scores poorly in recall values leading to poor F-score and FPN scores the highest recall value.
FPN has the best recall in Resnet-101 and consistently better than its smaller counterparts, ResNet-18 and ResNet-50.
The results of change detection on Farm pond dried/wetted (Task-2) taking ResNet-18, ResNet-50 and ResNet-101 as encoders are given in Table 8.
We can notice that Unet has the best F-Score of 0.9171 and the highest precision for ResNet-18.
For ResNet-50 encoder, we can observe that FPN has the best F-Score of 0.9125, though DeepLabV3 has best recall and UperNet has best precision values.
DeepLabV3+ has the highest recall value with comparatively smaller precision than the best performing model. If we observe carefully, Task-2 has higher overall scores irrespective of the encoder architecture. This may be due to the presence of farm ponds at the same locations in both temporal images where the change is registered and also due to the fact that the number of positives are much smaller than the number of negative images for the task.
The results of combining change detection tasks (Task-1 and Task-2) as Task-3 taking ResNet-18, ResNet-50 and ResNet-101 as encoders are given in Table 9.
For the ResNet-18 encoder, we can notice that BiT has the best F-score of 0.8741 and the highest recall.
For the ResNet-50 encoder, we can notice that PSPNet has the best precision but with poor recall values affecting the F-score.
We also conducted an ablation study on Task 3 to check if the absence of negative examples in the training set could affect the change detection task performed.
We can see that BiT achieves the best F-Score with ResNet-18 as the encoder, but we can also see that there is a decrease in precision when compared to the best model in the complete dataset, despite better recall values.
The results for F-scores also show a similar trend, suggesting that even with a high ratio, the performance is comparable even in the absence of negative images in the train set.
Similarly, we observe that UNet++ with the best F-Score trails the model trained on the entire dataset by only a few points; this further supports the idea that positive samples aid in learning while negative images may increase the model’s robustness.
In this section, we analyze the performance of object detection and instance segmentation task on the Farm pond OD/IS dataset. The results of comparison of object detection on various existing models are given in Table 11.
For the object detection task, Probabilistic Anchor Assignment (PAA) model achieves the best performance of 0.575 mAP with IoU(0.5:0.95) with the ResNet-50 backbone. We can notice that empirical attention with the attention component (0010) achieves the best mAP under IoU(0.75). Guided Anchoring with FasterRCNN model performs the best in bbox mAP(0.5) using ResNet-50 backbone. For the instance segmentation task, many models have the capability to generate segmentation masks along with bounding boxes results and the COCO metric for both are given in the Table 10. We can notice that CascadeRCNN with ResNet-50 performs the best with 0.577 bbox mAP(0.5:0.95) and 0.694 bbox mAP(0.75) indicating it to be a robust detector. Swin Transformer has the best bbox mAP(0.5) with its own independent backbone. For instance segmentation, Deformable ConvNetsv2 with MRCNN model using ResNet-50 as backbone performs the best on segm mAP metric with IoU(0.5:0.95), while MaskRCNN with ResNet-50 performs the best in segm mAP(0.75) and Swin Transformer performs the best under the segm mAP(0.5) metric.
5 CONCLUSION
The availability of high-resolution images due to advances in remote sensing capabilities has led to the use of deep learning models for change detection.
In this paper, we introduced FPCD,
a publicly available dataset for change detection tasks for minor irrigation structures - farm ponds.
We also introduced a small-scale public dataset for object detection and instance segmentation of four farm pond categories. Future works can address the issues associated with limited data availability and class imbalance issues in the dataset.
