Autonomous Sweet Pepper Harvesting for Protected Cropping Systems

TL;DR

This paper introduces Harvey, a robotic system capable of autonomously harvesting sweet peppers in protected cropping environments, combining vision algorithms and a novel end-effector to improve success rates.

Contribution

The paper presents a new robotic harvester with integrated vision and end-effector design tailored for sweet pepper harvesting in protected cropping systems.

Findings

46% success rate for unmodified crop

58% success rate for modified crop

90% detachment rate for favorable cultivar

Abstract

In this letter, we present a new robotic harvester (Harvey) that can autonomously harvest sweet pepper in protected cropping environments. Our approach combines effective vision algorithms with a novel end-effector design to enable successful harvesting of sweet peppers. Initial field trials in protected cropping environments, with two cultivar, demonstrate the efficacy of this approach achieving a 46% success rate for unmodified crop, and 58% for modified crop. Furthermore, for the more favourable cultivar we were also able to detach 90% of sweet peppers, indicating that improvements in the grasping success rate would result in greatly improved harvesting performance.

Tables

Table 1. TABLE I : Scanning Parameters

Trial	Parameter	Value
1	Boustrophedon (Width, Height, Segments)	0.35 m, 0.4 m, 3
	Scan row offset	0.25 m
	Scanning speed	0.1 m/s
2	Diamond (Radius)	0.4 m
	Scan row offset	0.3 m
	Scanning speed	0.1 m/s

Table 2. TABLE II : Parameters used for ranking grasp poses.

Parameter	Value
Grasp utility weights ($W_1$, $W_2$, $W_3$)	0.2, 0.5, 0.3
Surface Normal Patch Size	0.025 m

Table 3. TABLE III : Harvesting Result

	Trial 1 - Cultivar: Claire	Trial 2 - Cultivar: Redjet
Detach Rate	92% ($22/24$)	42% ($11/26$)
Attach Rate	58% ($14/24$)	81% ($21/26$)
Attach & Detach	58% ($14/24$)	42% ($11/26$)

Table 4. TABLE IV : Sweet Pepper Individual Harvesting Results

	Trial 1 - Cultivar: Claire				Trial 2 - Cultivar: Redjet
#	No. Attempts	Attach	Detach	Notes	No. Attempts	Attach	Detach	Notes
1	1	S	S	MD	6	S	S
2	2	S	S		2	S	F
3	1	S	S		3	S	S
4	4	F	S	OC	2	S	F
5	4	F	S	IS	2	S	S
6	4	F	F	OB, IS	3	F	F
7	3	F	S	MD	1	S	F	OB
8	3	S	S	OC	6	S	F	OB
9	1	F	S	OB, IS	4	S	S	XD
10	2	F	S		2	S	F
11	1	F	S	MD, OB	2	F	F
12	1	S	S		1	F	F
13	1	S	S		1	S	S
14	1	S	S	OC	2	S	S
15	1	S	S		2	F	F
16	1	S	S		1	S	F
17	2	S	S	MD	2	S	S	XD
18	2	S	S		4	S	S
19	2	F	F	XD, IS	2	S	F	XD
20	2	F	F	OC	1	S	S	OB
21	1	S	S	OB	3	S	S	OB
22	2	F	F	MD, OC, OB	1	S	F
23	4	S	S	MD	1	S	S	OB
24	2	S	S		1	S	F	XD
25	N/A				1	F	F	OB
26	N/A				1	S	F

Equations

$$U_i=\sum _{j=1}^3{W}_j{S}_{ij},\text{ given }\sum _{j=1}^3{W}_j=1$$

Full text

I INTRODUCTION

The horticulture industry remains heavily reliant on manual labour, and as such is highly affected by labour costs. In Australia, harvesting labour costs in 2013-14 accounted for 20% to 30% of total production costs [1]. These costs along with other pressures such as scarcity of skilled labour and volatility in production due to uncertain weather events is putting profit margins for farm enterprises under tremendous pressure.

Robotic harvesting offers an attractive potential solution to reducing labour costs while enabling more regular and selective harvesting, optimising crop quality, scheduling and therefore profit. These potential benefits have spurred research in the use of agricultural robots for harvesting horticultural crops over the past three decades [2]. Autonomous harvesting is a particularly challenging task that requires integrating multiple subsystems such as crop detection, motion planning, and dexterous manipulation. Further perception challenges also present themselves, such as changing lighting conditions, variability in crop and occlusions. A recent survey of 50 projects in robotic harvesting of horticulture crops [3] highlighted that over the past 30 years of research, the performance of automated harvesting has not improved substantially despite huge advances in sensors, computers, and machine intelligence. If robotic-crop harvesting is to become a reality, we believe there are three key challenges that must be addressed:

Detection: determining the presence/location of each crop 2. 2.

Grasp selection: determining the 3D pose and shape of each crop and selecting appropriate grasping and/or cutting points. 3. 3.

Manipulation: detaching the crop from the plant without harming the crop or plant.

In this paper, we present an autonomous sweet pepper harvester that works towards addressing these three key challenges. We demonstrate a simple and effective vision-based algorithm for crop detection, a 3D localisation and grasp selection method, and a novel end-effector design for harvesting. To reduce complexity of motion planning and to minimise occlusions we focus on picking sweet peppers in a protected cropping environment (see Figure 1) where plants are grown on planar trellis structures. Experimental results with minor modifications to the plant (removal of some leaves) demonstrated a harvesting success rate of 58%, a grasping success rate of 81% and a detachment success rate of 90% for favourable cultivar. We believe these results represent a significant improvement on the previous state-of-the-art and show encouraging progress towards the possibility of a commercially viable autonomous sweet pepper harvester.

The remainder of the paper is structured as follows. A review of current state-of-the-art methods for autonomous harvesting of horticultural crops is presented in Section II. The design of our autonomous harvesting platform is presented in Section III and methods for perception and planning are presented in Section IV. Finally, results of two field experiments are presented in Section V-B, before a discussion of the key challenges and future work in Section VI.

II Literature

Autonomous harvesting has been demonstrated on a number of horticultural crops such as sweet peppers [4, 5], cucumbers [6], citrus fruits [7], strawberries [8] and apples [9, 10]. Autonomous sweet pepper harvesting has previously been demonstrated by the Clever Robots for Crops (CROPS) project [4, 5]. The CROPS platform achieved a harvesting success rate of 6% for unmodified crop and 33% when occluding leaves and crop clusters were removed. Their work highlights the difficulty and complexity of the harvesting problem. Several key research challenges remain before widespread commercial adoption can occur [3]. These challenges can be broken into three categories: perception, motion planning and hardware design.

II-A Perception

Crop perception pipelines generally include detection, segmentation and 3D localisation stages.

Detection and segmentation are essential in knowing the approximate location of the crop within an image and approaches have been developed for both automated harvesting [11, 12, 13] and yield estimation [14].

After detection and segmentation, the 3D location (position, orientation and shape) of the crop is determined using a 3D sensor such as a ToF camera [4, 15], stereo vision [6, 8] or even a single-point laser range finder [16]. This 3D localisation step allows a grasp and/or cutting pose to be determined. Methods in the literature for 3D localisation are generally ad-hoc and crop specific.

II-B Motion Planning

Common methods of motion planning for autonomous crop harvesting include open loop planning [4, 17] and visual servoing [16, 8]. Open loop planning methods sequentially sense, plan and act, leaving such systems susceptible to unperceived environment changes [18]. Visual servoing methods require a high update rate, but are particularly useful for motion planning within dense vegetation where crop localisation from a single viewpoint can perform poorly due to occlusions [18].

Reliable grasping in a dense and cluttered environment remains an active research topic [19], often requiring tactile sensing to discern between rigid and deformable objects. As advocated in [3], simplifying the workspace or developing harvesting tools which simplify the harvesting operation can greatly improve the success of motion planning in cluttered horticultural environments.

II-C Harvesting Tools and Manipulators

A range of manipulator configurations have been used for autonomous harvesting projects including 3DOF Cartesian, anthropomorphic arms and 6DOF manipulators [3]. Several works have compared various joint configurations to optimise target-reachability in cluttered environments [20, 21].

A key component of any autonomous harvesting system is the harvesting end-effector that grasps and/or cuts the crop. Suction cups are a common gripping mechanism [4, 5, 8, 22] that are mechanically simple and only require access to a single exposed face of the crop.

Another gripping alternative are contact-based grippers [10, 23] which generally employ mechanical fingers that close around the crop. Contact-based grippers can grip the crop very securely, but are more prone to interference from nearby objects such as branches and other fruit. Some crops such as sweet pepper and cucumber must also be cut from the plant and so require an additional detachment tool such as a thermal cutter [6] or scissor-like mechanism [4, 8, 16]. End effectors customised to a particular crop are common, for example in 2014 Hemming et al. [4] developed a custom harvesting tool which simultaneously envelops and cuts-free sweet peppers with a hinged jaw mechanism. This mechanism was found to be more effective than a scissor mechanism, however size and geometry constraints restricted access to some sweet peppers [24].

III System Design

This section outlines the system design for our autonomous sweet pepper harvester. The overall procedure for harvesting sweet peppers is shown in Figure 2 and can be broken down into five key steps:

Scanning: The robot arm is moved in a pre-determined scanning motion to build up a 3D model using an eye-in-hand RGB-D camera. 2. 2.

Crop Detection: Sweet peppers from the 3D scene are segmented using colour information and localised by fitting a 3D parametric model to the segmented points. 3. 3.

Grasp Selection: Candidate grasp poses are computed using the segmented sweet pepper point cloud. 4. 4.

Crop Attachment: Suction cup grips the sweet pepper. 5. 5.

Crop Detachment: Oscillating blade cuts the sweet pepper from the plant.

The first three steps (scanning, crop detection, and grasp selection) form part of the perception system described in Section IV. The procedure for the crop attachment and crop detachment step, are described in Section III-C.

III-A Protective Cropping Systems

Sweet peppers are grown in both fields and protected cropping environments. In this work, we focus on picking sweet pepper’s in protected cropping environments. In these environments, sweet pepper’s are grown on a two dimensional planar surface, significantly reducing occlusion and the need for complex collision avoidance and motion planning around the crop structure. Also, many protective cropping environments have a translucent outer surface that diffuses incoming sunlight; creating favourable conditions for computer vision with relatively even lighting.

The protective cropping environment used in this work has crop grown in planar rows up to 2 m tall with rows spaced approximately 1 m apart. This layout informed the workspace requirement of our sweet-pepper harvesting platform design.

III-B Platform Design

The harvesting robot “Harvey”, is shown in Figure 1. The custom differential drive platform was designed to work independently within a protected cropping environment and manoeuvre between crop rows for up to 8 hours powered by an internal 3kWh lead-acid battery. The platform has a 6DoF revolute arm (Universal Robotics UR5) mounted on a prismatic lift joint (Thomson LM80). The differential drive mobile base houses the batteries, drive motors, gearboxes, computer hardware, robot controller and forward facing laser scanner for mobile navigation and obstacle detection.

III-C Harvesting Tool Design

Our custom harvesting tool shown in Figure 3 is able to grip sweet peppers with a suction cup, and then cut them free from the plant using an oscillating blade.

Natural variation in crop size, shape and orientation make choosing a single end-effector pose to simultaneously grasp and cut each sweet pepper challenging and unreliable. To overcome this difficulty, a key feature of our harvesting tool design is a passive decoupling mechanism that allows the gripping and cutting operations to occur sequentially, at independently chosen locations.

The decoupling mechanism is a flexible strip that tethers the suction cup to the body of the end effector. The suction cup is also magnetically attached to the underside of the cutting blade, allowing the robot arm to guide the suction cup during the attachment phase. After attachment, the cutting blade is lifted to decouple the suction cup from the cutting blade. The suction cup is then only attached to the end effector via the flexible tether, allowing the cutting blade to move independently of the suction cup through the cutting operation. After detachment, the sweet pepper falls from the plant and hangs freely from the flexible tether. The suction cup and cutting blade can be magnetically re-coupled ready for the next harvesting cycle using gravity by simply pointing the harvesting tool downwards. The sweet pepper is released into a collection crate by releasing the vacuum.

This simple and passive decoupling method requires no additional actuators and allows for a greater harvesting success rate.

The harvesting tool also contains an RGB-D camera (Intel®Realsense SR300 RGB-D) sensor for perceiving the crop and a micro-switch for checking whether the suction cup is coupled with the cutting blade. The body of the end effector contains a modified oscillating multi-tool for cutting stems. A pressure sensor on the vacuum line is able to detect sucessfull attachment of the suction cup.

III-D Software Design

The software design uses the Robot Operating System (ROS) to communicate between independent processes (nodes). Figure 4 illustrates the connection between software components.

The ROS MoveIt! [25] library was used for motion planning, with the TRAC-IK [26] Inverse Kinematics (IK) solver for improved solution rate and solution time compared to standard IK solvers. Trajectory execution was performed by the Universal Robots ROS controller package.

The Scene Registration node registers point clouds from the robot’s RGB-D sensor into a smoothed colour point cloud using the Point Cloud Library (PCL) [27] implementation of the Kinect Fusion algorithm. This node does not use joint state information from the robot controller which avoids the need for accurate time synchronisation between RGB-D frames and joint states.

The Detection and Segmentation node detects sweet peppers within a scanned scene. The node returns a list of point cloud segments and is implemented as a ROS service client.

The State Machine node implements the harvesting logic and coordinates the operation of other nodes to perform the harvesting operation. Figure 5 outlines the logic implemented in this state machine for a single harvesting cycle.

Following segmentation of the scene into individual sweet peppers, two concurrent state machines are spawned. The first state machine (below the dotted line in Figure 5) continuously processes segmented sweet peppers to calculate candidate grasping and cutting poses. The second state machine (above the dotted line in Figure 5) waits for a processed sweet pepper, then plans and executes the harvesting operation. This concurrent processing and harvesting of sweet peppers speeds up the harvesting process when a scene contains multiple detected sweet peppers.

Failure to attach the suction is detected via an in-line pressure sensor on the vacuum system and triggers a re-attachment at the next-best grasp candidate. After a fixed number of failed attachment attempts the state machine moves on to harvesting the next sweet pepper. Future work will also look into methods of sensing successful peduncle cutting to enable the state machine to attempt multiple cutting poses.

IV Perception and Planning

In this section we present our perception system for detecting, segmenting, and estimating the pose of sweet peppers, followed by the planning steps to perform the picking action.

This firstly involves moving the RGB-D camera in a scanning motion, combining these views into a single 3D model, and segmenting sweet peppers using colour information as described in Section  IV-A. Grasp poses are then calculated by one of two methods described in Section IV-B. Grasp poses are then used by the subsequent motion planning system (Section IV-C) to perform the harvesting operation.

An overview of the perception system is given below where a more in depth description of the perception system can be found in our previous work [15].

IV-A Scanning and Sweet Pepper Detection

The initial stage of the perception process involves scanning the crop row with the end-effector mounted RGB-D camera. The point clouds captured from the RGB-D sensor during the scanning motion are used as input to the point cloud registration method that creates a single 3D model of the scene using the Kinect Fusion algorithm. The RGB-D camera operates at approximately 30 frames per second with a depth resolution of 2 mm and range between 0.2 m to 1.5 m.

Combining multiple views in a single 3D model helps to reduce point cloud noise, and reduces the effect of occlusion by leaves since sweet peppers will generally be seen clearly from at least one view during the scanning process.

Two types of scanning trajectories have been used, a diamond and boustrophedon (lawnmower) pattern with a fixed offset between the camera and crop row. Figure 6 shows the trajectory (blue line) and pose (red arrows) of the RGB-D camera on the end effector recorded while scanning a sweet pepper as part of the experiment presented in Section V. The figure also shows the resulting registered point cloud generated from the Kinect Fusion algorithm produced during the scan. Scanning is currently performed perpendicular to the crop row, as this was simple to define and sufficient to build a detailed 3D model. This scanning method is inefficient in time and future work could look at selecting the next best view to maximise 3D information about a target crop.

The second stage of the perception system segments the sweet pepper from the background (leaves and stems). This task is challenging due to variation in crop colour and illumination as well as high levels of occlusion. Prior work by McCool et al. [13] proposed a highly accurate sweet pepper segmentation approach capable of segmenting both green and red sweet pepper. In this work, we are interested in harvesting red sweet pepper only and so use a simpler and faster colour-based segmentation approach outlined in [15] which applies a Naïve Bayes classifier to points in the fused point cloud in a rotated-HSV colour space. Parameters of the HSV model were determined by computing the mean and variance on a sample of sweet peppers in the field.

Once segmented, a Euclidean clustering step is used to group the remaining points into multiple distinct sweet peppers. Clustering is based on a minimum distance threshold, selected as 2 mm corresponding to the resolution of the depth camera. A limitation of this method is when two sweet peppers are in contact with each other. If multiple sweet peppers are in view this clustering step also determines the best candidate sweet pepper based on the cluster which has the largest number of detected 3D points (most information available) and has the closest centroid to the end effector. Lastly, smoothing and outlier removal is performed on the points to filter noise from the detection step.

IV-B Grasp Selection

Grasp poses for each sweet pepper are calculated using the the segmented 3D point cloud of a sweet pepper. An ideal grasp pose will place the suction cup squarely on a planar region of a sweet pepper. In this work we developed two different methods for selecting both grasp and cutting poses.

The first grasp selection method is outlined in [15] and fits a geometric model (superellipsoid) to the point cloud to estimate the sweet peppers size, position and orientation. The grasp pose is calculated to be in the centre of the front face of the sweet pepper, while the cutting pose is calculated to be offset from the top face of the sweet pepper.

The second method takes a different approach and selects multiple grasp poses directly from the point cloud data by calculating surface normals with a fixed patch size. These surface normals can be used directly as grasp poses however, we rotate these poses along their axis to keep the end effector upright, minimising large jumps in the wrist configuration.

Candidate poses are ranked based on a utility function that is the weighted average of three normalised scores $S_{i1},S_{i2}\text{ and }S_{i3}$ based on the surface curvature, distance to the point cloud boundary and angle with respect to the horizontal world axis, respectively, where $i$ is the current candidate pose. This utility function favours grasp poses that are close to the centre of the sweet pepper, on planar surfaces, aligned with the horizontal world axis and away from discontinuities caused by occlusion. The utility, $U_{i}$ of the grasp pose $i$ is calculated according to

[TABLE]

where $0\leq U_{i}\leq 1$, $0\leq S_{ij}\leq 1$ is the score of the grasp pose $i$ and $W_{j}$ are weighting coefficients that describe the importance of each score.

An advantage of this method is it finds multiple grasp poses, in contrast to a single grasp pose when using the model fitting method.

Cutting poses for this second method are estimated in a similar way to grasp poses, however with a modified utility function that favours vertical rather than horizontal normal vectors. This tends to find normal vectors on the top surface of the sweet pepper in a ring around the peduncle. We reject poses with low utility to remove outliers, translate them vertically to avoid damaging the sweet pepper and create the final cutting pose using the medians of the x, y and z-axes independently. The orientation of the final cutting pose is set so as to keep the end effector level and perpendicular to the plane of the crop row.

Both our strategies for calculating the cutting pose (parametric model fitting and surface normal estimation) assumes the sweet pepper is vertical with a peduncle near the top. We are currently investigating methods of directly detecting the peduncle location in the colour point cloud to accommodate highly angled sweet peppers.

An example of the grasp selection method applied to real sweet pepper point clouds is shown in Figure 7 where the utility of the grasping and cutting poses (surface normals) are represented as the gradient from red or green to black, where black has the lowest utility.

IV-C Motion Planning

Harvesting trajectories are calculated relative to the grasping and cutting poses described in the previous section. The attachment trajectory starts at a fixed offset back from the grasping pose and moves the suction cup along the selected approach axis. This motion, as computed by the planner, can be seen as the yellow line within Figure 6.

Once the attachment trajectory has been executed to attach the suction cup, the end effector is moved vertically from the attachment pose in order to decouple the suction cup from the cutting tool (as described in Section III-C).

Finally, a cutting trajectory is computed. This cutting trajectory is calculated so as to keep the end effector aligned with the horizontal world frame, as this was found to outperform trajectories aligned with the estimated orientation of the sweet pepper.

The resulting end effector trajectory for the attachment, separation and detachment stages for one harvesting cycle is shown in Figure 6.

As discussed in Section III-A, operating in a protected cropping environment results in a relatively planar workspace which simplifies the motion planning task for the robot. In this work, we avoid implementing complex obstacle avoidance around branches and leaves since in nearly all cases the robot performs “in and out” motions thanks to novel harvesting tool design. The motion planner of the robot takes into account self collisions with the robot arm and base platform, as well as a simple planar collision obstacle placed slightly behind the target crop row.

V Experiment and Results

Two field trials were conducted over two harvesting seasons on a farm in North Queensland (Australia) within a protected cropping system. Overall the robot platform has been tested on a total of 75 sweet peppers in a real protected cropping system. Within this work, two different sweet pepper cultivars were trialled; the Claire cultivar in the first field trial and the Redjet cultivar in the second.

Trial 1 applied the lab tested grasping methods presented in [15] in a real protected cropping environment. Changes to the platform between field trial 1 and 2 were also made in order to improve the autonomous behaviour of the system. These differences include:

• •

The scanning trajectory was changed from a diamond pattern to boustrophedon pattern

• •

an upgrade from a manual scissor lift base platform (see [15]) to a custom mobile platform (Figure 1),

• •

the addition of a prismatic lift joint (integrated into the motion planner) replacing the manual lift joint to improve the workspace and reduce planning failures,

• •

the addition of a vacuum sensor to detect successful attachment and a micro-switch to detect decoupling of the suction cup and cutting blade.

V-A Methodology

A formal experiment was conducted on the final day of each field trial, involving 24 and 26 sweet peppers for the first and second trial, respectively. Each trial was conducted within a single 10 m stretch of a crop row. During the first field trial initial testing resulted in successfully picking a total of 25 additional sweet peppers, though not under strict experimental conditions.

The methodology for each experiment is as follows. The base platform was placed at the start of a crop row and manually moved down the crop row after attempting to harvest a set of sweet peppers. The platform was positioned in order for the robot arms workspace to be within range of the next set of sweet peppers. For the first field trial, this involved physically pushing the platform and manually operating the scissor lift joint. For the second trial, the robot was remotely driven down the crop row, simulating an autonomous move base state (see Figure 5). There was no need to position the lift joint manually during the second trial as this was handled by the motion planner.

If a failure occurred during an attempt, the robot arm moved back to its start position and the attempt was retried. If obstructions or occlusions caused multiple failed attempts, the scene was modified by either removing leaves or by adjusting the position of the sweet pepper, and then re-attempted.

During each attempt, an attachment (the suction cup attached to the sweet pepper) and detachment (the peduncle was cut) success or failure was recorded. Additional notes were also recorded through the experiment and categorised as: major or minor sweet pepper damage (XD or MD), whether the sweet pepper was visually occluded or physically obstructed (OC or OB), or if the sweet pepper had an irregular shape (IS).

The scanning parameters such as size, speed and row offset for each trial are given in Table I. These parameters were empirically determined based on the protected cropping system and sweet pepper cultivar used. The scanning trajectories chosen did not change within the respective field trial.

The parameters for ranking surface based grasps are also given in Table II.

V-B Results

The success rates for the both field trials are presented in Table III. More in depth results for each of the 24 trials during the final experiment are shown in Table IV. A video of the robotic harvester demonstrating these results is available at https://youtu.be/Fm70wHJ_Lew.

Out of the total sweet peppers, $14/24$ (58%) and $11/26$ (42%) were successfully harvested for field trial 1 and 2 respectively, where a successful harvest includes a successful attachment and detachment. During field trial 1 and 2, a total of $22/24$ (92%) and $11/26$ (42%) sweet peppers were successfully detached from the plant, irrespective of an attachment success. On the other hand, attachment rates of $14/24$ (58%) and $21/26$ (81%) were achieved for field trial 1 and 2, respectively.

An example of two cases where detachment was unsuccessful (sweet peppers 6 and 19 within field trial 1) the sweet pepper was irregularly shaped which resulted in a poor grasp pose estimation. In the first case (sweet pepper no. 6), this led to the cutting blade repeatedly missing the peduncle, whereas in the second case (sweet pepper no. 19), the cutting blade caused major damage to the sweet pepper (see Figure 8(c)).

The average picking time was 35-40 seconds which includes the total time to perform a scan of each sweet pepper (approximately 15 seconds), fitting the model, pose and motion planning (5-10 seconds) and execution of the plan on the robot arm (10-15 seconds). The average number of attempts for each sweet pepper was 2.0 and 2.2 for field trial 1 and 2 respectively.

Example sweet peppers harvested by Harvey are shown in Figure 8. An example of a successfully harvested sweet pepper with minor damage (circled in blue) to its top face is shown in Figure 8(a). The model and grasp selection results for five separate trials are also provided in Figure 9. It can be seen qualitatively that, for most cases, the model and grasp selection methods find appropriate locations for grasp points. In some cases, a poor estimate of the model was found and this is believed to be caused by irregularly shaped sweet peppers. An example of a poor estimate can be seen in the top right example within Figure 9.

VI Discussion & Conclusion

This paper describes an autonomous crop harvesting system that achieves state-of-the-art results for sweet pepper harvesting in a protected cropping environment. The system is demonstrated using a custom end-effector that uses both a suction gripper and oscillating blade to successfully remove sweet peppers in a protected cropping environment. The high success rate of detachment (92%) for field trial 1 is a promising result, especially since detachment is regarded as one of the most challenging aspects of the harvesting process. However, further advancements are necessary for a more general and commercially viable system.

Most failures of field trial 1 occurred within the attachment stage. In particular, obstructions from leaves or string caused 40% of the attachment failures, while irregularly shaped sweet peppers caused 30% of the total attachment failures. In contrast, the attachment rates within field trial 2 are much higher. This can be attributed to the difference of grasp selections between field trial 1 and 2 (surface normals instead of the model fitting approach) and the ability to detect attachment failures using a pressure sensor. In some cases within field trial 1 the suction cup collided with the obstructions while approaching the sweet pepper causing the tool to separate prematurely. This problem was avoided within field trial 2 by the addition of a sensor measuring whether premature separation occurred.

The most common detachment failure was found to be the cutting tool missing either side of the peduncle. In this work the cutting point was calculated by assuming the peduncle protrudes vertically from the centre of the sweet pepper. This assumption occasionally breaks since some sweet peppers have peduncles that do not grow vertically (such as the second cultivar Redjet). To improve the detachment reliability, future work will be aimed not only at detecting sweet peppers, but also at detecting the peduncle. We have conducted preliminary work that shows promising results using Point Feature Histograms (PFH) for peduncle detection in [28].

The lower detachment success rate for field trial 2 is most likely attributed to the different cultivar (Redjet) involved in the second trial. It was observed that the Redjet cultivar had more challenges associated with it, including shorter, thicker peduncles and more crop growing on the inside of the canopy, increasing the number of obstructions. This result highlights the importance of selecting cultivar that are more suitable for automated harvesting.

The novel contributions of this work have resulted in significant and encouraging improvements in sweet pepper picking success rates compared with the state-of-the-art. The methods presented in this paper provides steps towards the goal of fully autonomous and reliable crop picking systems that will revolutionise the horticulture industry by reducing labour costs, maximising the quality of produce, and ultimately improving the sustainability of farming enterprises.