Automated apple harvesting has attracted significant research interest in recent years due to its potential to revolutionize the apple industry, addressing the issues of shortage and high costs in labor. One key enabling technology towards automated harvesting is robust apple detection and localization, which poses great challenges because of the complex orchard environment that involves varying lighting conditions and foliage/branch occlusions. In this dissertation, we first propose a suppression Mask RCNN to generally improve the accuracy for apple detection. Our developed feature suppression network significantly reduces false detection by filtering non-apple features learned from the feature learning backbone. At the same time, we propose a novel deep learning-based object detection method Occluder-Occludee Relational Network (O2RNet), which addresses the challenge of detecting and isolating clustered apples in apple orchards. Previous object detection techniques have exhibited limited success in handling fruit occlusion and clustering, which are common issues in agricultural settings. To overcome these challenges, O2RNet employs a two-stage approach. In the first stage, we use a custom deep Feature Pyramid Network (FPN) architecture to generate candidate regions of interest (ROIs) for potential fruit objects. The second stage feeds these candidate ROIs into the occluder branch and occludee branch respectively using a feature expansion structure (FES). By leveraging this two-stage approach, O2RNet can effectively isolate individual apples from clustered regions, thereby facilitating accurate apple detection. Then, we propose Active Laser-Camera Scanning (ALACS) to achieve a high-precision 3D localization of detected apples and overcome existing localization challenges like varying illumination conditions, complex occlusion scenarios, and limited geometric information. The hardware of ALACS includes a red line laser, an RGB camera, and a linear motion slide. All these components are seamlessly integrated for fruit localization by using an active scanning scheme and laser-triangulation technique. The technique integrates semantic information from O2RNet’s detection results with bounding boxes to generate accurate 3D coordinates for each detected apple. Additionally, we propose Skeleton-lead Segmentation Network (SkeSegNet) and introduce it to the Panoptic-Deeplab. SkeSegNet is used to address the challenges of segmenting complex branches by treating branches as a combination of skeletons. Combined with depth map, SkeSegNet generates 3D branches for efficient obstacle avoidance. Lastly, we evaluate each approach in the comprehensive experiments and superior experimental results demonstrated the effectiveness of the proposed approaches.
Read
