DESIGN AND EVALUATION OF END EFFECTORS AND AN INDOOR SIMULATED ORCHARD ENVIRONMENT FOR ROBOTIC APPLE HARVESTING By Nathan Dickinson A THESIS Submitted to Michigan State University in partial fulfillment of the requirements for the degree of Biosystems Engineering- Master of Science 2022 ABSTRACT DESIGN AND EVALUATION OF END EFFECTORS AND AN INDOOR SIMULATED ORCHARD ENVIRONMENT FOR ROBOTIC APPLE HARVESTING By Nathan Dickinson A recently developed vacuum-based harvesting robot has shown promise in picking fruit from clusters, providing better access to tree canopies and minimizing fruit bruising. One of the main technical issues for this harvesting robot is the design of an end effector that can effectively grip the fruit for detachment. Field fruit picking studies using manual straight pull and twisting picking methods were first conducted for three varieties of apple. The critical pulling force and torque needed to detach 95% of apples were determined to be 28.3 N and 0.257 N-m (equivalent suction force of 21.0 N for the current robot’s vacuum tube). Three new conformable silicone end effectors with different configurations (i.e., “Straight”, “Bellow”, and “Curved”) were designed and fabricated to provide more effective air sealing, and thus lower vacuum pressure for increased gripping force to effectively detach fruit compared to the robot’s original end effector. Laboratory and field picking performance studies with the harvesting robot showed that all three new end effectors performed significantly better than the original, non-conformable end effector. The straight end effector achieved 87% picking success rate; performing consistently better than the other two new end effectors based on multiple performance metrics, and hence should be used in further development of the robotic harvesting system. To enhance robotic harvest research, an indoor simulated orchard environment was constructed, which allows to mimic real fruit picking processes by using artificial trees embedded with specially designed tree branches and magnetic artificial stems for use with real fruit and a unique light system that can simulate different natural lighting conditions for different times of day. ACKNOWLEDGMENTS First and foremost, this thesis would not have been possible without the continued support from faculty advisor Dr. Renfu Lu. I would like to sincerely thank him for granting me the opportunity to start working with his outstanding research team during my time as an undergraduate at Michigan State University, and for offering me the opportunity to continue with the group throughout my graduate studies. His mentorship has shaped me into the engineer that I am today, and I feel extremely grateful to have learned from him for so many years. I would also like to thank committee members Dr. Zhaojian Li and Dr. Daniel Guyer for their support throughout my thesis research. Their expertise and guidance was invaluable, and has been integral in my development both as a student and as a professional. This research would not have been possible without access to the MSU Horticultural Teaching and Research Center to conduct our 2020 field test evaluation of our robotic system. For this I must thank Farm Manager, William Chase. For his assistance in the design and construction of the lighting system used in the artificial orchard testing environment, I thank Tom Stuecken. For his mentorship throughout both my undergraduate and graduate research and allowing use of his machine shop, I thank Phillip Hill. Last but certainly not least, I thank my parents Cynthia and Louis, as well as my siblings Brian, Sarah, and Julia. The continual support of my family inspires me to push through any challenge life puts in my way. Thank you! iii TABLE OF CONTENTS LIST OF TABLES ....................................................................................................................... vi LIST OF FIGURES ................................................................................................................... viii CHAPTER 1: INTRODUCTION ................................................................................................ 1 Justification of the Research .................................................................................................... 2 Current Status and Challenges of Robotic Harvesting Technology ..................................... 4 New Vacuum-based Robotic Harvesting System ................................................................... 7 Perception System ................................................................................................................... 7 Manipulation System .............................................................................................................. 8 Overview of Picking Process ................................................................................................ 12 Research Objectives ................................................................................................................ 13 CHAPTER 2: LITERATURE REVIEW ................................................................................. 15 End Effectors for Conventional Robotics Technology in Fruit Harvesting ...................... 16 Vacuum-Based Harvesting Robotics ..................................................................................... 19 Other Non-conventional Fruit Gripping Technologies........................................................ 22 Mechanisms of Fruit Detachment .......................................................................................... 25 Summary .................................................................................................................................. 27 CHAPTER 3: MEASURMENT AND ANALYSIS OF APPLE PICKING FORCES ......... 29 Introduction ............................................................................................................................. 29 Materials and Methods ........................................................................................................... 30 Pulling and Twisting Experiments ................................................................................... 30 Data Analysis .................................................................................................................... 34 Results and Discussion ............................................................................................................ 38 Conclusions .............................................................................................................................. 43 CHAPTER 4: END EFFECTOR DESIGN AND EVALUATION ........................................ 44 Introduction ............................................................................................................................. 44 Design of End Effectors .......................................................................................................... 46 Fabrication Process ................................................................................................................. 51 Laboratory Evaluation of End Effectors .............................................................................. 53 Gripping Force Measurements......................................................................................... 54 Calculation of Equivalent Torques and Vacuum Pressures ........................................... 55 Results and Discussion ..................................................................................................... 58 Field Test Evaluation .............................................................................................................. 63 Vacuum Pressure Measurement and Analysis ................................................................ 63 Fruit Dropping Mechanism .............................................................................................. 67 iv Picking Data Analysis ....................................................................................................... 68 Results and Discussion ..................................................................................................... 71 Overall Picking Success Rate......................................................................................... 71 Vacuum Pressure Evaluation ......................................................................................... 72 Orientation Analysis....................................................................................................... 76 Rotation Performance .................................................................................................... 78 Other Performance Parameters ..................................................................................... 80 Overall Discussion ................................................................................................................... 82 Conclusions .............................................................................................................................. 84 CHAPTER 5: DESIGN AND CONSTRUCTION OF AN INDOOR SIMULATED ORCHARD ENVIRONMENT .................................................................................................. 85 Introduction ............................................................................................................................. 85 Overview of the Indoor Simulated Orchard Environment ................................................. 86 Light System.......................................................................................................................... 88 Artificial Tree Structures...................................................................................................... 93 Artificial Fruit Stems and Tree Branches ........................................................................... 95 Summary ................................................................................................................................ 101 CHAPTER 6: CONCLUSIONS .............................................................................................. 102 APPENDICES ........................................................................................................................... 105 APPENDIX A: Vacuum Pressure Profiles Collected During 2020 Field Test ................ 106 APPENDIX B: MATLAB Script for Generating Pressure Profiles ................................. 118 APPENDIX C: 2020 Field Test Results Tables .................................................................. 119 APPENDIX D: ANOVA Tables Comparing End Effector Relative Pressure Drop Performance on Successful Picks ......................................................................................... 132 APPENDIX E: Standard Operating Procedure (SOP) for the Simulated Orchard Environment .......................................................................................................................... 135 REFERENCES .......................................................................................................................... 137 v LIST OF TABLES TABLE 3.1: Summary of the mean measured forces (standard deviations) required to detach fruit by a “straight pull” picking action .........................................................................................39 TABLE 3.2: Summary of the mean torques (standard deviations) calculated from the mean forces measured to detach apples by the twisting picking action ..................................................40 TABLE 3.3: Results of single-factor ANOVA comparing measured torque values for each tested variety ............................................................................................................................................41 TABLE 3.4: The critical force, torque, and vacuum pressure needed for the current robotic system with the thin-foam end effector to detach 95% of apples based on the measured pulling and twisting forces for three apple varieties ..................................................................................42 TABLE 4.1: Key physical properties of Dragon SkinTM 20 silicone rubber used for end effector fabrication ......................................................................................................................................47 TABLE 4.2: Average gripping forces and standard deviations for the control and three new end effectors and average equivalent torque and vacuum pressure calculated for the control and straight end effectors ......................................................................................................................61 TABLE 4.3: Technical specifications of the hall-effect pressure sensor used for recording vacuum pressure of the robotic system during 2020 field tests .....................................................64 TABLE 4.4: Overall picking success rate and number of picking attempts for each end effector design during field evaluation ........................................................................................................71 TABLE 4.5: Results of single-factor ANOVA comparing the relative pressure drop for successful picking attempts and failed picking attempts ...............................................................74 TABLE 4.6: Successful picking rate (%) of the rotation mechanism causing fruit detachment.. 78 TABLE 4.7: Successful picking rate (%) of the rotation mechanism at various attachment orientations .....................................................................................................................................79 TABLE 4.8: Performance of each end effector design alternative in the presence of environmental occlusions...............................................................................................................81 TABLE A.1: Picking scenarios and picking success rate ...........................................................119 TABLE A.2: Recorded pressure sensor output results ...............................................................125 TABLE A.3: Single factor ANOVA table comparing all four end effector designs ..................132 vi TABLE A.4: Single factor ANOVA table comparing Straight and Control end effector designs ......................................................................................................................................................132 TABLE A.5: Single factor ANOVA table comparing Bellow and Control end effector designs ......................................................................................................................................................133 TABLE A.6: Single factor ANOVA table comparing Curved and Control end effector designs ......................................................................................................................................................133 TABLE A.7: Single factor ANOVA table comparing Straight, Bellow, and Curved end effector designs..........................................................................................................................................134 vii LIST OF FIGURES Figure 1.1: Image showing v-trellis (left) and vertical planar (right) canopy structures (Photos courtesy of Dr. Renfu Lu) ................................................................................................................6 Figure 1.2: A vacuum-based robotic harvesting system with three main components marked on the image ..........................................................................................................................................7 Figure 1.3: Example of the visual perception system identifying target fruit using bounding boxes ................................................................................................................................................8 Figure 1.4: Diagram highlighting the two main mechanical components of the 3-DOF manipulator ......................................................................................................................................9 Figure 1.5: Image highlighting the presence of air leaks resulting from the original thin-foam end effector’s non-conformity to the irregular contours of fruit....................................................10 Figure 1.6: Diagram describing the systems rotation mechanism. Activation of the small 625 rpm motor generates rotation of the vacuum tube .........................................................................11 Figure 2.1: Three distinct picking motions. “Twist and pull” method (left), “bend and pull” (center), and “pendulum” motion (right) .......................................................................................27 Figure 3.1: Attachment of an apple to the digital force gauge for evaluating a “straight-pull” picking action .................................................................................................................................31 Figure 3.2: Image describing the robot’s approach to the cheek of the fruit (left) and twisting motion for detachment (right) ........................................................................................................32 Figure 3.3: Example of the end effector (the bellow design) attaching to the bottom of the target fruit. This image was taken in the indoor simulated orchard environment described in Chapter 5 ........................................................................................................................................................32 Figure 3.4: The rotation mechanism used to hold the fruit and measure the force required to detach with a twisting motion ........................................................................................................33 Figure 4.1: Front view of three silicone end effector designs generated by Fusion 360 software (Autodesk, San Rafael, CA)...........................................................................................................48 Figure 4.2: Contour sketch of each end effector design describing the major dimensions (same for all three designs) and distinction between “inner-lip” and “outer-lip” (left- Straight, center- Bellow, right- Curved) ...................................................................................................................49 viii Figure 4.3: The “bellow” end effector design alternative installed on the aluminum vacuum tube ........................................................................................................................................................49 Figure 4.4: Wire frame view of three end effector designs (left- straight, center- bellow, right- curved) annotated with key dimensions of the “inner-lip” and “outer-lip” for each design..........50 Figure 4.5: 3D printed master submerged in silicone creating a flexible mold ............................52 Figure 4.6: Final fabricated silicone end effector designs (white) alongside their hard plastic 3D printed masters (black) ...................................................................................................................53 Figure 4.7: The experimental setup to evaluate the gripping force of each end effector design alternative. The original thin-foam end effector is shown in this image .......................................55 Figure 4.8: Diagram showing gripping of an apple by the “straight” end effector. Red lines represent the points of contact between the fruit and end effector, around the apple’s perimeter. The apple ideally is gripped by the “inner-lip” and by the “outer-lip” around its entire perimeter ........................................................................................................................................................57 Figure 4.9: Diagram showing approximation of apple gripping by the end effector to a collar bearing (apple) sitting on a hollow circular contact area (end effector). The area of contact (shaded) between the apple and end effector with its inner and outer contact diameters being denoted as Ri and Ro. F refers to the gripping force and  is the friction coefficient between the apple and the silicone end effector.................................................................................................57 Figure 4.10: Mean gripping forces (25 replications) for the thin-foam control and three end effector design alternatives for two different sized apples with whiskers indicating standard deviation. * Indicates a statistically significant difference between the recorded means for the two different sized test fruit ...........................................................................................................59 Figure 4.11: Gripping force evaluation for the “small” fruit attached to the “straight” end effector design ................................................................................................................................62 Figure 4.12: Example of a vacuum pressure profile collected during the picking process ..........65 Figure 4.13: Vacuum cutoff mechanism in the open position (left) allowing vacuum flow through the tube, and in the closed position (right). This was manually operated to cause fruit to fall from the end effector after picking during 2020 field testing..................................................67 Figure 4.14: Classifications of fruit attachment orientations evaluated during the field test .......69 Figure 4.15: Average relative pressure drops with associated standard error (whiskers shown on individual bars) on successful and failed picking attempts ...........................................................72 Figure 4.16: The folding of the “curved” end effector design occurred during fruit attachment. 73 ix Figure 4.17: Average picking success rate (SPR) and relative pressure drop (RPD) for each design alternative at different attachment orientations ..................................................................76 Figure 5.1: Overall dimensions and layout of the simulated orchard environment (SOE). Shown inside the SOE frame is a 3.6 m (12 ft) movable tree frame for installing four artificial trees .....87 Figure 5.2: Custom designed triangular light assembly used in the simulated orchard environment. Side view of the triangular light assembly with a pin-locking mechanism allowing to select one of the three LED light panels with a specific incident angle to illuminate the tree canopy (right), and the opposite side of the triangular configuration with power inputs and dimming controls for the lights (left) .............................................................................................90 Figure 5.3: Photos of the cable/pulley light suspension system used to position the light system within the simulated orchard environment ....................................................................................91 Figure 5.4: Photo showing the I-beam which runs across the top of the simulated orchard environment, perpendicularly to the artificial canopy (below image) ...........................................92 Figure 5.5: Front view of light panels suspended from the winch cable/pulley system, mounted to I-beam. Safety rope (orange) is engaged ...................................................................................92 Figure 5.6: Artificial trees installed on the movable tree frame in the simulated orchard environment ...................................................................................................................................93 Figure 5.7: (left) Visual perception bounding box (blue) image detecting apples within the simulated orchard environment and (right) an example of a real orchard image used as background images for the simulated orchard environment ..........................................................95 Figure 5.8: Artificial stem used to magnetically attach a real apple to an artificial branch. (left) The naked artificial stem consisting of a brown 12 gauge wire with the two ends affixed with two steel collars and two magnetic disks, one of which is attached to a screw that is embedded into the stem end of a real apple and the other end attached to the artificial branch, and (right) the same stem with the two collar/magnet sets covered with brown tape to mimic the actual color of fruit stems.......................................................................................................................................96 Figure 5.9: Steel shaft collar installed on an artificial steel “branch” (left). Used to magnetically attach a real fruit using an artificial stem (right)............................................................................98 Figure 5.10: Real apples (Gala) hung in the artificial canopy in clustered formations and individually, using magnetic artificial stems .................................................................................98 Figure 5.11: Two real apples mounted using variable force electromagnets and magnetic artificial stems ..............................................................................................................................100 x Figure 5.12: Photos showing the light system properly positioned relative to the artificial tree canopy for testing of the robotic system ......................................................................................100 Figure A.1: Straight end effector design pressure profiles .........................................................106 Figure A.2: Bellow end effector design pressure profiles ..........................................................109 Figure A.3: Curved end effector design pressure profiles ..........................................................112 Figure A.4: Control end effector design pressure profiles..........................................................114 xi CHAPTER 1: INTRODUCTION Apples are the second most consumed fresh fruit in the United States because of their wide availability, flavor, texture, and nutritional benefits such as improving neurological health and decreasing risk of lung cancer (Zhang et al., 2019). As the largest producer of fresh market apples in the country, the state of Washington alone harvests 15-18 billion apples annually, mostly by hand-picking (Shamshiri et al., 2018), which requires significant manual labor. Most commercially produced apples across the world are still manually harvested. However, advancements in agricultural technology have fundamentally changed the structure of agriculture; shifting from strenuous and tedious manual operations to highly efficient mechanical or automated systems (Shamshiri et al., 2018; Silwal et al., 2017). While manual picking of fresh produce like apples has been practiced for centuries, there is a pressure on the fruit industry to modernize current production systems to improve production efficiency and reduce labor cost. It is generally agreed that the mechanization of harvesting systems is a crucial aspect of accomplishing these goals (Burks, Bulanon, & Mehta, 2017; Davidson, Hohimer, & Mo, 2016; Gray, 2007; Shamshiri et al., 2018; Zhang et al., 2019), as increasing efficiency and reducing labor requirements will ensure high productivity and competitiveness in the global market (Burks et al., 2017; Liu, Li, & Mao, 2015). Harvest mechanization can be categorized into two distinct groups: bulk harvesting and selective harvesting (Z. Zhang et al., 2019). Bulk harvesting systems enable large quantities of fruit to be harvested simultaneously. This could involve shaking trees to detach fruit and catching the falling fruit, or some other methods. Selective harvesting systems pick one fruit at a time by using robotic technology. Both types of systems have potential for improving production and alleviating labor concerns. Unfortunately, these types of systems are still not adopted by 1 growers due to insufficient performance, high machinery costs, limited production improvements, and resistance to change shared by many growers as manual harvesting has been utilized for centuries (Pothula et al., 2018; Scharfy et al., 2017). Fruit bruising is the single most common issue for mechanical and automated harvesting systems. Bruising may be acceptable for harvested fruit destined for processing (apple sauce, apple juice, etc.), but is unacceptable for fruit destined for the fresh market (Zhang et al., 2019). Individually ripening tree fruit are susceptible to bruising which results in loss of quality and lower crop value, leading to financial loss for growers and subsequent retailers (Liu et al., 2015; Pothula et al., 2018). Fifty years ago, researchers had recognized that the ideal fruit harvester is one that can be handled by a single operator, can selectively pluck fruit from trees effectively with no damage to the fruit and/or tree, and automatically transports them to a storage bin (Macidull, 1972). However, a viable solution is still not yet available for the industry. Justification of the Research With 90 million MT produced globally in 2016, apples are one of the mostly highly demanded fruits globally and their consumption will continue to grow (Zhang et al., 2019). Not only has the desire for fresh fruit grown, but global populations are also growing at alarming rates. It is estimated that the global population will reach 9.8 billion by 2050, equivalent to adding a new city of 1.4 million people every week (Shamshiri et al., 2018). This combination of increased desire for fresh produce and a rapidly growing global population puts modern farms in a situation where they are expected to produce higher quality fruit with increased yield in a way that is sustainable and less dependent on manual labor (Scharfy, Boccali, & Stucki, 2017; Shamshiri et al., 2018). 2 Moreover, low commodity prices in combination with high labor costs have put American growers in a difficult position to remain economically competitive (Burks et al., 2017). With imported fruit having equal or greater quality as homegrown fruit, it is critical that U.S. growers lower their production costs while increasing overall production efficiency and fruit quality. Adding additional strain to American growers, the agricultural labor force is depleting rapidly. Aging populations create an issue where there are fewer young, able-bodied people to work these strenuous harvesting jobs and support the increased production required to feed the growing and aging population (Whitesides, 2018). As a result, harvesting costs have increased significantly as not only have growers found it difficult to find laborers, but the cost of labor has increased as well (You & Burks, 2016; Zhang et al., 2019). It was reported that in the state of Washington State, harvesting labor alone accounts for roughly 1/3 of total annual variable costs, equal to the cost of pruning and thinning labor combined (Silwal et al., 2017). While growers are concerned about high labor cost, the wages offered to harvesting workers are still low or not competitive compared to unskilled labor positions in other non-agriculture industries in the United States. This results in a shortage of domestic laborers; Hence, the apple industry must rely on migrant, foreign workers. The adoption of mechanized harvesting technologies may alleviate these labor concerns for farmers. The most significant factor in the depletion of the agricultural workforce is likely the fact that these workers are generally undercompensated for the high physical demands of their work. In traditional apple harvesting, workers carry a “bushel basket” which hangs below their chest with a strap hung around their neck. Workers walk from tree to tree, frequently climbing up and down ladders, filling their baskets until they are full. The full baskets, weighing up to 20 kg (45 lbs.), are then carried over to a collection bin where the apples are dumped, and the process is 3 repeated (Zhang et al., 2019). This highly repetitive and strenuous work exposes workers to a wide range of occupational hazards including falls from ladders and ergonomic injuries, in addition to the inherent risks associated with farm work (heat exhaustion, dehydration, pesticide exposure, etc.) (Silwal et al., 2017). It is no surprise that this generation of younger American people would tend to look elsewhere for work. In recent years, mechanical harvest assist platforms have become widely available to alleviate the physical burden of manual harvesting and improve harvesting efficiency (Zhang et al., 2019). However, in a survey of 316 Washington apple growers, it was found that only approximately 1 of every 10 growers use mechanical harvest assist platforms, as many of the systems are not economically feasible or are not compatible with their orchards (Silwal et al., 2017). Hence, if a robust robotic system can be developed which maintains apple quality and achieves high productivity while decreasing the reliance on high-cost, unreliable labor, there would be huge potential benefits to growers. The economic benefits are obvious, since harvesting and post-harvest handling account for 50% of total production cost (Zhang, Pothula, & Lu, 2018). Robotics can also improve worker safety; decreasing employer liabilities by automating these strenuous and potentially dangerous jobs (Burks et al., 2017). Current Status and Challenges of Robotic Harvesting Technology Integration of robotics into harvesting systems is the apparent solution for automating and improving the efficiency of these machines. Robots have been used extensively in modern agriculture for operations like plowing, fertilizing, sowing, seeding, and plant treatment (Oliviera et al., 2021). These types of robots are taking agriculture into a new age. Unfortunately, after nearly half a century of research, robotic tree fruit harvesting still has not been realized. This is because tree fruit harvesting robots face a unique set of challenges that are different from most 4 other robotic applications within agriculture and other industries (Shi, Guo, Liu, Zhao, Chen, & Chiu et al., 2016). Selective harvesting robots must be able to touch, sense, and manipulate delicate targets (fruit), while navigating dynamic environments that are also vulnerable to damage (Shamshiri et al., 2018). In general terms, these robots must operate in “unstructured” environments whereas most industrial robotic systems are designed to operate in consistent and predictable “structured” environments, making the design of these robotic harvesting systems much more challenging than traditional robotic systems (Kourtev, 2018; Shamshiri et al., 2018). Moreover, these systems must also be cost-effective and maintain/improve overall production to compensate for the additional capital costs of automation (Shamshiri et al., 2018). To date, most successful applications of these systems have been limited to applications where fruit damage is not a crucial consideration (Burks et al., 2017). The potential of damage to the fruit and to the canopy creates challenges for researchers. Additionally, apples are a biological material with inherent variability in size, shape, weight, and motion while hanging on the tree (Silwal et al., 2017). This variability, coupled with variable natural lighting and/or uneven terraneous conditions, has become the main hurdle for much of the research over the past decades, which results in insufficiently robust fruit detection systems, ineffective end effectors for gripping and detaching the fruit, and high cycle times (Liu et al., 2015; Silwal et al., 2017). Over the past two decades, apple growers have transformed traditional orchards, where trees are wide and unstructured, into high-density structured tree architectures, such as the planar and V-trellis tree canopies shown in figure 1.1. These modern orchards produce higher yielding, better quality, and more consistent fruit while increasing efficiency for manual picking (Silwal et al., 2017). 5 Figure 1.1: Image showing v-trellis (left) and vertical planar (right) canopy structures (Photos courtesy of Dr. Renfu Lu) These planar canopies not only ease the work and improve efficiency of manual picking; they also provide huge potential for the implementation of robotic picking systems by removing several factors of variability from the work environment (Silwal et al., 2017). With a planar structure of these tree canopies, the unstructured biological environment becomes more like a controlled industrial environment. It would thus become technically feasible to develop a robust vision and robotic manipulation systems including an end effector that can effectively isolate and detach apples from a tree. Field testing is a critical step for evaluation of the performance of a robotic harvesting system. However, an apple harvest season typically lasts for four to six weeks. Therefore, it is critical to have a robust indoor environment that closely mimics real commercial apple orchard conditions (lighting, tree and fruit hanging structures, picking forces, etc.). Such a simulated environment would be useful for testing and evaluating design prototypes and optimizing system parameters when field testing is impossible or becomes too time consuming. (Hemming et al., 2010). 6 New Vacuum-based Robotic Harvesting System Recent research has found that vacuum-based end effectors can be reliable, effective, and compact enough to navigate these dynamic environments while avoiding collisions that could damage the robot and the adjacent environment (Kourtev, 2018). The first generation of the newly developed robotic apple harvesting system consists of a simple mechanical manipulator module combined with a single RGB-D camera vision module for detection and localization of fruit (Lu et al., 2020; Zhang et al., 2021). This robotic system is equipped with a vacuum tube to selectively grip and detach fruit using a rotating picking motion. Figure 1.2 shows a general overview of the harvesting robot. The first version of the robotic system showed promising preliminary results for the vacuum-based harvesting concept. However, the system also had clear shortcomings which required improvement to make the system more viable for implementation in commercial harvesting settings. Figure 1.2: A vacuum-based robotic harvesting system with three main components marked on the image Perception System The robot’s visual perception system consists of a single Intel RealSense D435i RGB-D (Intel Corporation, Santa Clara, CA), which is mounted on the frame of the robotic system. This 7 camera detects target fruit and provides 3-dimensional coordinates (fruit localization), along with other environmental information to the manipulation system, which then properly positions the vacuum head for picking. These additional environmental factors include the presence of occlusions such as leaves and/or branches, and color/variety of fruit. Figure 1.3 shows an example image of the visual perception algorithm detecting apples using green bounding boxes. Figure 1.3: Example of the visual perception system identifying target fruit using bounding boxes Manipulation System The robot’s mechanical system is a 3 degree-of-freedom (DOF) manipulator. This 3-DOF manipulator is mounted on a field navigating Segway RMP400 (Segway Robotics Inc., New Hampshire) allowing the robot to travel throughout an orchard. After detection of target fruit by the vision system, fruit localization information is fed through the robotic operating system (ROS) to facilitate manipulation of the vacuum tube (Zhang et al., 2021). The manipulator consists of a pneumatic linear slide, denoted as “prismatic joint”, and two revolute joints, denoted as “pan-and-tilt module” in figure 1.4. The simple and compact pan- and-tilt module offers sufficient DOFs for manipulation of the vacuum head to reach the target 8 fruit, as well as facilitating high-level motion control for optimizing motion trajectory to improve harvesting efficiencies. This type of pan/tilt module has been shown to be effective in various robotic “pick-and-place” applications (Lee, Lan, Chu, Lai, & Chen, et al., 2013; Murray, Li, & Sastry, et al., 1994). The prismatic joint is a pneumatically actuated Lintra “rodless” air cylinder and a slide carriage used to extend the vacuum tube into the depth of the canopy to reach the target fruit. Pneumatic actuation of this linear slide allows for high speeds while maintaining sufficient control to accurately reach the target fruit Figure 1.4: Diagram highlighting the two main mechanical components of the 3-DOF manipulator The 3-DOF manipulator directs the motion of a 44.5 mm diameter aluminum vacuum tube with a length of 0.71 m. The vacuum tube is connected to a 5.5 hp wet/dry vacuum system (Craftsman, Chicago, IL) (not shown in figure 1.2). This system generates roughly 145 cfm of vacuum flow rate for attracting and attaching fruit to the vacuum head. The original “end effector” of the first-generation system was simply a thin layer of polyurethane foam applied to the rigid end of the hollow aluminum vacuum tube. This thin layer of foam, as shown in figure 1.5, was intended to simply cushion the fruit during attachment to the rigid rim of the vacuum tube to minimize fruit bruising during the picking process. 9 Figure 1.5: Image highlighting the presence of air leaks resulting from the original thin-foam end effector’s non-conformity to the irregular contours of fruit This original foam end effector had several shortcomings. First and foremost, the rigid nature of the aluminum tube, coupled with the very thin foam was not conformable to allow sufficient gripping of the fruit. Irregular geometries of apples create air leaks in the attachment between the fruit and the vacuum tube as highlighted by the light blue circle in Figure 1.5. These air leaks result in a reduced vacuum pressure drop within the tube during fruit attachment, leading to a reduced holding force (Schmalz, 2021): 𝐹 = 𝐴 × (𝑞 − 𝑞) (1) where F = Holding force required to detach the target fruit A = Contact area (based on the inside diameter of an end effector design) 𝑞 = Atmospheric pressure 𝑞 = Pressure generated in vacuum tube during fruit attachment Throughout this thesis, the two terms of ‘vacuum pressure’ and ‘pressure differential’ are frequently used. The term ‘vacuum pressure’ is referred to the actual absolute pressure in the vacuum system that is below atmosphere pressure (=101.325 kPa), while the term ‘pressure 10 differential’ is defined to be the difference between atmosphere pressure and the actual pressure in the vacuum tube. This reduced holding force caused an insufficient overall harvesting rate regardless of proper positioning of the vacuum head by the robot’s visual perception and manipulation systems. The reduced holding force and insufficient friction of the thin foam end effector also resulted in the apple slipping when the rotation mechanism was activated to pick the fruit from the tree (figure 1.6). The rotation mechanism was intended to apply a “twist-and-pull” picking motion to improve automated harvesting efficiency by first weakening the connection between the fruit stem and branch, thus reducing potential damage to tree branches (Li et al., 2016): Figure 1.6: Diagram describing the systems rotation mechanism. Activation of the small 625 rpm motor generates rotation of the vacuum tube The insufficient gripping force applied to the fruit by the original thin foam end effector resulted in the fruit frequently slipping during tube rotation. Therefore, although the vacuum tube rotates, the fruit frequently remained static, rendering the rotation mechanism useless. In cases when the rotation mechanism did not result in the detachment of the fruit, the vacuum tube’s return to home position may result in fruit detachment by a “straight pull” picking action. However, straight pull is not the preferred approach for fruit removal, because it tends to remove fruiting 11 spurs and damage the branch. Preliminary field test evaluation in 2019 harvest season showed that the harvesting efficiency of the first-generation robotic system was significantly lower than the overall system goal of 95% picking success rate. Hence, the need for an improved vacuum- based end effector design was apparent after evaluation of the first generation of the vacuum system. Overview of Picking Process The “pan-and-tilt” revolute joints first position the vacuum head such that the end effector is in-line with the target fruit, outside the canopy. These joints operate first such that the end effector may be properly positioned without risk of hitting any branches or other occlusions in the canopy structure. Once in line with the target fruit, the pneumatic linear slide extends the vacuum tube/end effector into the canopy to finally reach the target fruit. The inflowing air through the vacuum tube generates negative (vacuum) pressure, which attracts the fruit to the vacuum head. Despite the high accuracy of the robot’s machine vision and mechanical manipulation systems, unpredictable environmental factors (wind, uneven terrain, fruit clustering, occlusions, etc.) may cause the end effector to have slight positional errors relative to the target fruit. Maintaining a sufficiently high vacuum flow rate allows the system to overcome this marginal positioning error by isolating, attracting, and attaching target fruit to the vacuum head for picking. When a target fruit attaches to the end effector, a sharp drop in pressure will occur within the vacuum tube, which is being monitored by a pressure sensor installed on the vacuum tube. Upon detection of this pressure drop, the vacuum tube is rotated to detach the fruit by a “twist- and-pull” picking motion. The small motor shown in figure 1.6 rotates the vacuum tube by 90° in 12 the clockwise direction, followed by a 180° rotation in the counterclockwise direction. Ideally, the fruit will be detached by rotating the fruit around the axis perpendicular to the stem-calyx axis, assuming that the fruit is hanging below the stem and the vacuum head attaches to the cheek of the fruit. If the rotation mechanism fails to detach the apple, the subsequent retracting of the vacuum tube to home position may detach the fruit by a “straight pull” picking action. Hence, it is critical that the end effector and vacuum system are designed such that the robot can generate sufficiently low vacuum pressure to effectively grip and detach fruit during robotic picking operations. Research Objectives The overall goal of the research was therefore to improve the fruit gripping and detachment by the robotic harvesting system and develop an indoor simulated orchard environment to facilitate the development and evaluation of new robotic harvesting systems. The research objectives were to: 1. Determine the forces and torque needed to detach 95% of fruit from trees with straight pulling and twisting picking experiments for different varieties of apple to provide guidance for the design and evaluation of new end effectors. 2. Design and fabricate new end effectors for the vacuum-based robotic apple harvester to improve the overall picking performance and evaluate the performance of the new and original end effectors in lab and in field for picking fruit, as measured by vacuum pressure, successful picking rate, fruit orientation, fruit dropping and other parameters. 3. Design and construct an indoor simulated orchard environment to mimic the real orchard in terms of tree canopy structures (trees, branches and fruit hanging mechanism) and lighting 13 conditions, for effective testing and evaluation of the perception and manipulation systems of new robotic harvester prototypes. 14 CHAPTER 2: LITERATURE REVIEW Researchers have attempted to develop automated fresh market fruit harvesting systems for several decades. Due to economic and technological factors, a commercial robotic harvesting system has not yet been available for the apple industry, despite significant efforts from start-up companies (Abundant Robotics Inc., California; FFRobotics, Israel) and many other researchers around the world. One major challenge in robotic harvesting involves the design and implementation of an end effector which can detach apples without damaging the target fruit, neighboring fruit, or surrounding canopy structures. An effective and efficient end effector would greatly improve the viability of a robotic harvesting system to be implemented in commercial harvesting settings. Conventional robotic harvesters use finger-type end effectors to grip and detach fruit. If these fingers are fabricated with hard material, the risk of bruising fruit is significantly increased especially when picking clustered fruit. While researchers have recently developed soft end effectors, these soft actuators still struggle to pick fruit in clusters and the gripping performance is heavily dependent on the size and orientation of fruit. Moreover, the variability of dynamic orchard conditions (occlusions, wind, un-even terrain, etc.) make it difficult for accurate fruit localization by the visual perception system. The utilization of vacuum-based harvesting concepts offers several potential advantages over the conventional robotic system for automated fruit harvesting applications (Bamotra et al., 2018). First, it simplifies the fruit gripping mechanism because fruit are directly attracted by vacuum air flow to the vacuum tube, which means that the end effector can be simple and passive (no actuation required). Second, a vacuum-based robot can tolerate some errors in fruit localization due to the inflow of air attracting fruit to the end effector. Third, the vacuum-based robot coupled with a soft end 15 effector is gentle in gripping fruit, thus minimizing potential bruising damage to harvested fruit. The main disadvantage of using vacuum-based harvesting technology is the additional power required to drive a vacuum machine and the noise associated with vacuum operations. End Effectors for Conventional Robotics Technology in Fruit Harvesting Much of recent research in this area utilizes conventional robotics hardware to pick target fruit. The end effectors used in these conventional robotic systems are mostly “active” devices, meaning they require some sort of actuation to grip/detach the fruit. Several promising concepts have been proposed and tested for conventional robotic systems, but each of them has some clear disadvantages for fruit harvesting applications. Hemming et al. (2014) conducted a study evaluating two distinct end-effector designs for harvesting sweet peppers. The first design was labeled the “fin-ray” method, featuring a double- acting pneumatic long-stroke gripper and two pneumatic drives for cutting. This end-effector used four fingers to grip the fruit, then cut the peduncle to detach it. The “fin ray effect” allows this design to have passive adaptation of the fingers with the varying geometry of target fruit. The two lower fingers of the end-effector rotate to manipulate the fruit and maximize the gripping power of the fingers based on optimal fruit orientation. This design had an average success rate of 75%, and had issues causing major damage to the tree and/or supporting wires (Hemming et al., 2014). The other design evaluated in the study was labeled a “lip-type” design. This end effector first used a very small diameter suction cup to attach to the fruit. The “lips”, powered by a small air compressor, then moves forward to encircle the fruit, analogous to a human putting food in their mouth. The lower lip contains sensors which cause the forward motion of the end effector 16 to stop when it detects the peduncle. The upper lips, equipped with a sharp cutting edge, then move down towards the lower lips, like a biting mouth, to cut the peduncle and detach the fruit. This design was only successful 45% of the time but showed promising concepts for the development of future end-effectors including the use of sensors to automate picking processes (Hemming et al., 2014). These end effectors were used for sweet peppers whose peduncle is directly behind the fruit relative to a robotic harvester. Therefore the “lip-type” design would need to be reoriented such that it could be compatible with apples, whose stems are often above the apples resulting in the fruit hanging below the branch rather than sitting adjacently to the branch. The study was conducted in a laboratory in the Netherlands using a simulated orchard environment, as well as in a real orchard in Israel (Hemming et al., 2014). Their study also showed that it could be useful to develop indoor artificial environments for experimentation (Davidson, Hohimer, & Mo, 2016). One of the major issues impacting end-effector designs in previous work was detaching fruit that grow in clusters. When the fruit grow in close clusters, it is difficult for an end-effector to isolate the individual fruit for selective harvesting. Researchers attempted to solve this problem by designing an end-effector capable of harvesting 4-6 tomatoes simultaneously (Kondo et al., 2010). This end-effector consisted of two upper fingers which grasp and sever the peduncle of the fruit, and two lower fingers with peduncle detection sensors. When the manipulator approaches a target cluster of fruit, the fingers on the end effector remain open until the main stem is detected by the sensors on the lower fingers. Upon detection, the lower fingers close around the stem and the end effector slides up the stem until a peduncle is detected by a strain gauge. When the peduncle is found, air compressors activate the upper fingers to close, 17 cutting the peduncle and detaching the cluster from the tree (Kondo et al., 2010). The success rate of this design was only 50%, with 70% of the failures being attributed to an inability for the clunky end-effector to extend fully into the plant to grasp the main stem (Kondo et al., 2010). Therefore, this design showed potential for the use of a grasping-concept end-effector but had several disadvantages with the areas for improvement. Furthermore, this type of cluster harvesting is not acceptable in apple harvesting, as it may damage the branches needed for future fruit bearing. Another grasping-type end effector was developed based on the results of a hand-picking study (Silwal et al., 2017). This end effector did not contain any sensors; the grasping of fruit was executed in an open-loop system with feedforward control. The end effector design showed a promising harvesting efficiency of nearly 85%. It was noted that the main cause of error for the end effector was failure to grab the fruit due to localization error. Thus, while this gripping-end effector performed very well in a field test, this study has also demonstrated the need for a new end effector to feature vacuum flow to overcome this slight localization error. Localization error is likely to occur due to unpredictable environmental factors like wind and branch movements or disturbances caused by the picking activity of the robotic harvester. Additionally, it was noted that the individually actuated fingers of the end effector were prone to breaking and required 10 minutes of down time to manually replace the broken finger (Silwal et al., 2017). It would be desirable to have an automated system with high durability such that it can operate fully autonomously and never require human intervention to avoid downtime. While these studies have showed some promising results for automated harvesting, there are several clear disadvantages of using these conventional end effectors. End effectors that utilize conventional technology tend to be mechanically complicated (Shi, Zhu, Zhai, Zhang, 18 Liu, Zhao, & Cu et al., 2018). With several mechanical components of the end effector, collisions or normal wear and tear can easily cause the end effector to break during operation (Silwal et al., 2017). Additionally, the use of actuated fingers and/or cutting mechanisms significantly increases the risk of damaging the fruit during the picking process. Fruit destined to be sold as a fresh market product must not have any type of bruising or cutting (Zhang et al., 2019), thus a viable end effector design must minimize the damage of fruit. Lastly, these conventional mechanical components are often bulky and have difficulty navigating the complex structures of an apple canopy (Kondo et al., 2010). More compact and simple designs are needed to reach more apples within the canopy, which is difficult to achieve with a larger end effector. Vacuum-Based Harvesting Robotics Vacuum-based technology for harvesting has been extensively researched for more than half a century. Researchers have recognized potential of using vacuum technology for harvesting mechanisms due to some unique advantages. A vacuum-based system may greatly simplify or even eliminate the end effector, as the force generated by vacuum flow would enable direct detachment of fruit. The pulling force resulting from airflow through the vacuum-head also allows for some error in the positioning of the head relative to the fruit. The air flow will be able to attract fruit to the vacuum-head within a nearby radius. Additionally, vacuum based end- effectors have shown potential to handle fruit gently, minimizing bruising effects. While vacuum technology has been widely researched for harvesting applications, most of these studies were conducted before the development of robotic technology, thus they mainly focused on manually operated harvesting devices (Bullock, 1954; Macidull, 1972; Chew, 1975; Gerber, 1985). Despite the potential advantages for robotic harvesting applications discussed earlier, vacuum-based technology has not been extensively considered in recent research. 19 Commercial start-up Abundant Robotics from California has developed a vacuum-based harvesting robot intended to reduce overall losses by minimizing bruising of fruit. The system operates by scanning the tree using stereoscopic cameras and artificial lighting, and collecting the fruit by vacuum suction (Veiros, Mesquita, & Gaspar, 2019). Its end effector mainly consists of a large-size vacuum tube, and it directly sucks apples into the tube, which then transports the fruit to the destination container. The system showed promising early results, reporting a picking rate of 1.5 sec/apple and an ability to reach 50-90% of fruit on trees. However, the large vacuum tube used in the Abundant Robotics system not only requires high power, but also causes problems accessing fruit inside the canopy. Moreover, as the large vacuum tube moves into the canopy, it would suck leaves and branches on its passage. The company has been reported to cease operation in 2021, because of the lack of growers’ interest in adopting the technology (Crowe, 2021). Several important factors must be considered in designing an optimal vacuum power source for a robotic apple harvesting system. Mainly, the system must be designed to accomplish the minimum required vacuum pressure to effectively grip and detach apples. A study was performed on tomatoes, which are similar to apples in terms of size, shape and mass (Chiu, Yang, & Chen et al., 2013). The experiment used a single suction cup end effector to test the effectiveness of five suction power levels (9.1, 8.6, 8.1, 7.7, and 7.2 N/cm 2) to pick and hold fruit of various sizes, and masses (Chiu et al., 2013). It was found that success rate of the suction cup did not improve at suction force greater than 8.1 N/cm2. This indicates that once a sufficient gripping force is achieved on the target fruit, overall successful harvesting rate will not improve by further increasing the power of the vacuum. Hence, determining a target gripping force and 20 associated vacuum pressure needed to detach fruit is critical to the design and evaluation of a vacuum-based harvesting end effector. Other important factors to consider in the vacuum system design and evaluation include vacuum (volumetric) flow rate, evacuation time, power consumption, and other qualitative metrics like required maintenance and durability (FESTO, 2021). Vacuum flow rate is a measure of the volume of air that can flow through a vacuum system in a given amount of time, therefore translating to how quickly targets can be gripped by a vacuum-based end effector. This is an important consideration as air flow attracts nearby fruit to the vacuum head and allows the system to overcome air leaks resulting from imperfect sealing of a suction cup (FESTO, 2021). Vacuum flow is affected by the size of the vacuum tube (diameter), as well as the power of the vacuum source. The volumetric flow is a direct result of the pressure deficit between the atmosphere and the vacuum source, and size of the vacuum tube. Evacuation time is another important consideration for minimizing the cycle time of the system. The evacuation time is the time required to evacuate air from the vacuum tube. A lower evacuation time will lead to lower cycle times by enabling quicker gripping of fruit at the vacuum head. Cycle time must be minimized in order to achieve high harvesting efficiency. Other operations of the robotic system such as localization and manipulation time also have significant impacts on the overall cycle time of the system. It is also important to consider power consumption, as the robot is intended for use in a field where power supply may be limited or not be readily available. Power consumption should be minimized with sufficient vacuum performance for overall system efficiency. Designing a robust control system for the vacuum system is also a critical consideration for the performance of a vacuum-based robot (Hemming et al., 2014). A control system should 21 be able to activate the suction power when the end effector is near a target fruit, detect apple attachment and subsequently cut off the vacuum when the apple needs to be released. This control system could be achieved by using vacuum pressure sensors communicating with control valves in the vacuum tubes. If all challenges can be met, a vacuum system with a robust control system can be designed and successfully integrated into an apple harvesting robot, such as the one described in chapter 1, to effectively supply suction power to multiple robotic picking arms. Other Non-conventional Fruit Gripping Technologies Outside of the agricultural research sphere, extensive research has been conducted on robotic gripping end effectors. Many of these non-conventional gripping concepts take inspiration from biological mechanisms, hence they could be potentially useful to solve a biologically driven problem like apple harvesting. It is known that the gripping capability of a vacuum suction cup depends on the time taken to induce negative pressure in the void between the cup and the target object, as well as the presence of air leaks (Bamotra, Walia, Prituja, Ren, & Ieee et al., 2018). Therefore, extensive research has been conducted with soft materials such as silicone, hydrogels, and elastic rubbers to mimic biological materials and gently handle fragile materials like apples. The flexibility of these materials makes them optimal for suction cups as they can adaptively fit to objects that vary in size and structure. Additionally, using these types of materials allows the cups to be fabricated by casting into 3-D printed molds, making it easy to experiment with various structures (Bamotra et al., 2018). Fujita et al. (2018) developed a “universal vacuum gripper” (UVG) and compared its performance to an “octopus-like” sucker. The “octopus sucker” used a vacuum pad with a membrane, using vacuum pressure to seal the space between the target and the membrane, therefore gripping the object (Fujita et al., 2018). The UVG design uses jamming transition of 22 granular materials like coffee grounds inside a balloon to grip objects that have non-uniform surfaces. The gripper is based on the ability of these granular materials to quickly transition between “jammed” rigid states in the presence of vacuum and “un-jammed” deformable states when vacuum pressure is relieved, allowing the gripper to adapt to objects of various shapes and sizes (Fujita et al., 2018). Testing showed that the UVG was able to maintain negative vacuum pressure and apply sufficient suction flow even if air leakage occurred. However, the biggest issue was that the fragile balloon would easily break as a result of the suction behavior. It was also found that the UVG was best suited for objects with square edges and corners, as the gripping is mostly based on friction between the membrane and the object (Fujita et al., 2018). While the study showed interesting concepts which may be useful for the design of an apple picking end effector, neither evaluated gripper is sufficient for fresh market apple picking without modifications. Soft robotics is a quickly emerging area of study that expands upon the previously mentioned ideas of taking inspiration from biological mechanisms to solve problems in modern robotics. Soft robots are described as those that can safely interact and conform to fragile and complex environments better than traditional, rigid bodied robotics (Tawk, Panhuis, Spinks, & Alici et al., 2018). This type of robotics technology is in infancy stages of development compared to traditional robotics (Whitesides, 2018). Soft robotics involves the integration of unconventional materials in robotic systems due to advantageous mechanical properties such as body shapes, elasticity, softness, etc. (Iida & Laschi et al., 2011). In this way, robotic system may mimic the biological suction functionalities of octopus and squid, which are able to grip a wide range of materials with their tentacles (Kessens & Desai et al., 2010). Since the issues 23 associated with automated fruit harvesting are largely biologically driven, biologically inspired robotic systems have great potential for solving some of these agricultural issues. Soft actuators play a key role in soft robotics; they are able to produce extremely smooth and life-like movements making them generally safer for operating in close contact with humans or fragile environments compared to traditional robots known for generating high velocities and high forces (Ainla, Verma, Yang, & Whitesides et al., 2017; Lee & Rodrigue et al., 2019; Miller & Wicks et al., 2018; Robertson & Paik et al., 2017; Tawk et al., 2018). Additionally, these soft robots show potential over hard robots to be adaptable to varying shapes and sizes of targets, as well as being more compatible with dynamic work environments (Kalisky et al., 2017; Whitesides, 2018). Different soft gripping end effectors have been developed recently, one of which is a “Starfish gripper” (Whitesides, 2018). This gripper uses tentacles inspired by the tongues of a chameleon. These tentacles wrap around a target object in a manner similar to a starfish attaching to a rounded surface (Whitesides, 2018). Many of these soft end effectors are still in a research stage and are not ready for agricultural applications, such as robotic apple harvesting. However, these are exciting concepts with great potential for agricultural applications. The integration of soft robotic concepts into an end effector design could help address some of the challenges presented by traditional “hard” end effector designs. A research team from Washington State University has developed a 3D-printed soft pneumatic actuator end effector for fresh market apple picking (Hohimer et al., 2019). The design consists of three soft pneumatic actuator fingers mounted on a flexible palm. The design is similar to the “starfish gripper” previously described, with the soft fingers wrapping around 24 the target fruit to grip for picking. The relatively soft material (Shore 85A) was selected based on durability to avoid damage from collisions with the canopy structures. The end effector was found to generate a pulling force of ~49.5 N, sufficient to detach most apples by a straight pull picking motion. This design utilizes promising soft robotic concepts but effectively acts as a conventional finger-type end effector. Therefore, it was found to experience several of the same difficulties as other conventional, non-vacuum-based systems, including issues isolating and detaching fruit in clusters. FFRobotics, a commercial startup company based in Israel, developed a similar soft pneumatic-actuator finger-based end effector (Brown & Sukkarieh et al., 2021). This design utilized a similar soft material and actuated three fingers to grip fruit. Like the Washington State study, the soft pneumatic actuator was successful in avoiding damage to the fruit and to the end effector itself due to collisions with nearby canopy structures. A significant disadvantage shown in these soft finger designs is the inability to sufficiently grip the fruit for a “twist-and-pull” picking method. These end effectors must use a “straight pull” picking method, which is not desirable, as it can cause damage to the tree branches and leave harvested apples without stems. Mechanisms of Fruit Detachment To design an effective end effector, one needs to understand the picking motions exerted by the end effector. The force required to detach fruit may vary significantly based on the technique used to detach the fruit. An experienced field worker will pick apples by using fingers to grip the cheek of the fruit, and then twisting the fruit around the stem-calyx axis while pulling away from the branch. This twisting weakens the stem, thus decreasing the force required to detach the fruit (Bu, Hu, Chen, Sugirbay, & Chen et al., 2020). This has several benefits for automated harvesting as a decreased required force reduces the risk of bruising the fruit during 25 picking, reduces the effects of branch reverberations knocking nearby fruit off the tree, and also reduces the risk of damaging the tree branches. Unfortunately, this twisting motion can be difficult to achieve for robotic harvesting systems due to the requirement of gripping the bottom of the fruit. Robotic harvesters sitting adjacent to the canopy tend to approach and grip the cheek of the fruit, making this twisting action very difficult to achieve. This concept was extensively studied by Li et al. (2016), using an instrumentation glove to track optimal picking forces/motions that could be integrated into an end effector. They reported that the fruit hanging from a tree branch can be viewed as a simple system in equilibrium under the effects of gravity, elastic force and elastic torque of the stem-branch joint, tension force, and a bending moment exerted by an end effector (Li, Karkee, Zhang, Xiao, & Feng, et al., 2016). Based on the assumptions listed above, Li et al (2016) studied the effectiveness of three picking techniques, mainly a “pendulum action”, and “bend and pull action” (figure 2.1). In the “pendulum action” method, the fruit stem is bent around the stem-branch joint, exerting a bending moment on the stem in the absence of tension. Conversely, the “bend and pull” method causes apples to be pulled along the axial direction of the stem, exerting a tension force while the stem is bent around the stem-branch joint, denoting a detachment angle as the critical bending angle at which the fruit is detached from the branch. 26 Figure 2.1: Three distinct picking motions. “Twist and pull” method (left), “bend and pull” (center), and “pendulum” motion (right) It is important to note that this study was conducted using finger-type gripping, not a vacuum gripping end effector. Results showed some differences between picking orientations. The study concluded that due to smaller required detachment forces, the “bend and pull” method shows greater promise than the “pendulum action” method for minimizing bruise potential. It should be mentioned that most of robotic harvesting systems, including our vacuum-based system, use the “bend and pull” method to detach fruit. However, since the term “twisting” is commonly used in literature, the term “twisting” is used, instead of “bending”, when discussing fruit picking tests (Chapter 3) and the robotic system, although the fruit detachment process actually involves a bending motion. Summary The end effector design is one of the main challenges in developing most robotic harvesting systems. Many end effector designs have been attempted with varying degrees of success as well as a range of causes for failure. Hence, further efforts should be made on designing novel, more effective end effectors for robotic apple harvesting systems. The use of 27 vacuum-based concepts for robotic harvesting has potential to simplify the design of the end effector, reduce fruit bruising and enable effective picking of fruit from clusters. The new end effectors to be designed and tested for this study should consider and incorporate the following characteristics:  Utilize a “Twist and Pull” picking method with proper fruit picking orientations to reduce required picking forces and minimize damage to the fruit, stems, and branches, while maximizing picking efficiency and effectiveness.  Simple, soft, and compact (with small projected area), so as to facilitate the robot to navigate into the canopies and pick fruit from clusters, while reducing damage to the fruit and canopies.  Hold the detached apple, and subsequently release the apple into receiving conveyors in a timely manner.  Work for different canopy architectures, as well as different apple varieties with varying size and shape. 28 CHAPTER 3: MEASURMENT AND ANALYSIS OF APPLE PICKING FORCES Introduction When picking an apple from the tree, an experienced worker uses his/her palm to hold the bottom section (the calyx end) and fingers to grip the cheek of the fruit and then twist the fruit to weaken its connection between the stem and the tree branch (Davidson, Silwal, Karkee, Mo, & Zhang et al., 2016). This twisting action coupled with a pulling action enables fruit detachment with less force, which has been termed the “twist-and-pull” method as previously described. Application of pulling forces alone to detach fruit should be avoided because this action could cause damage to the tree branches and buds, which are critical for future fruit bearing. Thus, in designing a robotic harvester, twisting and pulling must be used together to detach fruit from trees. To properly evaluate the capabilities of the robotic harvesting system to successfully pick apples, it is critical to understand the required forces. Determining the required force to detach mature apples is also critical to the design of a simulated orchard environment that is used for testing and evaluating the performance of automated harvesting systems in an indoor environment. By determining the required force to detach apples by a “straight-pull” and a “twist-and-pull” picking action, baseline performance requirements may also be established for vacuum-based end effectors and the vacuum system itself. This chapter describes the methods and procedures used to measure the pulling and twisting forces to detach apples and reports the data collected for three varieties of apples from an orchard in Michigan State University’s Horticultural Teaching and Research Center in Holt, MI. Based on the measured data, the critical force and torque with the corresponding vacuum 29 pressure needed to detach 95% of apples were estimated for the straight pull and twisting methods using the thin-foam end effector of the robotic system described in chapter 1. Materials and Methods Two apple picking force experiments, i.e., pulling and twisting, were conducted in Michigan State University’s Horticultural Teaching and Research Center at Holt, MI during the 2020 harvest season. The experiments were conducted on three different varieties of apples (i.e., ‘Gala’, ‘Golden Delicious’, and Red Delicious’) on 9/30/20 and 10/7/20. Each experiment was conducted on 30 fruit each of ‘Gala’, ‘Golden Delicious’ and ‘Red Delicious’ varieties (i.e., total 60 apples tested per variety per day and 120 per variety for two days). It should be noted that the pulling and twisting experiments were conducted on traditional, un-trained tree canopies which are quite different from the high-density, structured orchard that are more popular nowadays in U.S. apple growing regions for commercial production. Pulling and Twisting Experiments A 50 N (10 lb.) capacity digital force gauge (Wagner Instruments, Greenwich, CT) was used to determine the force required to detach mature Gala, Golden Delicious, and Red Delicious apples. For the pulling experiment, a cloth face mask was wrapped around the apple and held by the hooked end of the digital force gauge to measure the tension required to detach the fruit when pulling straight back from the fruit, perpendicular to the vertical plane of the canopy (figure 3.1). It is important to note that, due to manual operation, the detachment angle was not exactly perpendicular to this vertical plane. The exact angle of detachment was not measured or considered for this study. The speed of detachment was also not measured or considered for this study. 30 Figure 3.1: Attachment of an apple to the digital force gauge for evaluating a “straight-pull” picking action As discussed earlier in the chapter, an experienced worker would twist the fruit along the stem-calyx axis (or the vertical direction when the fruit hangs downward). However, it would present significantly more challenges in designing a robot to perform such twist actions. In most harvesting robot designs, including the new vacuum-based robotic harvester described in chapter 1, the robot’s vacuum tube approaches the target fruit parallel to the ground or with a small angle, as shown in figure 3.2. This approach angle results in the fruit being held on its cheek, for most cases, by the end effector. Once the fruit is gripped by the end effector, the robot then rotates the fruit in a direction horizontal to the stem-calyx axis, which is different from that by a human picker. Hence, it may not be appropriate or accurate to call the current picking action as a “twist-and-pull”, which is referred to the picking motion in which the fruit is being twisted around the stem-calyx axis (Li et al., 2016). This twisting of the stem is advantageous because it would effectively weaken the stem-branch junction and thus reduce the required force to detach fruit. 31 Figure 3.2: Image describing the robot’s approach to the cheek of the fruit (left) and twisting motion for detachment (right) For the current vacuum-based robotic system to accomplish a true “twist-and-pull” picking action, the end effector would need to attach the bottom portion of the fruit (opposite stem), as shown in figure 3.3, which requires a more complicated design for the robot and its end effector. There is also an increased risk of colliding with branches causing damage to the tree or to the robotic system itself when approaching the bottom of fruit. It is thus preferrable from the design perspective that the robotic system approaches the cheek of fruit to avoid canopy occlusions. Figure 3.3: Example of the end effector (the bellow design) attaching to the bottom of the target fruit. This image was taken in the indoor simulated orchard environment described in Chapter 5 32 Therefore, rather than twisting, the current rotation mechanism is simply applying a torque in a horizontal direction to the stem, which is expected to weaken the stem to ease the picking process but would not be as effective as the actual twisting action performed by a human picker. The robot achieves a picking motion more comparable to a pendulum motion. Nonetheless, this pendulum motion will be described as “twisting” throughout this study. To simulate the “twist” picking action of the robotic harvesting system, a simple hand- held rotation mechanism was constructed to evaluate the twisting force (torque) required to detach apples. The free-rotating bearing houses a 16 mm (0.625ʺ) diameter steel rod. One end of the steel rod is equipped with two needles used to puncture and grip the fruit for detachment. A string is wound on the opposite end of the rod. This string is attached to the hooked end of the digital force gauge as shown in figure 3.4. As the force gauge is pulled perpendicularly away from the rotation mechanism, it applies a rotating force to the apple, thus causing the fruit to be detached from the tree. This rotation mechanism would thus operate in an identical or similar manner to the robotic system when an apple is attached to the end effector on its cheek, as shown in figure 3.2(b). Figure 3.4: The rotation mechanism used to hold the fruit and measure the force required to detach with a twisting motion 33 The digital force gauge measures tension at a frequency of 100 Hz with an error of ± 0.15 N (0.03 lbs.), recording the peak tension prior to the apple detachment. This peak tension represents the force required to detach the mature apples. Computer software “MESUR Lite by Mark-10”, provided by the manufacturer (Wagner Instruments, Greenwich, CT), was used to record the continuous tension data. This data was exported to Microsoft Excel for further analysis. The forces measured in the twisting experiment were converted into torques, so that they could be used in the design of the vacuum-based robotic harvester. Data Analysis Torque (𝑇 , N-m) was derived using the net tangential force measured by the digital force gauge to detach the fruit (𝑃 , N) subtracted by the initial force (P0, N) needed to rotate the mechanism with no fruit being attached and then multiplied by the radius of the steel rod used to grip the fruit (r = 0.008 m or 0.313ʺ): 𝑇 = (𝑃 − 𝑃 ) × 𝑟 (2) Mean pulling forces and torques and standard deviations were calculated for apples measured in each week as well as their combined data. Analysis of variance (ANOVA) was then performed to determine if there were statistically significant differences in the pulling or twisting force for the three varieties. Furthermore, the critical straight pulling force (𝑃 ) and torque (𝑇 ) required to detach 95% or higher of apples were also estimated. In this study, the target goal for designing a new end effector or an improved robotic system was set at 95% successful picking rate. Therefore, the lowest 95% of the normal distribution will represent the required picking forces to detach fruit. Hence, the measured mean pulling force from the field experiment 34 was considered as the picking force (𝑃 ) to detach 50% of apples with a straight pull picking action. As this is biological data, a normal distribution may be assumed for the required picking forces to detach apples, despite the fact that the collected data did not exactly follow a normal distribution, due to the limited number of apples picked during the experiment. Under the assumption of a normal sample distribution, by the empirical rule, 95.4% of the population will fall within two standard deviations of the mean. This 95.4% is centered around the mean (50th percentile, 𝑃 ). Thus, this logic would imply that the highest and lowest 2.3% of apples would not be detached. It is assumed that the lowest 2.3% of apples would be most easily detached. To determine the critical force needed to detach apples in the lowest 95% of the normal distribution, a z-score method is needed. Based on a significance of 0.05, a z-score of 1.6 will give a cumulative percentage of 95% of a normal distribution. Plugging into the following z-score equation will enable us to obtain the critical pulling force. 𝑃 −𝑃 𝑧= 𝜎 or 𝑃 = 𝑃 + 1.6 × 𝜎 (3) where the standard error 𝜎 was approximated to be the sample standard deviation. After the critical pulling force, 𝑃 , has been calculated, the corresponding critical vacuum pressure 𝑞 , (kPa) for the vacuum-based robotic system described in chapter 1 was estimated based on the inner diameter of the vacuum tube using the following equation: 35 𝑃 𝑞 , =𝑞 − 𝐴 × 1000 (4) where 𝑞 is the atmosphere pressure (=101.325 kPa), 𝑃 is the critical pulling force (N) for detaching 95% of apples and A is the cross-sectional area of the vacuum tube (m 2) with an inner diameter of 44.5 mm. For the robotic system described in chapter 1, a thin rubber foam of less than 2.5 mm thick was attached to the vacuum tube. Since the combined thickness of the foam and tube wall was only ~ 2 mm, the twisting torque or moment generated by the vacuum system can be approximately calculated by the following equation: 𝑇 = 𝐹 × 𝑅 × 𝜑 (5) or, 𝑇 = ∆𝑞 × 𝐴 × 𝑅 × 𝜑 × 1000 = (𝑞 − 𝑞) × 𝜋𝑅 × 𝜑 × 1000 (6) where 𝐹 = ∆𝑞 × 𝐴 is the suction force (N), 𝑞 = (𝑞 − 𝑞) is the pressure differential (kPa) generated by the vacuum system, in which 𝑞 is the actual pressure (kPa) in the vacuum tube after the fruit attaches to it, A is the cross-sectional area of the inner tube, R is the effective radius (m) of the end effector contacting the fruit and 𝜑 is the static coefficient of friction between the apple 36 and the end effector. Equation (6) shows that torque is directly proportional of the pressure differential, the tube’s effective radius to power of 3, and the static coefficient of friction between the apple and the end effector. Increasing the tube size would increase the total suction force and torque under the same pressure differential. On the other hand, increasing the pressure differential will also lead to an increase in the suction force and torque. However, selection of tube size is also determined by fruit size and the vacuum power. After the tube size and vacuum system are determined, the torque that can be generated by the system will be dependent on the pressure differential, the effective contact radius and material of the end effector. To generate high pressure differential, the end effector must be able to provide effective sealing of the apple that is attaching to it. A larger contacting radius or area and higher coefficient of friction are also critical to generating higher torque. Using the same approach as that for calculating the critical pulling force [equation (3)], the critical torque 𝑇 to detach 95% of apples using the twisting action for the robotic system was calculated from the mean torque and standard error/deviation based on and the z-score equation. The critical suction force 𝐹 acting on the fruit that is attached to the end effector can be calculated from equation (7), or 𝑇 𝐹 = 𝑅× 𝜑 (7) 37 Likewise, the critical vacuum pressure 𝑞 , can be calculated from equation (8): 𝑇 𝑞 , =𝑞 − 𝜋𝑅 × 𝜑 × 1000 (8) Puchalski et al. (2003) reported that the coefficient of friction for “McLemore” apples on rubber ranged between 0.45-0.65. In this study, the median value of 𝜑 = 0.55 was used for calculating critical suction force, torque and vacuum pressure. These critical values provide important guidance in the design and evaluation of the robot’s vacuum system. Results and Discussion Table 3.1 summarizes results from the straight picking force (P) evaluation. The table shows the arithmetic mean of the peak tension recorded by the handheld digital force gauge prior to apple detachment, for each experiment denoted as “day 1” (9/30/20) and “day 2” (10/7/20), respectively, as well as the mean of total data for the two-day experiments. 38 TABLE 3.1: Summary of the mean measured forces (standard deviations) required to detach fruit by a “straight pull” picking action* Pulling Force (N) Fruit Variety Day 1 Day 2 Combined Gala 15.9 18.9 17.4 (9.0) Golden Delicious 16.6 14.6 15.6 (6.9) Red Delicious 15.2 16.4 15.8 (6.0) Combined 16.3 (7.5) * Mean values in the table for each week represent the average of 30 measurements ANOVA showed that the differences between the three apple varieties are not statistically significant (p = 0.33). This is important when considering these values as baseline performance requirements for the vacuum-based robotic system. The robotic harvesting system is intended to work effectively on any fruit variety. Hence, the averages of the three mean values of straight pulling force (16.3 N) and the three standard deviations (7.5 N) were used as the pulling force (𝑃 ) and standard deviation () to detach 50% of apples with a straight pull picking action. These values were further used for calculating the critical picking force 𝑃 for 95% of apples. It was estimated that a critical pulling force of Pc = 28.3 N is needed to achieve the 95% picking goal with a straight pull picking action. This straight pulling force can be achieved by the robot’s return to home position. It is, however, important to note that this straight pull picking action should be avoided, and it will only be required if the “twisting” picking action by the robot’s rotation mechanism fails. It is advantageous to pick fruit via the “twisting” picking 39 motion because it reduces the risk of damage to the tree canopy structures, neighboring fruit, and the target fruit itself. Table 3.2 shows the average torque values calculated for both “day 1” and “day 2” experiments, while Table 3.3 presents the single-factor ANOVA results for each variety. Results of the ANOVA shown in Table 3.3 confirm that there is not a statistically significant difference between the required torque to detach the three varieties of apples tested (p > α = 0.05). Thus, the average of the three tested varieties were used for estimating a critical torque (Tc) requirement for the robotic system to achieve the goal of picking 95% of fruit. TABLE 3.2: Summary of the mean torques (standard deviations) calculated from the mean forces measured to detach apples by the twisting picking action* Fruit Variety Gala Golden Red Total (all Delicious Delicious Varieties) Torque (N-m) 0.155 0.179 0.168 0.167 (0.050) (0.059) (0.056) (0.056) * Mean values in the table represent the average of 60 measurements 40 TABLE 3.3: Results of single-factor ANOVA comparing measured torque values for each tested variety Single-Factor ANOVA SUMMARY Groups Count Sum Average Variance Gala 60 9.282 0.155 0.002 Golden Delicious 60 10.712 0.179 0.004 Red Delicious 60 10.057 0.168 0.003 ANOVA P- Source of Variation SS df MS F value F crit Between Groups 0.017 2 0.009 2.817 0.062 3.047 Within Groups 0.536 177 0.003 Total 0.553 179 The mean torque of the three apple varieties was calculated to be 0.167 N-m with a standard deviation of 0.056 N-m (Table 3.2). It was found that the critical torque (Tc) of 0.257 N- m is needed to detach 95% of fruit by action of the rotation mechanism alone. This critical torque should be used as a guidance in the design and evaluation of new end effectors and the vacuum system. Table 3.4 compares the equivalent critical force, torque and vacuum pressure values that would be required to detach 95% of apples when pulling and twisting actions are used for the thin-foam end effector of the robot’s vacuum system described in chapter 1. First, the critical suction force and equivalent torque needed for pulling fruit (28.30 N and 0.346 N-m, respectively) is 35% higher than the one needed for twisting (21.00 N and 0.257 N-m). This means that if the robot design were based on the pulling mechanism, it would require lower 41 vacuum pressure (83.13 kPa) and thus more vacuum power. Twisting would be gentler or cause less damage to fruit, trees and branches, not just because this pattern of picking action is gentler to trees and branches, but also because it requires less suction force, compared to pulling. Hence in designing a robotic system, a rotation mechanism for twisting fruit is required, so as to reduce potential damage to fruit, branches and trees. Moreover, it may be advantageous for the robotic system to utilize a “twist-and-pull” picking action, as suggested by Li et al. (2016). If the target fruit fails to be detached by twisting, the subsequent pulling action may still enable to detach the fruit, because the stem may have been weakened during twisting, thus reducing the pulling force required to detach the fruit, which would lead to overall higher successful picking rates. Finally, it should be mentioned that the critical vacuum pressure calculated in table 3.4 is for the thin- foam end effector only. The critical vacuum pressure will be different, depending on the design of a new end effector. TABLE 3.4: The critical force, torque, and vacuum pressure needed for the current robotic system with the thin-foam end effector to detach 95% of apples based on the measured pulling and twisting forces for three apple varieties* Critical Value Pulling Action Twisting Action Force (N) 28.30 21.00** Torque (N-m) 0.346** 0.257 Vacuum Pressure (kPa) 83.13 87.82 * The critical pulling force and twisting torque were estimated based on the mean values measured from the pulling and twisting study, respectively, while the remaining values in the table were derived from these two values using equations (5)-(8) 42 ** The critical torque for pulling and force for twisting were calculated using the coefficient of friction value of 0.55 for apples on a silicone material Conclusions The mean forces and torques for detaching ‘Gala’, ‘Golden Delicious’ and ‘Red Delicious’ apples using straight pulling and twisting actions were not significantly different between each variety. Based on the assumption that the force required to pick all apples follows a normal probability distribution, a critical pulling force of 28.30 N is needed to detach 95% of apples by using a straight pulling action, while a critical torque of 0.257 N-m is needed when using a twisting action. The equivalent force calculated based on the critical torque of 0.257 N-m by using twisting with the current robot system to detach 95% of apples was calculated to be 21.00 N, which is 35% lower than that for pulling. This study has confirmed that compared to pulling, twisting would be gentler or cause less damage to fruit, branches and/or trees, and therefore should be a preferred mode of detaching fruit. Hence, a new end effector, along with the vacuum system, should be such designed that the vacuum pressure drop inside the vacuum tube due to fruit attachment to the end effector need be able to generate a torque of at least 0.257 N-m in order for the rotation mechanism to achieve 95% fruit detachment rate. The findings from this study will be further compared and validated through the laboratory and field evaluation of new end effectors, described in chapter 4. 43 CHAPTER 4: END EFFECTOR DESIGN AND EVALUATION Introduction The end effector is responsible for gripping and detaching fruit from trees, and it is a critical component for all robotic fruit harvesting systems. In a conventional robotic harvester, the end effector needs to perform complex actions like human hands (i.e., opening, closing, and rotating/twisting). These end effectors are ‘active’ end effectors, to differentiate from ‘passive’ end effectors like ones described in this chapter, which do not have locomotive components. Since these active end effectors are generally quite complex and bulky in design, they may have difficulty navigating complex canopy structures and avoiding collisions with branches and even other fruit, which may cause damage to the canopy or the end effector itself. Active end effectors also struggle isolating and picking fruit from clusters, resulting in a lower successful picking rate. Furthermore, a complex end effector would require more time to grip, detach, and subsequently drop the fruit to a receiving area, which means a longer cycle time and thus lower harvesting efficiency. A robotic harvester with low harvesting efficiency is less likely to be adopted by the apple industry. One key advantage of using a small vacuum tube to grip and detach fruit is that the end effector is simple, compact, and passive with no locomotive components. This is likely to improve the fruit picking performance of the robot while minimizing damage to target fruit and improving durability of the end effector itself. As discussed in Chapter 2, in designing a vacuum- based robotic harvester, three critical design parameters must be considered: air flow rate, suction force, and cycle time. Suction force is dependent on the configuration of the end effector. The more effective the end effector is at generating vacuum sealing, a greater suction force will be exerted on the fruit, thus lowering the vacuum requirement for effective picking. A well- designed end effector must be able to provide sufficient suction force to grip and detach the fruit, 44 while avoiding bruising damage to the harvested fruit. In addition, after the fruit is picked, the vacuum system should be able to quickly release or drop the fruit at a predetermined destination to maintain overall harvesting efficiency. A major shortcoming of the first version robotic harvesting system was the absence of an appropriate end effector. The system simply utilized a thin adhesive foam wrapped around the rigid end of the 44.5 mm diameter vacuum tube. The thin foam was intended to cushion the fruit, reducing bruising from the vacuum grip. The vacuum flow supplied by the 5.5 hp wet/dry vacuum system showed promising results in the ability to isolate individual fruit, attract nearby fruit to the vacuum head, and navigate the complex canopy structures of young apple canopies. Unfortunately, there were several issues with the over-simplified end effector. The cushioning effect was insufficient in minimizing bruising, and the rigid nature of the aluminum tube resulted in a high presence of air leaks during vacuum attachment. This was especially apparent for irregularly shaped and/or small fruit, and when the vacuum tube was failing to grip the cheek of the fruit due to a bad approach angle. Figure 1.5 highlights the air leaks present during apple gripping by the first-generation robot’s end effector. This imperfect vacuum seal significantly reduced the harvesting efficiency of the system. The presence of air leaks is a major factor in suction technology not being widely adopted for robotic “pick-and-place” applications (Nakamoto, Ohtake, Komoda, Sugahara, & Ogawa et al., 2018). Often, although fruit were properly attracted and attached to the vacuum head, the imperfect vacuum seal resulted in a gripping force which was too low to successfully detach apples from the tree with either a “twist-and-pull” motion or a straight picking motion. This insufficient harvesting efficiency made apparent the need for an improved end effector design for the second generation of the harvesting robot. Based on the concepts of soft robotics, it was 45 hypothesized that by augmenting the rigid end of the vacuum tube with a more soft and conformable end effector design, air leaks will be greatly reduced between the vacuum tube and target fruit. This improved vacuum seal would result in a lower vacuum pressure within the vacuum tube and a greater gripping force applied to the fruit, leading to increased harvesting efficiency. Design of End Effectors Prior to the design and fabrication of new end effectors, an extensive literature review was conducted of soft robotic systems and soft end effectors. Suction cups are widely used with pneumatic systems for industrial applications, such as lifting parts or products. As such, more than 10 types of suction cups were acquired from commercial manufacturers and evaluated for their suitability as an end effector for our robotic harvesting system. Through preliminary evaluations, it was found that these commercial suction cups worked well for lifting or attracting objects with a large flat surface or of round shape. However, since apples vary greatly in size and shape, these commercial suction cups were unable to provide sufficient vacuum seal or would undergo excessive deformation when a relatively large twisting force was applied to the object due to inappropriate cup configurations or the use of a material that was too soft. It was thus decided that a new end effector needed to be designed and fabricated for the robotic harvesting system. In the design of new end effectors, several important qualitative and quantitative factors were considered. First, the material selected for an end effector must have proper physical properties, including hardness and deformability. Proper hardness is critical to minimizing fruit bruising. Appropriate deformability is also required such that the end effector can be easily conformable to fruit of different shapes and sizes for greater vacuum sealing effects. Moreover, 46 the ideal end effector should not undergo excessive deformation when it is subjected to a twisting action by the vacuum tube to detach the fruit from a branch. Second, the configuration of an end effector should be relatively simple, easy to fabricate and able to accommodate different sizes and shapes of fruit. In preliminary iterations, several uniform-material suction- cup-type end effectors were designed, prototyped, and evaluated in laboratory. Due to its conformable properties, silicone rubbers are widely utilized for both soft robotics and industrial suction applications. The tested silicone rubbers ranged in shore hardness from 10A-45A. The non-uniform shape of apples requires this type of conformable material to generate a sufficient vacuum seal to detach the fruit. These soft materials would also provide sufficient cushioning to minimize bruising on the fruit during the picking process which is critical to maintaining crop value. Based on the recommendations from Dr. Xiaobo Tan of the Electrical and Computer Engineering Department at Michigan State University and his team member Hongyang Shi, and after several trials with different silicone materials, a silicone rubber called Dragon Skin 20 TM was selected. Table 4.1 summarizes the material properties of Dragon Skin TM 20 pourable silicone rubber (Smooth-on, Inc., Macungie, PA): TABLE 4.1: Key physical properties of Dragon SkinTM 20 silicone rubber used for end effector fabrication Shore Hardness 20 A Tensile Strength 3792 kPa 100% Modulus 338 kPa Elongation at Break 620% 47 Preliminary laboratory evaluations showed that the physical properties listed above provide a proper balance between the softness that is needed for the end effector to conform with the irregular contours of fruit and elasticity that is required to grip/twist to detach the fruit. This balance is critical for biological applications of these silicone rubbers (Auroy, Duchatelard, Zmantar, & Hennequin et al., 1996). After consideration of different design alternatives, three final end effectors were fabricated and evaluated for this study. These design alternatives are shown in figure 4.1, denoted as “Straight”, “Bellow”, and “Curved”, respectively. These designs were refined and optimized through the gripping force experiments conducted in laboratory and finally verified with field experiments. Figure 4.1: Front view of three silicone end effector designs generated by Fusion 360 software (Autodesk, San Rafael, CA) Each design alternative includes a base cylindrical section (40 mm length) to slide onto the end of the 44.5 mm diameter vacuum tube. The inner and outer diameters of this cylindrical section are 40 mm and 55 mm, respectively, for each of the designs. A narrow slit is cut on the bottom side of the cylindrical base, such that the silicone cups may slide onto the end of the 44.5 mm diameter vacuum tube, as shown in figures 4.2 and 4.3. 48 Figure 4.2: Contour sketch of each end effector design describing the major dimensions (same for all three designs) and distinction between “inner-lip” and “outer-lip” (left- Straight, center- Bellow, right- Curved) Figure 4.3: The “bellow” end effector design alternative installed on the aluminum vacuum tube The “Straight” design was first chosen, with the goal of providing two stages of vacuum sealing. An “outer lip” protruding from the top edge of the cylindrical cup provides the first stage of vacuum sealing on the fruit, as shown in Figure 4.4. The “outer lip” (varying diameter) is longer 49 than the inner lip (27.7 mm diameter), such that it can more effectively conform to the irregular contours of the fruit. The inner lip has a smaller diameter than the outer lip based on the rounded contour of the fruit. Figure 4.4: Wire frame view of three end effector designs (left- straight, center- bellow, right- curved) annotated with key dimensions of the “inner-lip” and “outer-lip” for each design In an ideal case of gripping the fruit by its cheek, both the inner (27.7 mm) and outer (51 mm) lips of the “Straight” end effector will be fully engaged and conformed to the contours of the apple. The “Bellow” and “Curved” designs were created as modified versions of the “Straight” design. Each design includes the same “inner lip” and “outer lip” concept. Each design is named after the geometry of its “outer lip”. While the diameter of the cylindrical base and “inner lip” for each design is identical, the dimensions of the “outer lip” section differs for the ‘curved’ design, as the intention of this design is to enable a larger surface contact between the end effector and the fruit. Bellowed suction cups are widely used in industrial suction applications for gripping irregularly shaped objects. This geometry is advantageous for allowing local compression of the suction cup around its diameter. This design is conducive to creating a more perfect vacuum seal 50 on apples, as apples are not spherical. This concept is also beneficial in cases when the fruit is not attached to the end effector perfectly on the cheek. When gripping the apple near the stem or on the bottom (calyx end), the contours of the fruit tend to be more profound. These more profound geometries provide a greater challenge for gripping the fruit. The “Curved” design was developed based on the idea that more contact area between the end effector and the target fruit would provide a greater gripping force. The longer, curved outer-lip will provide more surface contact with the fruit than the other two designs. In this way, each design alternative is intended to have different potential benefits; thus, they were fabricated and evaluated, compared to the original thin foam end effector. Fabrication Process Each design alternative was fabricated using a casting process. First, the designs were drawn using Fusion360 (Autodesk, San Rafael, CA). These drawings could be easily modified to allow for rapid prototyping and design optimization for each end effector. Front views of these Fusion360 drawings are shown in figure 4.1. These drawings were exported from Fusion360, and 3D printed using an Ultimaker S5 printer (Ultimaker, Utrecht, The Netherlands). The printer used a hard plastic filament to build the exact geometries of the Fusion360 drawings. The hard plastic versions of the end effectors are called “masters”, which were used for creating molds. As shown in figure 4.5, the “masters” were submerged in pourable silicone rubber, different in its physical properties than the Dragon Skin 20 used for the final end effector, to create reusable molds for each design. 51 Figure 4.5: 3D printed master submerged in silicone creating a flexible mold The soft, flexible nature of these silicone rubbers is ideal for making molds, as they make it easier to remove final products from the mold compared to more rigid materials. Additionally, the silicone rubber will not adhere to any other material but itself, allowing the final silicone product to easily be removed from the soft silicone mold. Once cured, two priming materials were applied to the molds before the Dragon Skin 20 was poured to form the final end effector. “InhibitX” (Smooth-On, Inc., Macungie, PA) was first applied to the mold. This product is designed to reduce cure inhibition for platinum-based silicone rubbers. Next, “Ease-release” (Smooth-On, Inc., Macungie, PA) was applied to the mold to allow for easier removal of the final product from the mold. These two priming steps were optional but made the process easier and more reliable. Once the Dragon Skin 20 had fully cured in the mold, the final silicone end effector was removed from the mold. As described, a slit was then cut on the bottom of the cylindrical base section of the end effector to allow the suction cup to slide onto the end of the aluminum vacuum 52 tube. The final product for each design, alongside the hard plastic “masters” are shown in figure 4.6. Figure 4.6: Final fabricated silicone end effector designs (white) alongside their hard plastic 3D printed masters (black) Laboratory Evaluation of End Effectors Preliminary laboratory tests were conducted using two different-size artificial apples to evaluate the gripping forces of the control and new end effector designs and the suction force characteristics of the vacuum system. Use of artificial apples, instead of real apples, would provide more consistent evaluation of the three new silicone end effectors and the original thin- foam end effector, since these apples can be used repeatedly without bruising/degradation. The results of this experiment were first compared with the results of the picking force experiment described in chapter 3. Next, the equivalent torque for the straight end effector was estimated from the measured gripping force, which was then compared with the critical torque determined from the orchard’s twisting study described in chapter 3. Finally, the vacuum pressure based on 53 the gripping experiment of the straight end effector was calculated and compared with the critical vacuum pressure reported in chapter 3. Gripping Force Measurements A linear slide experimental setup was constructed for use in this gripping force evaluation. The experimental setup houses the vacuum tube, which is supplied with vacuum by the 5.5 hp wet/dry vacuum used for the robotic system, on a structure which can freely slide linearly along the length of the setup. The same hand-held digital force gauge used to evaluate the required force to detach real apples, as described in chapter 3, is mounted to the setup as shown in figure 4.7. A looped string is imbedded into artificial foam apples such that the artificial apples attach to the hooked end of the digital force gauge on one side and to the vacuum head/end effector on the other side (figure 4.7). When the linear slide is manually pulled away from the digital force gauge, tension is generated on the force gauge due to the holding force exerted on the artificial apple by the vacuum and end effector. At a certain level of tension, the apple will detach from the vacuum head/end effector. Thus, the peak tension recorded by the digital force gauge indicates the holding/gripping force exerted on the fruit by the vacuum system. Theoretically, this gripping force is the maximum straight pulling force that the end effector could exert for detaching an apple. It should be mentioned that the speed of pulling was not considered for this study. 54 Figure 4.7: The experimental setup to evaluate the gripping force of each end effector design alternative. The original thin-foam end effector is shown in this image Two different artificial apples, which were made of plastic material, with their shape and size being similar to typical real apples, were used in the experiment. The artificial apple denoted as “large”, had a major diameter (width) of 79 mm and a minor diameter (height) of 63.5 mm. The other apple denoted as “small” had a major diameter of 66 mm and a minor diameter of 50.8 mm. The experiment was conducted with 25 repetitions (n=25) for each apple size and each end effector (n=50 total for each end effector). Calculation of Equivalent Torques and Vacuum Pressures Knowledge of the vacuum pressure generated by the robot’s vacuum system during the gripping experiments can help better understand the potential performance of the robotic system, thus enabling us to optimize the system design for fruit picking. Since the vacuum pressure was not measured during the experiments, its value had to be estimated indirectly. For the control end effector, the equivalent torque 𝑇 and vacuum pressure 𝑞 , were calculated using the following two equations, respectively: 55 𝑇 =𝐹×𝑅×𝜑 (9) and 𝐹 𝑞 , = 𝑞 , – 𝐴 × 1000 (10) where 𝑞 is the atmosphere pressure (=101.325 kPa), F is the gripping force in N, 𝜑 is the static coefficient of friction (=0.55), and R and A are the radius (0.02225 m) and its cross-sectional area (=0.00155 m2) of the vacuum tube. For the three end effector alternatives, accurate calculations of equivalent torque and vacuum pressures are not a simple task because of their complex geometries and the variation in fruit size and shape, which makes the actual contact area for the apple on the end effector difficult to estimate. Moreover, imperfect contacts between the fruit and end effector further increase the difficulty of accurate estimation of the vacuum pressure. However, an approximate method may be used for estimating the equivalent torque based on the measured gripping force for the straight end effector because of its relatively simple geometry, compared to the ‘Curved’ and ‘Bellow’ end effectors. When an apple is attracted to the straight end effector, as shown in figure 4.8, it would be in contact with the outer lip and inner lip of the end effector (figure 4.2) due to its conformability. The actual contact area between the fruit and end effector is expected to be somewhat different from the original area encompassing the outer lip and inner lip when no fruit is not being held by the end effector. 56 Figure 4.8: Diagram showing gripping of an apple by the “straight” end effector. Red lines represent the points of contact between the fruit and end effector, around the apple’s perimeter. The apple ideally is gripped by the “inner-lip” and by the “outer-lip” around its entire perimeter Figure 4.9: Diagram showing approximation of apple gripping by the end effector to a collar bearing (apple) sitting on a hollow circular contact area (end effector). The area of contact (shaded) between the apple and end effector with its inner and outer contact diameters being denoted as Ri and Ro. F refers to the gripping force and  is the friction coefficient between the apple and the silicone end effector To estimate the equivalent torque, it is first assumed that: 1) the apple will be in full contact with the area of the end effector encompassed by the inner lip and the outer lip, and 2) the contact area 57 between the apple and end effector can be simplified to be two flat surfaces (figure 4.9). Based on these two assumptions or simplifications, the contact between the apple and the straight end effector can be treated like a loading situation, as shown in figure 4.9, in which a collar bearing is in contact with a stationary hollow circular disk, where the apple is treated as the collar, while the end effector is the flat surface of the disk. Therefore, the following friction equation may be used to estimate the equivalent torque 𝑇, when the apple is being gripped by the straight end effector (Baker & Haynes, 2020): 2 𝑅 𝑅 𝑇= 𝜑𝐹 3 𝑅 𝑅 (11) where F is the measured gripping force in N, 𝑅 is the radius of the outer lip (=0.0251 m) and 𝑅 is the radius of the inner lip (=0.01385 m). After the mean equivalent torque 𝑇 value has been calculated, it was then compared with the critical torque 𝑇 determined from the twisting study reported in chapter 3. This would allow us to determine whether the new end effector and the vacuum system will meet the 95% fruit detachment requirement using the twisting action. Results and Discussion Figure 4.10 shows the results of gripping force evaluation for the control and three end effector design alternatives. Each end effector design had more consistent averages for the two different sized apples than the thin-foam control. For the large fruit, the average gripping force for the control end effector was not significantly different from that for the ‘straight’ and ‘curved’ designs but was much higher than that for the bellow end effector. For the smaller apple, the control had a significantly lower gripping force than each of the silicone design 58 alternatives, which indicates that fruit size had a significant effect on the gripping force and hence the detaching capability of the control. A good end effector should be able to generate consistent gripping forces regardless of fruit size. Gripping Force Evaluation- Fruit Size Comparison Average Gripping Force (N) 30.0 25.0 20.0 15.0 10.0 5.0 0.0 Control* Straight* Curved Bellow Small Apple Large Apple Figure 4.10: Mean gripping forces (25 replications) for the thin-foam control and three end effector design alternatives for two different sized apples with whiskers indicating standard deviation. * Indicates a statistically significant difference between the recorded means for the two different sized test fruit Each of the silicone end effector designs performed more consistently than the control, as measured by the average gripping force for the two artificial apples (figure 4.10). This suggested that the picking performance of the three end effector design alternatives would be less affected by the size of fruit, compared to the control. This is expected because the control end effector has the poor conformity to fruit shape due to its thin foam. A larger fruit has a more uniform or flatter cheek surface and also tends to have a large contacting area with the end effector. When the ‘control’ attached to the larger fruit, fewer air leaks were expected, compared to the smaller 59 fruit which has more profound curvature on the cheek. This preliminary experiment indicated that the conformity of the end effector design would have a profound impact on picking performance in a real orchard harvesting setting. Based on the average gripping forces for the three new end effectors, the ‘straight’ design alternative would be expected to perform better in a real orchard harvesting setting, followed by the ‘curved’ design. It is, however, difficult to determine whether the ‘bellow’ design would perform better than the ‘control’. One should be cautioned that the picking performance is determined by the actual torque that can be generated by each end effector as well as their overall ability of holding fruit during the twisting actions. As discussed earlier, the equivalent torque is not just dependent on the gripping force generated by the vacuum system, but also on the characteristics of the actual contact area between the fruit and the end effector. As reported in chapter 3, an equivalent critical force of 21 N is required for picking 95% of target fruit. Table 4.2 shows that the average gripping force generated by the control for the two artificial apples was only 13.3 N, far lower than this critical value. Hence, it is expected and has also been confirmed in 2019 harvesting tests that the control end effector would be far short of meeting the picking rate requirement. However, the average force (22.1 N) for the straight end effector is higher than the critical force, while the curved end effector has an average gripping force of 19.7 N. Hence, these two end effectors may be close to reach, but are still short of meeting, the picking rate requirement. The bellowed end effector had an average force of 14.4 N, which is far lower than the equivalent critical force. Based on the critical force analysis, it may be inferenced that all three end effectors are still short of meeting expectations in suction force needed to detach 95% of apples. 60 TABLE 4.2: Average gripping forces and standard deviations for the control and three new end effectors and average equivalent torque and vacuum pressure calculated for the control and straight end effectors Average Standard Average Average End Effector Gripping Force Deviation Equivalent Equivalent Design (N) (N) Torque (N-m) Vacuum (n = 50) Pressure (kPa) Control 13.3 9.2 0.163 101.774 Straight 22.1 6.5 0.242 * Curved 19.7 7.5 * * Bellow 14.4 8.5 * * * Values were not calculated. Further analysis of equivalent torques for the control and three new end effectors showed similar results to that based on equivalent suction forces (Table 4.2). The control end effector would be far short of meeting the 95% picking rate requirement, since the calculated equivalent torque is only 0.163 N-m versus the critical torque of 0.257 N-m. For the straight end effector, although its average suction force is higher than the critical force, the average equivalent torque calculated based on equation (11) is only 0.242 N-m, which is 6% lower than the critical torque. Hence, the straight end effector would still be short of meeting the picking rate requirement. Despite these analysis outcomes, the ‘straight’ and ‘curved’ design alternatives are still expected to perform significantly better than the thin-foam control end effector during field evaluation due to the average gripping force being significantly higher than the average required picking force. Results in table 4.2 show that the standard deviation for this experiment is relatively high. The high vacuum flow rate caused the apple to attract to the end effector from a small distance. Once the apple was attached, it was not possible to shift the fruit due to the applied gripping force. In this experiment, it was intended to attach the fruit directly on its “cheek” for each 61 replication, as shown in figure 4.11. However, slight changes in this attachment orientation would cause the end effectors to conform differently with the fruit. This was likely the cause of the relatively high standard deviation for this experiment. Conversely, the attachment orientation of fruit is not easily controlled in real orchard settings, thus, this variability in attachment actually provides a more realistic evaluation of end effector performance. Figure 4.11: Gripping force evaluation for the “small” fruit attached to the “straight” end effector design The gripping experiment and twisting torque analysis showed that the vacuum pressure generated during the picking process may need to be further reduced in order to reach the system goal of 95% picking rate. While maintaining vacuum tube diameter, there are two methods of improving the suction force generated by the picking process. These methods include reducing air leaks in the vacuum system, optimize the radii of the contact area, as shown in equation (11), and increasing the power of vacuum supplied to the system (FESTO, 2021). The new end effector design alternatives were designed and fabricated with the intention of reducing the presence of air leaks in the system. Based on the results of this evaluation, the 5.5 hp wet/dry vacuum system may be insufficient in supplying enough suction power to reach the overall goal 62 for the robotic system. Therefore, in future versions of the robotic system, a more powerful vacuum source and further optimization of the end effector design should be considered to increase the gripping force and torque generated by the end effector. Field Test Evaluation During the 2020 apple harvesting season in September and October, full system evaluations were conducted for the new version of the robotic harvesting system. Full system evaluation was conducted in a research orchard of the Horticultural Teaching and Research Center in Holt, MI. End effector evaluation was done in conjunction with the full system experiment. It is important to note that data collected for end effector evaluation does not include picking attempts in which failure of other subsystems (visual perception, mechanical manipulation) resulted in a failure to reach apples. The orchard used for the end effectors evaluation was planted with high-density ‘Ida Red’ apple trees. The field experiment was intended to evaluate the performance of the three new end effectors versus the original thin-foam effector. To achieve this goal, vacuum pressure data was recorded during the fruit picking process for each of the four end effectors, using a hall-effect pressure sensor (Seeed Technology Co., Ltd., Nanshan, China). Moreover, video was also collected during the field tests and synchronized with the collected pressure data to help evaluate the performance of the three end effector design alternatives and the original thin-foam end effector. Vacuum Pressure Measurement and Analysis The hall-effect pressure sensor detects the pressure of clean, non-corrosive gases and liquids with a range of 0-1.2 MPa (table 4.3). It was installed through the vacuum tube, located 63 roughly 60 cm from the vacuum head. It measures pressure within the tube based on the reference pressure outside the tube (atmospheric pressure = 101.325 kPa). Although the sensor can detect vacuum pressure (less than atmospheric pressure), the conversion equation provided by the manufacturer is only applicable for an output voltage range of 0.5 V-4.5 V (atmospheric pressure gives an output of 0.53V). Voltage outputs below 0.5V indicate vacuum pressure (full specifications shown in table 4.3), but no conversion equation was provided by the manufacturer. TABLE 4.3: Technical specifications of the hall-effect pressure sensor used for recording vacuum pressure of the robotic system during 2020 field tests Working Voltage (DC) 5.0 ± 0.5V Working Pressure Range 0 - 1.2 MPa Working Temperature -20 ℃ + 105 ℃ Measurement Accuracy ± 1.5% Full-Scale Response Time ≤ 2.0 ms This pressure sensor acts as a controller for the robotic system to activate the picking process by detecting the pressure drop within the vacuum tube which results from a fruit attachment to the vacuum head. Preliminary evaluation of the vacuum system determined a threshold sensor output which indicates fruit attachment. Sensor output during fruit picking is continually monitored by an inhouse developed program in ROS at a data acquisition rate of 500 Hz. Once pressure within the vacuum tube falls below the threshold, ROS waits for 0.5 s before triggering the vacuum tube rotation which detaches the apple. Although the actual vacuum pressure cannot be calculated, this rotation mechanism activation threshold was found to be adequate and effective during the 2020 field test of the 64 system. During the field test, the rotation mechanism was triggered 100% of the time when an apple was attached to the vacuum head. Although the rotation mechanism was not always successful in detaching the fruit, the timely activation of the rotation mechanism is helpful in minimizing the cycle time for the overall system. Figure 4.12 shows a pressure profile collected during the 2020 field evaluation of the robotic harvesting system. A drop in the pressure profile indicates the attachment of an apple to the vacuum head during the picking process. In this example, there are three distinct pressure Figure 4.12: Example of a vacuum pressure profile collected during the picking process drops which each correspond to an apple attaching to the robot’s end effector. It can be seen in the example, and throughout the rest of the pressure profiles collected (appendix A) that the pressure drop is nearly instantaneous. Therefore, a 0.5 s waiting time to trigger the rotation mechanism may be excessive and can be shortened in the design of a new version of the robotic harvesting system. These pressure readings were collected as data bags by the robot operating system (ROS). These ROS bags were then uploaded to MATLAB. The MATLAB code used to estimate the pressure drops based on the generated pressure profiles are given in appendix B. 65 Each drop in pressure as shown in figure 4.12 indicates the attachment of an apple to the vacuum head. The pressure sensor used to collect this data provides a voltage output which maps linearly to pressure values by the conversion equation below (Seeed Technologies Co., Ltd., Nanshan, China): 𝑉 = 𝑉 (0.75𝑞 + 0.1) (12) where Vout = Sensor’s voltage output (0-4.5 V) Vcc = Operating voltage (5 V) q = Pressure (MPa). While this pressure sensor can detect pressures from 0-1.2 MPa, the conversion equation in (12) is only valid for the output range 0.5-4.5 V for pressures above atmospheric pressure, which gives a voltage output of 0.5258 V. Thus, the actual pressure generated in the vacuum tube during the picking process could not be determined. Although the actual pressure values were not determined, the sensor output reliably detected pressure changes in the vacuum range (less than atmospheric pressure). Additionally, it was found that the vacuum pressure sensor did not return to a constant base pressure while there was no fruit attached to the system. This may have been caused by such factors as debris entering the vacuum tube. For these reasons, the ability of each end effector design to seal the fruit and generate lower vacuum pressures within the vacuum tube was only evaluated by a relative pressure drop value ∆𝑉 ( %): 𝑉 −𝑉 ∆𝑉 ( %) = × 100 𝑉 (13) 66 where 𝑉 = mean of the last 5 data points prior to pressure drop in volts and 𝑉 = mean of all data points while apple was attached (lower pressure portion). Fruit Dropping Mechanism As previously described, the other major component of cycle time for the robot’s vacuum system is the ability of the system to successfully drop the picked fruit and regenerate vacuum flow rate to pick the next apple. To efficiently drop apples attached to the vacuum head, a cut-off mechanism was installed between the vacuum tube and the shop-vac vacuum supply, as shown in figure 4.13. For the current version of the robot, the cut-off mechanism was operated manually. In the new version, this cut-off mechanism will be automated in synchronization with the manipulation system’s return to a pre-determined position to catch the fruit. Figure 4.13: Vacuum cutoff mechanism in the open position (left) allowing vacuum flow through the tube, and in the closed position (right). This was manually operated to cause fruit to fall from the end effector after picking during 2020 field testing Before picking, the cut-off mechanism is in the open position (left) to allow vacuum flow through the vacuum tube. After fruit detachment, the manipulator brings the fruit to a dropping location. At this point, the cut-off mechanism is closed (right), cutting off the vacuum tube from the source. Although the new end effector designs improve the seal of the fruit to the vacuum 67 tube, this is still not a perfect vacuum seal. Thus, the pressure differential between the tube and the atmosphere causes outside air to enter the vacuum tube, pushing the apple from the vacuum head. It can be seen in figure 4.12 that the time required to restore equilibrium (regenerate vacuum pressure) after the fruit is dropped from the vacuum tube is nearly instantaneous as well. It is worth noting that not all new end effector designs performed the same in terms of success in dropping the fruit. In most cases, the cut-off of the vacuum caused the apple to fall by gravity from the vacuum tube instantaneously. However, in some cases the fruit failed to fall from the tube, requiring human intervention to remove the fruit and significantly increasing the cycle time of the system for attempts in which the dropping failed. Picking Data Analysis The recorded pressure data and video were processed in synchronization based on recorded time stamps in the pressure data. By synchronizing the relative pressure drop of each picking attempt with the video of the field test experiments, several important performance factors can be investigated. In a general sense, the relative pressure drop may be compared with the “picking success” of each “picking attempt”. In this study, a “picking attempt” was only recorded in cases when the end effector was properly positioned by the robotic system to grip the target fruit. During the full system evaluation, there were occurrences in which the end effector was not positioned properly due to issues with either the mechanical manipulation or the perception system. These failures were not included in the evaluation of end effector performance as the end effector never had an opportunity to successfully pick the fruit. “Picking success” was recorded as a binary value based on if the picking attempt resulted in the detachment of the apple from the tree. This field experiment was conducted such that “picking attempts” were repeated until each silicone end effector design and the original thin- 68 foam end effector (control) achieved 40 successful picks (“picking success” = 1). Therefore, the total attempts for each end effector may vary due to different picking success rates. Another important performance factor is the success of the rotation mechanism for fruit detachment, as this was a major shortcoming of the first generation thin-foam end effector. The thin-foam end effector was insufficient in detaching fruit via rotation due to insufficient gripping force exerted on the fruit. This insufficient gripping force caused the fruit to frequently slip on the end of the tube during rotation. Hence, the rotation was rendered useless for many picking attempts. The “rotation mechanism success” was also recorded as a binary value. The “rotation mechanism success” was given a “1” if and only if the rotation mechanism was the direct cause of the fruit detachment. Picking orientation was also recorded during the field test evaluation. Each picking attempt was classified based on the general angle of attachment between the fruit and the vacuum tube. The picking orientation was classified during video review after the field test. Picking orientations were classified into three classes according to the diagram shown in figure 4.14, i.e., top (stem end) = 1, cheek = 2, bottom (calyx end) = 3. Figure 4.14: Classifications of fruit attachment orientations evaluated during the field test 69 Another important performance parameter for the robotic harvesting system is the ability of each end effector to promptly drop the fruit after a successful pick. As seen in the recorded pressure profiles (figure 4.11), the pressure regeneration in the vacuum tube after fruit detachment was nearly instantaneous. Therefore, this time for pressure regeneration was considered negligible in the overall cycle time of the system. The end effector would only constrain the overall cycle time when it failed to drop the fruit. “Dropping success” was recorded as a binary value for each successful pick. If the apple was detached from the end effector instantaneously on the first attempt of cutting off the vacuum, the “dropping success” was given a value of 1. If the fruit failed to detach from the end effector, the dropping was marked as 0. Other performance measures include the performance of end effectors with occlusions present during picking process. These occlusions include the presence of leaves, other branches, or the branch attached to the target fruits itself between the fruit and end effector during a picking attempt. The specific condition of the occlusions was marked qualitatively for each picking attempt. The ability of each design alternative to “overcome occlusions” is defined by the picking success rate of each alternative when an occlusion was present. The picking profiles corresponding to each experiment video for each silicone design and the thin-foam control, and tables recording each of the performance parameters above can be found in appendix A and D, respectively. The data tables include video numbers that correspond to each pressure profile. Evaluation of these performance metrics will help guide a selection for the end effector design most fit for use in future versions of the robotic harvesting system. 70 Results and Discussion Overall Picking Success Rate Table 4.4 summarizes the overall harvesting performance of each end effector design. As previously described, the field test experiment was conducted such that picking attempts were conducted until each end effector achieved 40 successfully picked fruit, thus, the number of picking attempts for each end effector (n) varies. TABLE 4.4: Overall picking success rate and number of picking attempts for each end effector design during field evaluation End Effector Design Picking Success Rate (%) Picking Attempts (n) Straight 87.0 46 Bellow 83.3 48 Curved 83.3 48 Control 48.8 82 As shown in Table 4.4, each silicone design had a much higher picking success rate than the thin-foam control. The three end effectors had similar overall picking success rates ranging between 83% and 87%, with the ‘Straight’ end effector being four percentage points better than the other two designs. The performance of the ‘bellow’ end effector is better than expected, since the average gripping force for this end effected was only 14.4 N from the laboratory gripping study. On the other hand, the picking success rates for the ‘straight’ and ‘curved’ end effectors are generally in agreement with the predicted or expected performance based on the analysis of gripping forces from the laboratory study. It should be noted that while the straight design performed the best, other performance factors should also be considered before an optimal end effector is selected. The relatively consistent performance by the three end effectors suggests that 71 the conformability of the silicone designs played a critical role in contributing to the better overall harvesting performance. Vacuum Pressure Evaluation Comparing the overall success rates with the relative pressure drops generated by each design alternative will help confirm the hypothesis that a lower pressure generation caused by a more conformable end effector will result in higher harvesting efficiency. Figure 4.15 shows the average relative pressure drop achieved by each design alternative and the control for successful picks and failed picking attempts (failure to detach fruit from the tree after attachment to end effector). 11.00 10.50 Relative Pressure Drop (%) 10.00 9.50 9.00 8.50 8.00 7.50 7.00 Straight Bellow Curved Control Successful Pick Failed Pick Figure 4.15: Average relative pressure drops with associated standard error (whiskers shown on individual bars) on successful and failed picking attempts It is seen that for the three new designs, the relative pressure drop was significantly larger on successful picking attempts than failed picking attempts. This confirms that the generation of a lower pressure in the vacuum system upon fruit attachment leads to a higher rate of harvesting. 72 Furthermore, the difference in pressure drop between successful picks and failed picks was not significant for the ‘control’. This was likely due to the rigid nature of the ‘control’, which would have caused similar air leaks in both cases due to non-conformity. Large differences in pressure drop for the successful and failed picks for the silicone designs are because they conformed better on fruit that were successfully harvested, thus generating sufficient suction force to detach the fruit from the tree. For failed picking attempts, poor conformity was likely to be the cause for a less profound pressure differential and holding force. It was observed that in some fruit picking attempts, the silicone end effectors had undergone excessive deformation with inconsistent folding, as shown in figure 4.16. This folding was especially profound for the ‘curved’ design, resulting in incomplete conformation to the contours of the target fruit and thus pronounced air leaks. These poor fruit deformations likely caused even greater air leaks than the rigid ‘control’, resulting in a much lower pressure drop on failed picking attempts. Figure 4.16: The folding of the “curved” end effector design occurred during fruit attachment Single factor ANOVA conducted to compare the relative pressure drop for successful picking attempts (at a significance level of α=0.05) is shown in table 4.5. The null hypothesis for 73 this statistical test states that there is no statistical difference between the results for each end effector: TABLE 4.5: Results of single-factor ANOVA comparing the relative pressure drop for successful picking attempts and failed picking attempts ANOVA: Single Factor SUMMARY Groups Count Sum Average Variance Straight 40 402.6795 10.06699 2.856543 Bellow 40 414.1454 10.35363 2.461793 Curved 40 390.8913 9.772283 0.40966 Control 40 373.118 9.32795 1.308456 ANOVA Source of Variation SS df MS F P-value F crit Between Groups 23.0262 3 7.675401 4.363222 0.005562 2.66256 Within Groups 274.4216 156 1.759113 Total 297.4478 159 The ANOVA confirmed that there was a significant statistical difference between the average relative pressure drops of the end effectors. ANOVA were conducted between each silicone alternative (‘straight’, ‘bellow’, ‘curved’) and ‘control’, which resulted in the P-values of 0.02472, 0.00128, and 0.03515, respectively. Thus, there were significant differences in the pressure drop between each of the silicone designs and the ‘control’. There was no significant statistical difference between the three end effector designs, based on a p-value of 0.175 (Appendix D). Hence, it can be concluded that the silicone end effectors increased conformity to the irregular contours of apple, resulted in a significantly lower vacuum pressure during the picking 74 process. This lower vacuum pressure, in turn, resulted in significantly higher picking success rates, thus greatly improving harvesting efficiency of the robotic harvesting system. Full ANOVA tables for each end effector design alternative can be found in appendix D. It is interesting to note that although the ‘bellow’ design had the overall lowest gripping forces among the three silicone end effectors (table 4.2), it has performed much better during the field evaluation, with the successful picking rate being equal to that for the curved end effector. This may be explained by the relative pressure drop by this end effector. As shown in figure 4.15, the mean pressure drops for both successful and failed pick attempts by the bellow end effector are either equal to or higher than that for the straight and curved end effectors. In particular, the mean pressure drop for failed picking attempts is significantly higher than that for the other two end effectors. This suggests that it had better conformability to apples than the other two silicone end effectors. However, since fruit detachment is mainly determined by the generated torque, it was likely that the bellow end effector may not have been able to generate sufficient torque, even with higher pressure drops, due to its unique geometry and greater deformability. On the other hand, although the ‘straight’ design did not generate the greatest relative pressure drop, the harvesting rate indicates that the geometry of the design may have allowed for greater gripping forces and torque to be exerted on the fruit. This is supported by the preliminary gripping force evaluation as well as several other performance parameters evaluated during the field test evaluation. Although each silicone design performed significantly better than the ‘control’, further evaluations of other performance metrics would help determine if the ‘straight’ design indeed performed significantly better than the other silicone designs 75 Orientation Analysis The ability to successfully pick fruit in different attachment orientations to the target fruit is critical for the robotic harvester as the approach angle to the target fruit is variable. Figure 4.17 shows the average success rate and relative pressure drop generated for picking attempts at each attachment orientation, for each of the silicone designs and ‘control’. Figure 4.17: Average picking success rate (SPR) and relative pressure drop (RPD) for each design alternative at different attachment orientations All three design alternatives had consistent and relatively high picking success rates for orientation 2, compared to the two other orientations, which showed greater variability in picking performance. It is also interesting to note that relative pressure drops (RDP) for the straight end effectors was lower for orientation 2 than for the other two orientations, but the corresponding successful picking rate (SPR) for this orientation was higher than SPR for orientation 1. This 76 may suggest that it would be easier to detach fruit in attachment orientation 2 than in the other two orientations, although further studies are needed to confirm this finding. Complex biological factors have an impact on each of these harvesting scenarios. It is important to note that the frequency of incidences encountered for each picking orientation (1-3) varied greatly: 1- 13%, 2- 49%, 3- 37%, for the four end effectors combined. Aside from picking orientation 1, which had a very low incidence, the ‘straight’ design had higher picking success rates (88.5% and 88.9%) for the other two, more common, picking orientations, compared to both ‘curved’ and ‘bellow’ designs. This indicates that the ‘straight’ design is likely the best at conforming to the contours of the fruit. The ‘bellow’ design performed better than the other two designs in picking orientation 1. This could be attributed to the fact that the bellow end effector was more flexible to conform to fruit orientation, due to its special structure features. However, it should be noted that picking orientation 1 occurred on only 13% of total picking attempts. Thus, the results for orientation 1 are not as impactful because of much lower attachment incidence for this orientation. As previously described, the rotation mechanism of the robotic harvesting system attempts to detach fruit using a pendulum motion, rather than a “twist and pull” method, which has shown to improve harvesting efficiency in previous studies (Li et al., 2016). Based on this “twist and pull” picking method, the fruit is meant to be rotated around the axis of the stem, thus weakening the stem before pulling the fruit straight down, away from the branch above, to detach the fruit. In attachment orientation 1, the apple is attached to the end effector at the top of the fruit. Thus, while the rotation mechanism is rotating the fruit around the proper axis, the fruit is being pulled in the opposite direction (towards the branch). This “twist and pull” picking method is most closely followed in picking orientation 3, in which the end effector attaches to 77 the bottom of the fruit (calyx end). It was found that an overall picking success rate of 81% for the three silicone end effectors were achieved for this attachment orientation. In comparison, the three silicone end effectors achieved an overall picking success rate of 87% for orientation 2 (cheek). Hence, to optimize system performance, the robot should attempt to approach fruit such that attachment orientation 2 can be achieved more often. Although this attachment orientation yielded the overall highest picking success rate, it was only achieved on 49% of picking attempts. Additionally, the ‘straight’ design, which had the best overall harvesting performance, achieved 88.5% of picks in orientation 2. Rotation Performance Tables 4.6 and 4.7 show the success rate of the rotation mechanism for each end effector design alternative and the success rate of the rotation mechanism at each attachment orientation (combined data for each silicone alternative). The “rotation mechanism success rate” is defined as the percent of successfully detached fruit that were detached by the rotation mechanism alone, rather than the vacuum tube’s return to home position: TABLE 4.6: Successful picking rate (%) of the rotation mechanism causing fruit detachment End Effector Design Success Rate (%) Straight 65 Bellow 57 Curved 49 Control 27 78 TABLE 4.7: Successful picking rate (%) of the rotation mechanism at various attachment orientations Orientation Straight Bellow Curved Combined 1 38 44 80 48 2 72 71 55 65 3 88 52 23 50 It can be seen that in addition to having the most consistent performance amongst attachment orientations and the highest overall picking success rate, the ‘straight’ design also had the highest rotation mechanism success rates, especially for orientations 2 and 3 (Table 4.6 and 4.7). This indicates that the ‘straight’ design is likely to generate the greatest torque on the fruit, which is again, supported by the gripping force evaluation previously described. It was also shown that the rotation mechanism yielded overall higher successful picking rates for attachment orientations 2, compared to the other two orientations. This suggests that a new version of the robotic system should be such designed or controlled that it would achieve this attachment orientations most frequently. By optimizing the approach angle of the vacuum tubes relative to the target fruit, the rotation mechanism performance will likely improve, which will in turn improve overall harvesting efficiency and reduce the overall cycle time of the robotic harvesting system. It was found that 50% of picking attempts that failed to detach fruit from the tree with the silicone end effectors, was caused by flexibility of the branch which followed the fruit all the way back to the robot arm’s home position without detaching. In cases when the rotation mechanism failed to detach fruit, the system relied on a “straight pull” picking action to detach the fruit, while the vacuum arm was returning to its home position. 79 Additionally, when the system returns to home position after each picking attempt, this significantly adds to the overall cycle time of the system. Thus, the effectiveness of the rotation mechanism has several important implications on overall system performance, including harvesting rate and cycle time. It was also seen in field evaluations that when the fruit was detached as a result of the vacuum arm returning to home, excessive force was exerted to the tree, causing vibration throughout the tree. This vibration frequently resulted in other fruit falling from the tree. It was found that 25% of picking attempts that resulted in target fruit detachment by the vacuum arm’s return to home position (straight pull) caused neighboring fruit to fall from the same or other branches within the canopy. These fruit that were knocked off the tree would be lost permanently. This could have a significant negative economic ramification for the grower and thus must be minimized. It should be noted that field evaluation was conducted in an orchard with young trees with many small and slender branches that have not reached full maturity. Other Performance Parameters Apple canopies are complex and unpredictable structures, which present great challenges for automated suction-based harvesting systems (Kourtev, 2018; Shamshiri et al., 2018). These complex canopies will cause fruit occlusions to be present for a significant portion of picking attempts. The presence of occlusions was noted during the field test evaluation. There were typically four types of occlusions, which were classified as “leaves”, if one or more leaves were caught between the target fruit and the end effector during the picking process, “twigs” if small branch sub-structures were occluding the attachment, “branches” if larger branch canopy structures were present, and “stem-branch junction” in cases when the vacuum flow caused the apple to attach to the end effector such that the branch that bears the target fruit itself occluded the attachment. 80 Each of these occlusion types, and each case when occlusions were present would have a different degree of impact on the quality of attachment to the end effector. For example, the presence of a single leaf occluding the attachment is clearly not as impactful as the presence of a large branch. However, due to the difficulty of quantifying the magnitude of individual occlusions, all occlusions were considered the same for the purpose of comparison. Each end effector design was evaluated by its ability to “overcome occlusions”. This percentage indicates the number of successful picks achieved by each design while occlusions were present in the picking attempt. Table 4.8 shows the performance of each end effector design when overcoming occlusions. TABLE 4.8: Performance of each end effector design alternative in the presence of environmental occlusions End Effector Picking Success Rate Total Occlusion Avg Pressure Drop (%) Type (%) Cases Straight 100.0 15 9.82 Bellow 87.5 16 10.62 Curved 90.0 10 9.75 Control 36.3 11 8.96 The ‘straight’ design again performed the best in this evaluation, as shown in the table. This further suggests that this design has the best ability to conform to irregular contours to grip target fruit, and it should therefore be chosen for a future version of the robotic harvesting system. Lastly, the ability of the end effector to quickly drop the fruit is an important performance and/or design consideration based on its implications with system cycle time. Picking attempts that did not successfully pick the fruit were not counted in this rate, as the end effector had no opportunity to drop the fruit because it was never detached from the tree. The evaluated end 81 effector alternatives ‘straight’, ‘bellow’, ‘curved’, and ‘control’ achieved dropping success rates of 60%, 95%, 92% and 98%, respectively. It was noted that the ‘straight’ design, which performed best in nearly all evaluated performance parameters, had such a significantly lower ability to drop the fruit. This poor ability to drop the fruit is likely a product of the ‘straight’ design exerting the greatest gripping force on the fruit, as shown in the preliminary “gripping force evaluation” and supported by evidence throughout this field evaluation. For each of the other end effectors, when the vacuum was shut-off, the weight of the fruit caused it to fall from the grip of the end effector. While the efficiency of the dropping mechanism was high for the ‘bellow’, ‘curved’, and ‘control’, it was still not 100% effective. This suggests that the method of simply cutting off the vacuum supply to induce the apple dropping from the end effector is not an acceptable solution, especially for the ‘straight’ design. If the ‘straight’ design is selected for use in future versions of the robotic system, a new, automatic dropping mechanism should be developed. Failure of the end effector to drop the fruit would require human intervention before harvesting is resumed, which will greatly reduce overall harvesting efficiency. Overall Discussion The field harvesting performance evaluation showed that all three end effector designs performed significantly better than the original thin-foam end effector, but they have failed to reach the goal of 95% picking rate, which has also been confirmed by the theoretical analysis of the orchard pulling and twisting experiments and laboratory gripping study. Hence, a more powerful vacuum source and an improved end effector design should be considered in future versions of the robotic system. Theoretical analysis showed that the straight end effector, which had overall best performance for different evaluation parameters, could generate the average 82 equivalent torque of 0.243 N-m, compared to the critical torque (0.257 N-m) needed to detach 95% of apples based on the pulling and twisting studies for three apple varieties. A few important factors must be evaluated to ensure an improved vacuum system will not inadvertently reduce the overall performance of the robotic harvesting system. The first of these factors is the potential of bruising the target fruit. As it is explained earlier, the improved end effector design alternatives were designed and constructed such that bruising will be minimized. Bruising effects on the fruit were not evaluated during the field test evaluation. When a higher suction force is applied to the fruit, bruising effects should be studied to ensure the robotic harvesting system is fit for picking fresh market apples. Additionally, the ability of the system to effectively drop the fruit should be evaluated under increased vacuum conditions. Although the straight end effector overall had superior performance, it had poor performance in dropping picked apples with the current manual dropping mechanism. Hence an improved, automated fruit dropping mechanism should be consider for future versions of the robotic system. The control system for the robot’s vacuum supply should also be improved to optimize the performance of the system. Based on the near instantaneous drop in pressure upon apple attachment to the end effector, the 0.5 s waiting time to initiate the rotation mechanism should be reduced, which will have a significant impact in achieving the overall system goal of 3 s cycle time. Finally, to enhance the overall picking success rate, the robot’s control and planning algorithms should be such designed that it will enable the robot’s end effector to approach target fruit from orientation 2 (the cheek). To ensure quality of the end effector over time, studies should be conducted to understand the degradation of the silicone material as it is used in the field. By contacting 83 branches and twigs, collecting dirt/dust, and getting wet, the silicone may degrade overtime which could negatively impact its overall performance. Conclusions Several important conclusions can be made based on the laboratory and field evaluation of the three new silicone end effector designs, comparing to the original thin-foam end effector used in the first generation of the robotic system. First, laboratory gripping force tests and theoretical analysis showed that the vacuum system coupled with the straight end effector could significantly improve the picking performance, but it was still short of meeting the goal of detaching 95% of apples, which was further confirmed by the picking performance of the end effector. The thin foam-based end effector was, however, far short of meeting the picking performance requirements, based on laboratory test and analysis as well as the field picking evaluation. Second, the soft and conformable end effector designs minimized air leaks during fruit attachment, leading to lower vacuum pressure during fruit attachment and thus a greater suction force being applied to the fruit. As a result, each of the three silicone designs performed more consistently for fruit of different sizes and significantly better than the original thin foam ‘control’, as measured by both average relative pressure drops and successful picking attempts. Third, the ‘straight’ design performed consistently better than the other two silicone alternatives, and it has achieved 87% overall picking success rate, 65% success rate of the rotation mechanism and more consistent picking performance for the three fruit attachment orientations. Hence, this end effector should be chosen for future versions of the robotic harvesting system. Finally, environmental occlusions (i.e., leaves, twigs, branches, etc.) had little or no effect on fruit picking success rate, which was especially evident for the ‘straight’ design. 84 CHAPTER 5: DESIGN AND CONSTRUCTION OF AN INDOOR SIMULATED ORCHARD ENVIRONMENT Introduction Field testing and evaluation in the real orchard environment is a critical step in the development of robotic harvesting systems. However, apple harvest season is short, typically lasting for only about four to six weeks per year. It is thus very useful to have a robust indoor environment that closely mimics apple orchard conditions (i.e., lighting, tree structures, picking forces, etc.). This type of artificial environment would enable experimentation and new design prototype optimization more conveniently and cost effectively, year around. This will accelerate the progression of the robotic system research and development. In designing a simulated orchard environment (SOE), several important factors must be considered. First, accurate visual representation of a real apple orchard is needed for evaluation and improvement of the perception system. This will require realistic looking tree structures, fruit, and background. Second, the SOE should have the capability to simulate realistic lighting conditions (i.e., bright, cloudy/overcast, direct lighting, back lighting) with different orchard orientations and different times of day. Third, it is equally important for the SOE to be an accurate test environment for the robot’s manipulation system. Sufficiently robust canopy structures that can mimic realistic environmental occlusions, and robust fruit hanging scenarios (clusters, etc.) should be included in the SOE as these are the most problematic picking scenarios based on field evaluation. To provide a realistic canopy environment, tree branches must be formed in realistic structures, fruit must be mounted in realistic scenarios with desired detachment forces. By accomplishing each of the previously mentioned tasks, a robust indoor 85 orchard testing environment can be developed, which would be immensely beneficial to robotic apple harvesting research. This chapter first provides an overview of the overall layout or design of an indoor SOE. It then discusses the design and fabrication of the light system, tree branches and fruit hanging mechanism to mimic the real tree canopy in orchard. Overview of the Indoor Simulated Orchard Environment An indoor SOE was designed and constructed in the Prototyping Annex of Farrall Agricultural Engineering Hall at Michigan State University. In the design of the SOE, spatial constraints had to be carefully considered, balancing floor space available and the minimum space that would be needed for testing and evaluation of new robotic harvesting systems. The basic requirements for the new SOE include the installation of one row of minimum four artificial apple trees, each of which is approximately 2.4 m high and 0.9 m in width (8 ft. x 3 ft.), and sufficient open floor space for the robot prototypes to travel forth and back along the artificial canopy. Moreover, the SOE should have sufficient vertical and horizontal space for installing a movable light system that can simulate natural lighting conditions for different times of day. After consideration of these requirements, the new SOE was constructed using aluminum frames with the floor space dimensions of approximately 4.7 m long and 3 m wide (15.5 ft ×10 ft) and a height of 3.7 m (12 ft). Figure 5.1 shows a SketchUp drawing highlighting the dimensions of the outside frame and the general layout of the SOE. 86 Figure 5.1: Overall dimensions and layout of the simulated orchard environment (SOE). Shown inside the SOE frame is a 3.6 m (12 ft) movable tree frame for installing four artificial trees To minimize the interference of ambient light, the SOE is enclosed with an opaque curtain on each of the four side walls and covered with corrugated roof panels on the top of the SOE. By blocking outside light, the lighting conditions inside the SOE can be controlled based on realistic orchard conditions. Four artificial trees were installed on a “tree frame” that sits on roller wheels inside the SOE, unattached to the SOE frame. Wheels on the tree frame enable swiveling relative to the light system in a manner such that various canopy row orientations can be simulated (north- south, east-west, etc.). This tree frame provides 3.7 m (12 ft) canopy length for testing the harvesting robot. The trees are divided into two sections. One tree is equipped with variable force electromagnets, such that required picking force can be manually set and adjusted for realistic conditions, which would be used for evaluating the performance of the end effectors and the vacuum system for picking fruit. The other three trees are equipped with flexible steel rods acting as “branches”. These flexible rods are formed into realistic canopy structures and used to 87 create a more realistic visual representation of commercial apple canopies. A detailed description of the design of artificial branches and fruit stems is given in a later section. Light System In the design of the SOE, construction of an appropriate light system that can simulate different natural lighting conditions is critical for testing and evaluating the perception system of a robotic harvester. As such, the system should be able to mimic different natural lighting conditions (clear sky, cloudy/overcast sky, etc.). The lights should also be able to provide direct (clear sky) or diffused (cloudy or overcast) illumination on the tree canopy with an area that is equal to or greater than the imaging area of the robot’s visual perception system. For the vacuum-based robotic harvesting system, the working area is approximately 0.6 m by 0.6 m (2 ft by 2 ft). Finally, the light system should be adjustable in its incident angle to simulate different times of day and different row orientation of trees in orchard. With these requirements for the light system, the selection of appropriate lights for the SOE was initially done by Dr. Renfu Lu and Mr. Pengyu Chu, a Ph.D. student of the research team working on the visual perception system of the robotic harvester. After extensive review and comparison of different types and designs of lights, it was determined that LED light panels with proper color temperatures would best meet the requirements. Three LED light panel assemblies (each 604 mm wide by 457 mm tall) each with different color temperatures (5000 K, 5700 K, and 6500 K) were acquired from a commercial company (MECREE, Shenzhen, China) to simulate different natural lighting conditions. The lowest color temperature light, 5000 K, simulates evening conditions when the sun appears to cast a more orange/red color temperature (Jou et al., 2009). It is estimated that the average color temperature of daylight at noon is 5500 K, thus the 5700 K light panel is used to simulate mid-day, clear conditions. Lastly, the 6500 K light 88 panel is used to simulate afternoon conditions on cloudy days, when the sun will cast a more white color temperature as the light is naturally diffused by clouds (Jou et al., 2009). Each LED panel delivers 60000 lumens (Note: a bright direct sunlight has an illumination intensity of about 120,000 lux) (Lu, Su, Pylnev, Long, Wu, Tsai, & Wei et al., 2021). Hence, the selected LED panels should provide sufficient illumination to simulate different outdoor lighting conditions, since the panel LED lighting area is only 0.276 m 2, with a small divergent angle of 15°. As described earlier, the light system should be such designed that it is movable in all three directions (horizontal and vertical) to provide different angles of illumination on the tree canopy and adjust its position relative to the robot. In addition, the system should allow easy, quick selection of one of the three LED light panels for different lighting conditions. To meet these requirements, a special light system was designed and installed inside the SOE, with technical support provided by Mr. Tom Steucken of Prototech, Inc., a local contractor specializing in mechanical systems design and fabrication. The three LED light panels are installed on a specially designed triangular light configuration, as shown in Figure 5.2, which allows for three distinct lighting conditions. Each LED light is mounted on one surface of the equilateral triangle assembly (120 degrees apart). The triangular assembly is held on a U-shaped frame through a rotation shaft. The light assembly is constructed with two triangular plates, each with specific function. Figure 5.2 highlights the two triangular plates, with the three light panels mounted to each edge of the triangular plates. One plate of the light assembly features a pin-locking mechanism used to allow 360° free rotation of the triangular light assembly at 15° increments (the right photo of figure 5.2), when it is suspended in the air and disconnected from a power cord. In this way, each light panel can be used at a desired angle relative to the tree frame. The other plate (left, figure 5.2) on the opposite 89 side of the light configuration features power inputs and dimming control inputs for each lighting panel. Figure 5.2: Custom designed triangular light assembly used in the simulated orchard environment. Side view of the triangular light assembly with a pin-locking mechanism allowing to select one of the three LED light panels with a specific incident angle to illuminate the tree canopy (right), and the opposite side of the triangular configuration with power inputs and dimming controls for the lights (left) Figure 5.3 and figure 5.4 show the design of the light suspension system. A rigid aluminum I-beam runs across the top of the light frame perpendicularly to the tree frame. This I- beam is mounted on each end to a roller-track, allowing the I-beam to travel the length of the SOE. A 1500-W lifting winch with the lifting/holding capacity of 1000 kg (2200 lbs.) is installed on the I-beam. This winch uses a cable-pulley system to hold/lift the triangular light system. The light suspension system can roll on the I-beam via two rolling wheels mounted on each side of the I-beam. The light system is adjusted by first manually positioning the I-beam perpendicularly 90 to the tree frame, and then manually positioning the winch or suspension system at a proper location along the I-beam for the desired illumination position. The height of the light system is done remotely by operating the lift functions of the winch. In addition, an additional safety rope locking mechanism (i.e., the orange rope and an aluminum beam mounted on top of I-beam, as shown in figure 5.3) is incorporated with the light suspension system to ensure safety of the operator and the light system. A full description of the standard operating procedure for properly positioning the light system within the SOE is given in appendix E. Figure 5.3: Photos of the cable/pulley light suspension system used to position the light system within the simulated orchard environment 91 Figure 5.4: Photo showing the I-beam which runs across the top of the simulated orchard environment, perpendicularly to the artificial canopy (below image) Figure 5.5: Front view of light panels suspended from the winch cable/pulley system, mounted to I-beam. Safety rope (orange) is engaged 92 Artificial Tree Structures A rolling frame was constructed to mount artificial trees within the artificial orchard environment (figure 5.6). By installing the trees on a rolling frame, separate from the main frame of the SOE, it becomes more convenient to position the artificial canopy row to simulate various orchard orientations (north-south, east-west etc.) relative to the light system which is used to simulate natural lighting conditions. Figure 5.6: Artificial trees installed on the movable tree frame in the simulated orchard environment The tree frame was designed based on realistic commercial apple orchard dimensions. Nowadays, uniform, high-density tree architectures, such as vertical planar canopy, are widely adopted for apple production in U.S, which not only produce higher yielding, better quality fruit, but are also advantageous for automated harvesting (Silwal et al., 2016). Hence, artificial trees with a vertical planar canopy structure were selected to mimic real trees for the indoor orchard testing environment. Tree spacing along orchard rows depends largely on the maturity of trees, rootstock, and type of harvesting equipment being used (Roper et al., 1992). The typical tree height in commercial orchards ranges from 3-4.2 m (10-14 ft), with the tree width ranging from 93 1.5-2.1 m (5-7 ft) and an in-row tree spacing of roughly 1.5 m (5 ft) (Parker, Unrath, Safely, & Lockwood, 1998). Due to size constraints of the lab space, these orchard dimensions were scaled down for the construction of the indoor orchard testing environment. Four artificial fiscus trees with a height of 2.4 m (8 ft) and a width of 112 cm (44 in.) were acquired from a commercial vendor and installed to make up the four-tree canopy row. These trees were spaced by roughly 0.9 m (3 ft) each. Hence, the canopy in the SOE is scaled at roughly 2/3 the size of an average commercial orchard. A uniform, opaque background behind the artificial tree canopy would provide an inappropriate representation of the real orchard and could affect the testing and realistic evaluation of the perception system with the developed machine learning algorithms. To provide a more realistic visual representation of outdoor orchards, screen-printed images of real apple orchards were installed on the wall of the SOE behind the tree frame, relative to the position of the robot. This would create a background scene that is like that seen in a real field test setting for the robot’s visual perception system. These images were sized such that the apples in the images had an average diameter of roughly 7.6 cm (3 in.), which is approximately the actual size of mature apples (Stajnko, Lakota, & Hočevar, et al., 2004). Figure 5.7 shows an image representing the view of the robot’s RGB-D perception system in the SOE. 94 Figure 5.7: (left) Visual perception bounding box (blue) image detecting apples within the simulated orchard environment and (right) an example of a real orchard image used as background images for the simulated orchard environment Artificial Fruit Stems and Tree Branches In testing and evaluating a vacuum-based or conventional robotic harvesting system, it is important to have a realistic representation of tree branches and apples including stems in terms of their appearance and physical characteristics or properties (such as the size, stiffness or flexibility of branches and stems). This would allow realistic assessment of the performance of a robot’s gripping mechanism or end effector for detaching fruit from trees and its perception system for detection and localization of target fruit. Using artificial apples made of plastic or similar materials may provide a good visual representation, but these artificial apples often do not have the same surface color and texture as that of real apples. In addition, the weight and mechanical properties of artificial fruit are also different from that of the real apples (friction, etc.). Hence, they may not give a realistic representation of the fruit picking process. 95 The best way to realistically mimic real apples is to utilize real apples in the environment. To realize this goal, artificial stems are needed to mimic real apple stems that can be easily attached to fruit and branches. After several preliminary trials for different stem design concepts, we came up with a simple method for fabricating artificial stems. The artificial stem design, as shown in figure 5.8, is made up of a brown 12-gauge wire of roughly 4 cm long, with two steel shaft collars affixed to each end of the wire. The shaft collars are magnetically attached to 9.5 mm diameter neodymium magnets. These magnets have a maximum holding force of 31 N, which is within the range of forces that are needed to detach fruit from trees (Chapter 3). Figure 5.8: Artificial stem used to magnetically attach a real apple to an artificial branch. (left) The naked artificial stem consisting of a brown 12 gauge wire with the two ends affixed with two steel collars and two magnetic disks, one of which is attached to a screw that is embedded into the stem end of a real apple and the other end attached to the artificial branch, and (right) the same stem with the two collar/magnet sets covered with brown tape to mimic the actual color of fruit stems 96 A brown tape, which looks similar in color to the apple stem, is used to cover the magnet/shaft collar segment of the stems so that the metallic magnet will not interfere with the machine vision system caused by light reflection. One end of the artificial stems is attached to a number 10 steel screw, which is imbedded into the real apple tissue through its stem end. The magnet on the opposite end of the artificial stem is attached to an artificial branch made of a steel rod. Figure 5.9 shows an artificial branch made of a steel rod, hanging with a real apple through the artificial stem described above. To magnetically attach real apples to the flexible steel rod branches, steel shaft collars were fastened to the steel rods by a set screw. These steel shaft collars offer natural magnetism which is sufficient to attach the artificial stems with the weight of real apples. As shown in figure 5.9, one side of the shaft collars was flattened such that the magnetic stems will not slip when a pick is attempted. By flattening one edge of the shaft collars, it also allows the fruit to be hung in different orientations than straight down from the branch. Many different fruit hanging orientations are found in real apple trees. Hence, by mounting these shaft collars in various directions on the customizable steel branches, many picking scenarios can be replicated to create a robust artificial orchard testing environment. These shaft collars may also be mounted closely together on the steel branches to mimic fruit clustering together, which are commonly seen on apple trees (figure 5.10). 97 Figure 5.9: Steel shaft collar installed on an artificial steel “branch” (left). Used to magnetically attach a real fruit using an artificial stem (right) Figure 5.10: Real apples (Gala) hung in the artificial canopy in clustered formations and individually, using magnetic artificial stems The 2020 field test evaluation of the robotic system showed that one of the most common causes of failed picking attempts during the field test experiment described in Chapter 4, was the flexible nature of fruit bearing branches in apple canopies. When the rotation mechanism of the robotic system failed to cause fruit detachment, the system relied on the motion of the vacuum tube returning to home position to cause fruit detachment with a “straight-pull” picking action. 98 Flexibility of the branches frequently caused the apple to remain attached to the branch during this “straight-pull” picking action. To mimic this characteristic of real apple trees, three sizes of flexible steel rods with diameters of 3.2 mm, 4.7 mm, and 6.2 mm were installed within the artificial tree canopy to mount apples. The different size steel branches provide various degrees of branch flexibility within the artificial canopy. These steel branches were bent and installed in customized geometries to mimic common canopy structures seen in commercial harvesting settings. While using the artificial stems and branches described above enables hanging real apples on artificial trees with a realistic canopy structure, a picking force measuring mechanism is also needed in the design and testing of different end effectors and evaluating the effectiveness of, or the power needed for, a vacuum system. The steel shaft collars used to mount the magnetic artificial stems generates an average magnetic holding force of 11.7 N, lower than the critical forces calculated in Chapter 3. Therefore, to ensure that realistic picking forces may also be mimicked in the SOE, one of the trees was equipped with variable force electromagnets to mount apples (figure 5.11). Based on results from the required picking force experiment described in Chapter 3, these electromagnets can achieve a holding force of up to 30 N. The holding force of these variable force electromagnets can be adjusted by varying the supplied current. These electromagnets in conjunction with the magnetic artificial stems create a realistic testing environment which mimics realistic force required to detach fruit with a straight pull picking force. The other three trees in the artificial canopy are reserved for replicating realistic canopy structures and fruit attachment orientations. 99 Figure 5.11: Two real apples mounted using variable force electromagnets and magnetic artificial stems Figure 5.12 shows the SOE ready for use with an artificial apple attached to the variable force electromagnet, and the light system properly positioned to simulate an early morning lighting scenario. Figure 5.12: Photos showing the light system properly positioned relative to the artificial tree canopy for testing of the robotic system 100 Figure 5.12 (cont’d) Summary The construction and use of an indoor simulated orchard environment can greatly enhance the future development of the robotic harvesting system. The realistic picking processes are simulated using semi-flexible steel rods as branches, magnetic artificial stems to be used with real apples, and variable force electromagnets mounted within a canopy of artificial tree structures for evaluating the picking force of the robotic system. Moreover, a unique light system with three adjustable LED panels of different color temperatures has been constructed to simulate the lighting conditions at different times of day and in different weather conditions for optimization of the robot’s visual perception system in the indoor environment. The custom three-panel light assembly and light suspension system allows robust customization of the lighting conditions in the SOE. The new SOE allows the picking performance of the robot to be realistically evaluated and optimized throughout the year without being constrained to the short harvesting season. 101 CHAPTER 6: CONCLUSIONS This research was aimed at developing new soft end effectors to improve the overall performance of the newly developed vacuum-based robotic harvester, and an indoor simulated orchard environment to enhance robotic harvesting research. To provide guidance for the design of end effectors and the improvement of the vacuum system, the forces and torques required for fruit picking using straight pulling and twisting actions were evaluated for three varieties of apples in an orchard in 2020. Results indicate that a suction force of 28.3 N (equivalent vacuum pressure of 83.13 kPa) is required for the vacuum-based system to achieve the overall system goal of 95% picking success rate by a straight pulling method. When a twist picking motion is used to detach fruit, a critical torque of 0.257 N-m is needed to achieve the same goal of 95% success rate, which would be equivalent to a lower suction force of 21.0 N (vacuum pressure of 87.82 kPa), when using the original version of the end effector. Hence, the twist picking motion should be a preferred mode of fruit detachment by robotic harvesting systems because it requires a lower suction force, minimizing potential damage to fruit and tree canopies. Three new end effectors of different geometries, i.e., “straight”, “bellow” and “curved”, were designed and fabricated using casted silicone rubbers and 3-D printing. Laboratory gripping experiments with two different sized artificial apples for the three new end effectors and the original thin-foam end effector (control) showed that the picking performance of the control end effector, measured by the gripping force, was greatly influenced by fruit size due to its unconformable properties, while the three silicone end effectors have exhibited more consistent gripping performance regardless of fruit size. Further theoretical analysis showed that the equivalent average torque for the straight end effector design was 0.242 N-m, only 4% lower than the critical torque needed to detach 95% of apples. This suggested that the straight end 102 effector along with the curved end effector would be close to reach the target picking success rate. Field picking evaluations of the robotic system performance in 2020 harvest season showed that all the three new end effectors performed significantly better than the control end effector based on multiple performance metrics (i.e., vacuum pressure, overall picking success rate, fruit attachment orientation, picking success rate by the rotation mechanism, and occlusion effect). Each of the silicone designs generated a significantly greater pressure drop during apple attachment than the original thin-foam end effector on successful picking attempts. The robotic system performed best when attaching to fruit in orientations 2 (i.e., the cheek of apples) and orientation 3 (the bottom or calyx section). The straight end effector had overall better, more consistent picking performance, compared to the other two end effectors; it achieved 87% overall picking success rate, 65% picking success rate by the rotation mechanism alone, and ~88% for the preferred cheek attachment orientations. Hence, the straight end effector should be chosen for future developments of the robotic harvester. Further analysis of the vacuum pressure changes during fruit dropping with a manual fruit dropping mechanism suggested that the pre-set waiting time of 0.5 s can be greatly reduced, and a more efficient, automated fruit dropping mechanism should be used for the next versions of the robotic harvesting system to improve the overall harvesting time cycle. The new indoor SOE provides a realistic environment for testing and evaluating the robotic harvesting system. The realistic picking processes are simulated using artificial trees embedded with specially designed branches and magnetic artificial stems to be used with real apples, and variable force electromagnets are mounted within a canopy of artificial tree structures for evaluating the suction force of the robotic system. Moreover, a unique light system 103 with three adjustable LED panels of different color temperatures was constructed to simulate the lighting conditions at different times of day and in different environmental conditions for evaluation and optimization of the robot’s visual perception system in the indoor environment. While the new silicone end effectors have greatly improved the picking performance of the harvesting robot, they are still short of meeting the goal of 95% picking success rate. Hence, further improvements to the end effector and vacuum system should be considered. First, the straight end effector should be further optimized for its geometry and physical properties (i.e. hardness and deformability) so as to improve its conformability to the irregular contours of fruit, thus reducing air leaks when fruit attach to the end effector. Second, a new end effector should be considered, so that it can generate greater torques under a given vacuum pressure. This may be achieved by optimizing the contact area between the fruit and the end effector and increasing the contact distance from the center of the vacuum tube (thus increasing the lever length). Furthermore, replacing the current vacuum system with a more powerful vacuum system would provide a quick, effective solution, but this would also lead to increased energy consumption. Finally, improvements to the robot’s manipulation and perception systems should be considered, so that the robot can approach target fruit with a higher degree of positioning accuracy and from the preferred attachment orientation (i.e., the cheek of fruit, calyx end). 104 APPENDICES 105 APPENDIX A: Vacuum Pressure Profiles Collected During 2020 Field Test Pressure data from the robotic operation system (ROS) was processed using the MATLAB script in Appendix B. The y-axis for these pressure curves represents the output voltage (voltage divided by 10) from the pressure sensor installed within the vacuum tube. The x-axis represents the time and pressure was recorded at a frequency of 10 Hz. The pressure profiles correspond to a recorded video which was synchronized to the vacuum pressure data. The ‘Video #’ noted for each pressure profile can be found in the full Field Test results tables shown in Appendix C, such that each picking attempt can be quantified based on vacuum pressure and picking success rate for several different picking scenarios. Some pressure profiles correspond to multiple videos and some videos include multiple pressure profiles, as noted. Video #1 Video #2 Video #3 Figure A.1: Straight end effector design pressure profiles 106 Figure A.1 (cont’d) Video #4 Video #5 Video #6, 7, 8 Video #9 107 Figure A.1 (cont’d) Video #10 Video #11 (1) Video #11 (2) 108 Video #1 Video #2 Video #3 Figure A.2: Bellow end effector design pressure profiles 109 Figure A.2 (cont’d) Video #4 Video #5 Video #6 110 Figure A.2 (cont’d) Video #7 Video #8 (1) Video #8 (2) Video #9 111 Video #1 (1) Video #1 (2) Video #2 Video #3 Figure A.3: Curved end effector design pressure profiles 112 Figure A.3 (cont’d) Video #4, 5 Video #6 Video #7 113 Video #1 Video #3 Video #4 Video #5 Figure A.4: Control end effector design pressure profiles 114 Figure A.4 (cont’d) Video #6 Video #7 Video #8 Video #9 115 Figure A.4 (cont’d) Video #10 (1) Video #10 (2) Video #11 (1) Video #11 (2) 116 Figure A.4 (cont’d) Video #11 (3) 117 APPENDIX B: MATLAB Script for Generating Pressure Profiles 118 APPENDIX C: 2020 Field Test Results Tables TABLE A.1: Picking scenarios and picking success rate Straight Video # Picking Orientation Cluster? Occlusion Occlusion Rotation Picking Drop Other Notes Attempt (1-3) (1/0) (1/0) Notes mech Success Success (Branch, (1/0) (1/0) (1/0) Leaves, other) 1 1 2 0 0 1 1 1 1 2 1 0 1 leaves 0 1 0 1 3 3 0 1 leaves 1 1 1 1 4 1 1 0 1 1 1 1 5 1 0 0 1 1 1 1 6 2 0 0 1 1 1 1 7 2 1 0 1 1 1 2 1 3 0 1 twig and 1 1 0 leaves 2 2 2 0 1 leaves 1 1 1 2 3 3 1 0 0 0 knock off neighbor fruit 2 4 2 0 0 0 1 1 2 5 3 1 0 1 1 0 2 6 2 0 1 leaves 1 1 0 2 7 2 1 0 0 0 2 8 2 1 1 twig and 1 1 1 leaves 2 9 3 1 0 1 1 1 visible fold 2 10 1 0 0 1 1 1 3 1 2 0 0 0 0 3 2 2 0 0 1 1 0 4 1 2 0 0 1 1 1 4 2 1 0 1 twig and 0 0 leaves 4 3 1 0 1 twig and 0 0 leaves 4 4 1 0 1 twig and 0 1 1 same fruit as leaves previous two 5 1 1 0 1 twig and 0 1 0 knock off leaves neighbor fruit 5 2 2 0 1 leaves 0 1 0 5 3 2 0 0 0 1 0 6 1 2 0 1 twig and 1 1 1 leaves 6 2 1 0 1 twig and 0 1 1 leaves 6 3 2 0 0 1 1 1 6 4 2 0 0 1 1 0 6 5 2 0 0 0 0 6 6 2 0 1 leaves 1 1 1 119 Table A.1 (cont’d) 7 1 3 1 0 1 1 1 7 2 2 0 0 1 1 1 8 1 2 1 1 leaves 1 1 0 8 2 3 1 1 leaves 1 1 0 9 1 2 0 0 1 1 0 9 2 2 0 0 1 1 0 10 1 3 1 1 leaves 1 1 0 10 2 2 0 0 1 1 0 10 3 2 0 0 1 1 1 10 4 1 0 1 leaves 1 1 0 10 5 3 0 0 1 1 1 10 6 1 0 0 0 1 1 11 1 2 0 0 0 1 1 poor positioning 11 2 2 1 0 0 1 1 poor positioning 46 0.65 0.87 0.6 Bellow Video # Picking Orientation Cluster? Occlusion Occlusion Rotation Picking Drop Other Notes Attempt (1-3) (1/0) (1/0) Notes mech Success Success (Branch, (1/0) (1/0) (1/0) Leaves, other) 1 1 3 0 1 leaves 1 1 1 1 2 2 1 0 1 1 1 1 3 3 0 0 0 0 1 4 2 0 1 twig and 0 1 1 leaves 1 5 3 1 0 0 1 1 knock off neighbor fruit 1 6 3 0 0 1 1 1 1 7 2 0 0 1 1 1 1 8 3 0 0 0 0 flexible branch 2 1 1 0 0 1 1 1 2 2 2 0 0 0 1 1 2 3 2 0 0 1 1 1 3 1 1 0 0 1 1 1 3 2 1 1 0 0 1 1 poor positioning 3 3 2 0 0 1 1 1 3 4 3 0 0 1 1 1 4 1 2 0 1 leaves 1 1 1 4 2 3 1 0 0 1 0 4 3 3 1 0 0 1 1 4 4 1 0 1 twig and 0 1 1 leaves 4 5 1 1 1 leaves 0 0 flexible branch 120 Table A.1 (cont’d) 5 1 2 0 0 1 1 1 5 2 3 1 0 1 1 1 5 3 2 1 0 1 1 1 5 4 3 1 0 0 0 flexible branch 5 5 3 1 0 0 0 same fruit 5 6 2 0 0 0 0 same fruit 5 7 2 0 0 0 0 flexible branch 5 8 3 0 0 1 1 1 5 9 1 0 1 leaves 1 1 1 6 1 3 1 0 0 1 1 6 2 1 1 1 junction 0 1 1 6 3 3 1 1 leaves 1 1 1 6 4 3 1 0 1 1 1 poor positioning 6 5 3 1 0 1 1 1 6 6 3 1 0 1 1 0 knock off neighbor fruit 7 1 2 0 0 1 1 1 knock off neighbor fruit 7 2 2 0 1 leaves 1 1 1 8 1 2 0 1 leaves 0 1 1 poor positioning 8 2 3 0 0 1 1 1 8 3 1 0 0 0 1 1 8 4 3 0 1 leaves 0 0 flexible branch 9 1 2 0 1 leaves 1 1 1 poor positioning 9 2 2 0 1 leaves 1 1 1 9 3 3 1 1 leaves 0 1 1 knock off neighbor fruit 9 4 1 0 1 leaves 1 1 1 9 5 3 1 1 leaves 1 1 1 9 6 2 0 1 leaves 1 1 1 Average: 57% 83% 95% Curved Video # Picking Orientation Cluster? Occlusion Occlusion Rotation Picking Drop Other Notes Attempt (1-3) (1/0) (1/0) Notes mech Success Success (Branch, (1/0) (1/0) (1/0) Leaves, other) 1 1 2 0 0 1 1 1 visible folding 1 2 2 1 0 1 1 1 1 3 2 0 1 leaves 1 1 1 knock off neighbor fruit 1 4 2 1 0 1 1 1 knock off neighbor fruit 2 1 2 0 0 0 1 0 2 2 2 0 0 0 1 1 2 3 2 0 0 0 1 1 121 Table A.1 (cont’d) 2 4 2 0 0 1 1 1 2 5 2 0 0 1 1 1 3 1 2 0 0 0 1 1 3 2 3 1 0 0 1 1 3 3 2 0 0 1 1 1 knock off neighbor fruit and visible fold 3 4 2 0 0 1 1 1 3 5 3 0 0 1 1 1 3 6 2 0 0 1 1 1 3 7 3 1 0 0 1 1 knock off neighbor fruit 3 8 3 1 0 1 1 1 3 9 2 0 0 0 1 1 4 1 3 0 0 0 1 1 knock off neighbor fruit 4 2 2 0 0 0 1 0 4 3 3 1 0 0 1 1 4 4 3 1 0 0 1 1 poor positioning 4 5 2 1 0 0 0 poor positioning and knock off neighbor fruit 5 1 2 0 0 0 0 poor positioning and knock off neighbor fruit and visible folding 5 2 2 1 0 0 0 poor positioning 5 3 3 1 0 0 1 1 5 4 1 1 1 leaves 1 1 1 visible folding 5 5 3 1 0 1 1 1 5 6 2 1 1 leaves 0 1 1 5 7 2 1 1 leaves 1 1 1 6 1 1 0 1 junction 1 1 1 6 2 2 0 0 1 1 1 6 3 2 0 0 0 1 1 knock off neighbor fruit 6 4 2 0 1 leaves 1 1 1 6 5 3 0 0 0 0 knock off neighbor fruit and visible fold 6 6 3 0 0 0 0 knock off neighbor fruit and flexible branch 7 1 1 1 1 junction 0 0 flexible branch 7 2 2 0 1 leaves 1 1 1 7 3 3 1 0 0 1 1 knock off neighbor fruit and visible fold 7 4 2 1 0 1 1 1 7 5 1 0 0 1 1 1 122 Table A.1 (cont’d) 7 6 3 1 0 0 1 0 7 7 1 0 0 1 1 1 poor positioning 7 8 2 0 0 0 0 flexible branch 7 9 2 0 0 0 0 same fruit 7 10 2 0 1 leaves 1 1 1 7 11 2 0 1 twig 1 1 1 Average: 49% 83% 92% Control Video # Picking Orientation Cluster? Occlusion Occlusion Rotation Picking Drop Other Notes Attempt (1-3) (1/0) (1/0) Notes mech Success Success (Branch, (1/0) (1/0) (1/0) Leaves, other) 1 1 2 1 0 0 1 1 1 2 1 0 0 1 1 1 1 3 1 0 0 0 0 1 4 3 1 0 0 1 1 3 1 2 0 0 0 0 3 2 2 0 0 0 0 same fruit 3 3 2 0 0 0 0 same fruit 3 4 2 0 0 0 0 same fruit 3 5 2 0 0 0 0 same fruit 3 6 2 0 0 0 0 same fruit 3 7 2 0 0 1 1 1 same fruit 3 8 2 0 0 1 1 1 3 9 3 0 0 0 0 fruit slips in rotation 3 10 3 0 0 0 0 fruit slips in rotation 3 11 3 0 0 0 0 fruit slips in rotation 3 12 3 0 0 0 0 fruit slips in rotation 3 13 3 0 0 0 0 fruit slips in rotation 3 14 2 0 0 1 1 1 4 1 3 0 0 0 1 1 4 2 3 0 1 branch 0 0 fruit slips in rotation 4 3 3 0 1 branch 0 0 fruit slips in rotation 4 4 3 0 1 branch 0 0 fruit slips in rotation 4 5 2 1 0 0 0 knocks off neighbor fruit 4 6 3 0 1 branch 0 0 fruit slips in rotation 4 7 3 0 1 branch 0 0 same fruit a (2,3,4,6) 4 8 3 1 0 0 1 1 123 Table A.1 (cont’d) 4 9 3 1 0 0 0 4 10 3 0 1 branch 0 0 same fruit as earlier 4 11 3 1 0 0 0 4 12 3 1 0 0 0 4 13 3 1 1 branch 1 0 fruit detached and collided with branch during return to home 5 1 3 1 0 1 1 1 5 2 2 0 0 1 1 1 5 3 3 0 0 0 0 poor positioning 5 4 3 1 0 0 0 5 5 3 1 0 0 0 fruit slips in rotation 5 6 3 1 0 0 0 5 7 3 1 0 0 0 fruit slips in rotation 5 8 3 1 0 0 0 poor positioning 5 9 3 1 0 0 0 5 10 3 1 0 0 1 1 5 11 3 1 0 0 0 fruit slips in rotation 5 12 3 1 0 0 1 1 5 13 3 0 0 0 0 fruit slips in rotation 5 14 3 0 0 0 1 1 same fruit 6 1 2 0 0 1 1 1 6 2 2 0 0 1 1 1 7 1 3 0 0 1 1 1 7 2 2 0 0 0 1 1 8 1 3 0 0 0 0 8 2 3 0 0 0 1 1 same fruit 8 3 3 1 0 0 1 1 8 4 3 1 0 0 1 1 9 1 2 1 0 0 1 1 9 2 2 1 0 0 0 10 1 2 0 0 1 1 1 10 2 2 0 1 leaves 1 1 1 10 3 2 1 0 1 1 1 10 4 2 0 0 0 1 1 10 5 2 0 0 0 0 fruit slips in rotation 10 6 2 0 0 0 0 fruit slips in rotation 10 7 2 0 0 0 0 fruit slips in rotation 10 8 2 0 0 0 0 same fruit 124 Table A.1 (cont’d) 10 9 2 0 0 0 1 1 10 10 2 0 0 0 0 10 11 3 0 0 1 1 1 10 12 2 0 0 1 1 1 10 13 2 1 0 1 1 0 10 14 1 0 0 0 0 10 15 1 0 1 junction 1 1 1 10 16 2 0 0 1 1 1 11 1 2 0 0 0 1 1 11 2 2 1 0 0 0 11 3 2 1 0 1 1 1 11 4 2 0 0 0 1 1 11 5 3 0 0 0 0 11 6 3 0 0 0 0 11 7 3 0 0 0 1 1 same fruit 11 8 3 1 0 0 1 1 11 9 2 0 1 leaves 0 1 1 12 1 2 0 1 leaves 1 1 1 12 2 2 0 0 1 1 1 12 3 3 0 0 1 1 1 Average: 27% 48% 98% TABLE A.2: Recorded pressure sensor output results Straight Video # Picking Pre-Attach Pressure Post-Attach Pressure Pressure Drop Relative Pressure Drop Attempt (V) (V) (V) (%) 1 1 0.013692 0.012475 0.001217 8.89 1 2 0.01372 0.012345 0.001375 10.02 1 3 0.01378 0.01247 0.00131 9.51 1 4 0.013748 0.01218 0.001568 11.41 1 5 0.01392 0.01262 0.0013 9.34 1 6 0.013892 0.012576 0.001316 9.47 1 7 0.01372 0.01272 0.001 7.29 2 1 0.013836 0.012398 0.001438 10.39 2 2 0.013864 0.012298 0.001566 11.30 2 3 0.013806 0.012639 0.001167 8.45 2 4 0.013718 0.012327 0.001391 10.14 2 5 0.013748 0.01229 0.001458 10.61 2 6 0.013602 0.012493 0.001109 8.15 125 Table A.2 (cont’d) 2 7 0.01369 0.01238 0.00131 9.57 2 8 0.013662 0.012375 0.001287 9.42 2 9 0.013634 0.01237 0.001264 9.27 2 10 0.013748 0.01238 0.001368 9.95 3 1 0.013692 0.01243 0.001262 9.22 3 2 0.013718 0.01243 0.001288 9.39 4 1 0.01363 0.01233 0.0013 9.54 4 2 0.01378 0.01247 0.00131 9.51 4 3 0.01363 0.01276 0.00087 6.38 4 4 0.01367 0.012507 0.001163 8.51 5 1 0.01374 0.01235 0.00139 10.12 5 2 0.013864 0.01247 0.001394 10.05 5 3 0.013776 0.012369 0.001407 10.21 6 1 0.013748 0.012545 0.001203 8.75 6 2 0.013804 0.012545 0.001259 9.12 6 3 0.013894 0.01254 0.001354 9.75 6 4 0.013488 0.01247 0.001018 7.55 6 5 0.013516 0.01249 0.001026 7.59 6 6 0.01378 0.01233 0.00145 10.52 7 1 0.01372 0.01232 0.0014 10.20 7 2 0.01381 0.01227 0.00154 11.15 8 1 0.013808 0.01244 0.001368 9.91 8 2 0.013806 0.01242 0.001386 10.04 9 1 0.01378 0.0124 0.00138 10.01 9 2 0.01379 0.01232 0.00147 10.66 10 1 0.01378 0.01233 0.00145 10.52 10 2 0.013815 0.01231 0.001505 10.89 10 3 0.01369 0.012286 0.001404 10.26 10 4 0.01378 0.01237 0.00141 10.23 10 5 0.013808 0.012224 0.001584 11.47 10 6 0.01383 0.01127 0.00256 18.51 11 1 0.01378 0.01233 0.00145 10.52 11 2 0.01378 0.01232 0.00146 10.60 Average: 0.001358804 9.88 Bellow Video # Picking Pre-Attach Pressure Post-Attach Pressure Pressure Drop Relative Pressure Drop Attempt (V) (V) (V) (%) 1 1 0.013604 0.01243 0.001174 8.63 1 2 0.01372 0.01232 0.0014 10.20 1 3 0.013808 0.01255 0.001258 9.11 126 Table A.2 (cont’d) 1 4 0.013806 0.012209 0.001597 11.57 1 5 0.013808 0.01234 0.001468 10.63 1 6 0.013836 0.012342 0.001494 10.80 1 7 0.01372 0.01238 0.00134 9.77 1 8 0.01378 0.01154 0.00224 16.26 2 1 0.01378 0.01243 0.00135 9.80 2 2 0.01363 0.01225 0.00138 10.12 2 3 0.013602 0.01226 0.001342 9.87 3 1 0.01378 0.01238 0.0014 10.16 3 2 0.01381 0.01213 0.00168 12.17 3 3 0.01378 0.01125 0.00253 18.36 3 4 0.013836 0.012222 0.001614 11.67 4 1 0.01378 0.012386 0.001394 10.12 4 2 0.01377 0.01221 0.00156 11.33 4 3 0.01378 0.01218 0.0016 11.61 4 4 0.01371 0.01219 0.00152 11.09 4 5 0.01357 0.01259 0.00098 7.22 5 1 0.01378 0.012347 0.001433 10.40 5 2 0.01378 0.01251 0.00127 9.22 5 3 0.01378 0.012538 0.001242 9.01 5 4 0.01363 0.012634 0.000996 7.31 5 5 0.013668 0.01245 0.001218 8.91 5 6 0.01372 0.01247 0.00125 9.11 5 7 0.013776 0.01238 0.001396 10.13 5 8 0.01368 0.01236 0.00132 9.65 5 9 0.01392 0.01247 0.00145 10.42 6 1 0.013634 0.012346 0.001288 9.45 6 2 0.013692 0.01234 0.001352 9.87 6 3 0.01363 0.01233 0.0013 9.54 6 4 0.01369 0.01231 0.00138 10.08 6 5 0.01363 0.01253 0.0011 8.07 6 6 0.013806 0.01247 0.001336 9.68 7 1 0.01378 0.01233 0.00145 10.52 7 2 0.01378 0.01247 0.00131 9.51 8 1 0.01369 0.01233 0.00136 9.93 8 2 0.01348 0.01222 0.00126 9.35 8 3 0.013688 0.01218 0.001508 11.02 8 4 0.01363 0.01272 0.00091 6.68 9 1 0.01363 0.01247 0.00116 8.51 9 2 0.01369 0.01234 0.00135 9.86 127 Table A.2 (cont’d) 9 3 0.01366 0.01218 0.00148 10.83 9 4 0.01378 0.01226 0.00152 11.03 9 5 0.01378 0.01242 0.00136 9.87 9 6 0.01366 0.01228 0.00138 10.10 Average: 0.001397872 10.18 Curved Video # Picking Pre-Attach Pressure Post-Attach Pressure Pressure Drop Relative Pressure Drop Attempt (V) (V) (V) (%) 1 1 0.01371 0.01237 0.00134 9.77 1 2 0.01363 0.01231 0.00132 9.68 1 3 0.01367 0.01229 0.00138 10.10 1 4 0.01366 0.01227 0.00139 10.18 2 1 0.01369 0.0124 0.00129 9.42 2 2 0.01378 0.01247 0.00131 9.51 2 3 0.0136 0.012389 0.001211 8.90 2 4 0.01366 0.01241 0.00125 9.15 2 5 0.01375 0.01247 0.00128 9.31 3 1 0.01378 0.01247 0.00131 9.51 3 2 0.01378 0.01235 0.00143 10.38 3 3 0.0136 0.01241 0.00119 8.75 3 4 0.01378 0.01247 0.00131 9.51 3 5 0.01378 0.01233 0.00145 10.52 3 6 0.01378 0.01247 0.00131 9.51 3 7 0.01363 0.01233 0.0013 9.54 3 8 0.01378 0.01247 0.00131 9.51 3 9 0.01378 0.01262 0.00116 8.42 4 1 0.01381 0.01238 0.00143 10.35 4 2 0.01363 0.0123 0.00133 9.76 4 3 0.01366 0.01235 0.00131 9.59 4 4 0.01371 0.01247 0.00124 9.04 4 5 0.01363 0.01262 0.00101 7.41 5 1 0.01368 0.01262 0.00106 7.75 5 2 0.01356 0.01242 0.00114 8.41 5 3 0.01378 0.01235 0.00143 10.38 5 4 0.01375 0.0124 0.00135 9.82 5 5 0.01385 0.01224 0.00161 11.62 5 6 0.0137 0.01243 0.00127 9.27 5 7 0.01378 0.0123 0.00148 10.74 6 1 0.01366 0.01228 0.00138 10.10 6 2 0.01367 0.01229 0.00138 10.10 6 3 0.01366 0.01233 0.00133 9.74 128 Table A.2 (cont’d) 6 4 0.01385 0.01232 0.00153 11.05 6 5 0.01363 0.0126 0.00103 7.56 6 6 0.01372 0.01247 0.00125 9.11 7 1 0.0136 0.01273 0.00087 6.40 7 2 0.0137 0.01235 0.00135 9.85 7 3 0.01369 0.01245 0.00124 9.06 7 4 0.01378 0.01233 0.00145 10.52 7 5 0.01392 0.01247 0.00145 10.42 7 6 0.01371 0.01238 0.00133 9.70 7 7 0.01366 0.01237 0.00129 9.44 7 8 0.01363 0.01247 0.00116 8.51 7 9 0.01363 0.01254 0.00109 8.00 7 10 0.0136 0.01238 0.00122 8.97 7 11 0.01378 0.01241 0.00137 9.94 Average: 0.001296191 9.45 Control Video # Picking Pre-Attach Pressure Post-Attach Pressure Pressure Drop Relative Pressure Drop Attempt (V) (V) (V) (%) 1 1 0.01378 0.01247 0.00131 9.51 1 2 0.01378 0.01247 0.00131 9.51 1 3 0.01363 0.01262 0.00101 7.41 1 4 0.01378 0.01262 0.00116 8.42 3 1 0.01378 0.01247 0.00131 9.51 3 2 0.01378 0.01247 0.00131 9.51 3 3 0.01363 0.01233 0.0013 9.54 3 4 0.01378 0.01247 0.00131 9.51 3 5 0.01378 0.01247 0.00131 9.51 3 6 0.01378 0.01247 0.00131 9.51 3 7 0.01363 0.01247 0.00116 8.51 3 8 0.01378 0.01247 0.00131 9.51 3 9 0.01378 0.01262 0.00116 8.42 3 10 0.01392 0.01247 0.00145 10.42 3 11 0.01378 0.01247 0.00131 9.51 3 12 0.01392 0.01262 0.0013 9.34 3 13 0.01392 0.01262 0.0013 9.34 3 14 0.01378 0.01262 0.00116 8.42 4 1 0.01378 0.01247 0.00131 9.51 4 2 0.01378 0.01276 0.00102 7.40 4 3 0.01378 0.01262 0.00116 8.42 4 4 0.01378 0.01262 0.00116 8.42 129 Table A.2 (cont’d) 4 5 0.01392 0.01247 0.00145 10.42 4 6 0.01378 0.01262 0.00116 8.42 4 7 0.01378 0.01276 0.00102 7.40 4 8 0.01392 0.01247 0.00145 10.42 4 9 0.01392 0.01262 0.0013 9.34 4 10 0.01392 0.01262 0.0013 9.34 4 11 0.01392 0.01262 0.0013 9.34 4 12 0.01378 0.01262 0.00116 8.42 4 13 0.01407 0.01247 0.0016 11.37 5 1 0.01378 0.01262 0.00116 8.42 5 2 0.01378 0.01262 0.00116 8.42 5 3 0.01378 0.01247 0.00131 9.51 5 4 0.01392 0.01262 0.0013 9.34 5 5 0.01378 0.01262 0.00116 8.42 5 6 0.01378 0.01276 0.00102 7.40 5 7 0.01378 0.01262 0.00116 8.42 5 8 0.01378 0.01291 0.00087 6.31 5 9 0.01392 0.01291 0.00101 7.26 5 10 0.01378 0.01291 0.00087 6.31 5 11 0.01392 0.01247 0.00145 10.42 5 12 0.01392 0.01262 0.0013 9.34 6 1 0.01378 0.01247 0.00131 9.51 6 2 0.01392 0.01247 0.00145 10.42 7 1 0.01392 0.01247 0.00145 10.42 7 2 0.01378 0.01247 0.00131 9.51 8 1 0.01392 0.01247 0.00145 10.42 8 2 0.01392 0.01262 0.0013 9.34 8 3 0.01392 0.01276 0.00116 8.33 8 4 0.01378 0.01276 0.00102 7.40 9 1 0.01378 0.01262 0.00116 8.42 9 2 0.01378 0.01247 0.00131 9.51 10 1 0.01378 0.01247 0.00131 9.51 10 2 0.01378 0.01247 0.00131 9.51 10 3 0.01392 0.01276 0.00116 8.33 10 4 0.01392 0.01247 0.00145 10.42 10 5 0.01378 0.01262 0.00116 8.42 10 6 0.01392 0.01262 0.0013 9.34 10 7 0.01392 0.01247 0.00145 10.42 10 8 0.01392 0.01247 0.00145 10.42 10 9 0.01392 0.01247 0.00145 10.42 130 Table A.2 (cont’d) 10 10 0.01378 0.01276 0.00102 7.40 10 11 0.01378 0.01262 0.00116 8.42 10 12 0.01392 0.01247 0.00145 10.42 10 13 0.01378 0.01233 0.00145 10.52 10 14 0.01392 0.01233 0.00159 11.42 10 15 0.01407 0.01247 0.0016 11.37 10 16 0.01392 0.01233 0.00159 11.42 11 1 0.01392 0.01247 0.00145 10.42 11 2 0.01378 0.01233 0.00145 10.52 11 3 0.01378 0.01276 0.00102 7.40 11 4 0.01392 0.01262 0.0013 9.34 11 5 0.01378 0.01247 0.00131 9.51 11 6 0.01407 0.01233 0.00174 12.37 11 7 0.01392 0.01247 0.00145 10.42 11 8 0.01378 0.01262 0.00116 8.42 11 9 0.01378 0.01276 0.00102 7.40 12 1 0.01378 0.01247 0.00131 9.51 12 2 0.01378 0.01233 0.00145 10.52 12 3 0.01392 0.01247 0.00145 10.42 Average: 0.001282469 9.26 131 APPENDIX D: ANOVA Tables Comparing End Effector Relative Pressure Drop Performance on Successful Picks TABLE A.3: Single factor ANOVA table comparing all four end effector designs SUMMARY Groups Count Sum Average Variance Straight 40 402.6795 10.067 2.857 Bellow 40 414.1454 10.354 2.462 Curved 40 390.8913 9.772 0.410 Control 40 373.118 9.328 1.308 ANOVA Source of Variation SS df MS F P-value F crit Between Groups 23.026 3 7.675 4.363 0.006 2.663 Within Groups 274.422 156 1.759 Total 297.448 159 TABLE A.4: Single factor ANOVA table comparing Straight and Control end effector designs SUMMARY Groups Count Sum Average Variance Control 40 373.118 9.328 1.308 Straight 40 402.6795 10.067 2.857 ANOVA Source of Variation SS df MS F P-value F crit Between Groups 10.924 1 10.924 5.245 0.025 3.963 Within Groups 162.435 78 2.082 Total 173.359 79 132 TABLE A.5: Single factor ANOVA table comparing Bellow and Control end effector designs SUMMARY Groups Count Sum Average Variance Control 40 373.118 9.328 1.308 Bellow 40 414.1454 10.354 2.462 ANOVA Source of Variation SS df MS F P-value F crit Between Groups 21.041 1 21.041 11.161 0.001 3.963 Within Groups 147.040 78 1.885 Total 168.080 79 TABLE A.6: Single factor ANOVA table comparing Curved and Control end effector designs SUMMARY Groups Count Sum Average Variance Control 40 373.118 9.328 1.308 Curved 40 390.8913 9.772 0.410 ANOVA Source of Variation SS df MS F P-value F crit Between Groups 3.949 1 3.949 4.596 0.035 3.963 Within Groups 67.007 78 0.859 Total 70.955 79 133 TABLE A.7: Single factor ANOVA table comparing Straight, Bellow, and Curved end effector designs SUMMARY Groups Count Sum Average Variance Straight 40 402.680 10.067 2.857 Bellow 40 414.145 10.354 2.462 Curved 40 390.891 9.772 0.410 ANOVA Source of Variation SS df MS F P-value F crit Between Groups 6.760 2 3.380 1.770 0.175 3.074 Within Groups 223.392 117 1.909 Total 230.152 119 134 APPENDIX E: Standard Operating Procedure (SOP) for the Simulated Orchard Environment Activating the System: 1. Retrieve remote control from the designated location in the robot lab (“the cage”). 2. Plug in the orange extension cord to supply power to the winch/light system. 3. Turn knob on the remote control to the “on” position. A red light located on the bottom of the winch will turn on. 4. Hold the green start button on remote control for 5 seconds to activate the winch. A green light will flash on the winch to indicate successful activation. Positioning the Lights (Moving up): 1. With the light system resting on the ground, confirm tension in the pulley cable. Briefly press ↑ button on remote control to raise the light system off the ground. Warning: Do not attempt to reposition lights while resting on the floor. Note: The second winch for side movement is deactivated, thus  and  buttons on remote control have no function. 2. Pull pin on the triangular face of the light system, rotate the light system such that the desired panel is facing the orchard row with a specific angle of light incidence. 3. Pull a pin on the top of the light system to rotate the light system vertically to a desired angle. 4. Connect the power cable to a proper plug on the opposite triangular face of the light system to turn on the desired light panel. 5. To position lights, pull down on the white hanging ropes to release stops. Use rope attached to the winch to move in/out relative to the trees. Use the rope attached to the I-beam track (the main frame of the environment) to move along the orchard row. Note: Only little force is required to move lights. If you find any significant strain, there may be a problem. Please stop and contact supervisor. 6. Once lights are properly positioned, use ↑ button on the remote control to raise lights to a desired height. Slack will accrue on the orange safety rope as the lights are raised. This is normal. 7. Pull down on the orange rope below the ascender (orange climbing component) until the rope is taught, to ensure the safety system is engaged. Warning: Do not stand directly underneath the light system until the orange safety rope is engaged. Note: The lights are expected to be repositioned horizontally while being raised. Positioning the Lights (Moving Down): 1. Use a ladder to climb up and manually release the orange safety rope. Create slack on the orange safety rope by releasing the ascender mechanism and then pulling the rope up through the ascender. Black electrical tape on orange rope indicates the length of rope required to return the light system back to the ground. 135 Note: If the safety rope is not disengaged before the lights are lowered, the cables for the pulley system will disengage. In this case, the safety mechanism will hold the lights, carefully use ↑ button on the remote control to restore tension in the pulley cable. Ensure the cable returns properly to the grooves on the circular pulley. 2. Press ↓ button on the remote control to lower the light system to a desired height. 3. Follow Steps 2 and 3 in the previous section to adjust the light system to desired angles in both vertical and horizontal directions. 4. Pull down on the orange rope below the ascender (orange climbing component) until the rope is taught, to ensure the safety system is engaged. Warning: Do not stand directly underneath the light system until the orange safety rope is engaged. Shutting Down the Lights: 1. Unplug the power cable from the triangular face of the light system to shut off the light panel. 2. Turn knob on the remote control back to the “off” position. Unplug the orange extension cord outside the environment to disconnect power from the system. 3. Return the remote control back to the designated location in the robot lab. 136 REFERENCES 137 REFERENCES Ainla, A., Verma, M. S., Yang, D., & Whitesides, G. M. (2017). Soft, Rotating Pneumatic Actuator. Soft Robotics, 4(3), 297-304. doi:10.1089/soro.2017.0017 Auroy, P., Duchatelard, P., Zmantar, N. E., & Hennequin, M. (1996). Hardness and shock absorption of silicone rubber for mouth guards. The Journal of Prosthetic Dentistry, 75(4), 463-471. doi:https://doi.org/10.1016/S0022-3913(96)90045-1 Baker, Daniel W., Haynes, William. (2020). Engineering Statics: Open and Interactive. Chapter 9.7: Disc Friction. Retreived from https://open.umn.edu/opentextbooks/textbooks/1047 Bamotra, A., Walia, P., Prituja, A. V., & Ren, H. (2018, July). Fabrication and characterization of novel soft compliant robotic end-effectors with negative pressure and mechanical advantages. In 2018 3rd International Conference on Advanced Robotics and Mechatronics (ICARM) (pp. 369-374). IEEE. Bullock, G. E. (1954). United States Patent No. 2775088. Burks, T., Bulanon, D., & Mehta, S. (2017). Opportunity of Robotics in Precision Horticulture. In Q. Zhang (Ed.), Automation in Tree Fruit Production: Principles and Practice, 234. Chew, C. W. (1975). United States Patent No. 3898785. Chiu, Y. C., Yang, P. Y., & Chen, S. (2013). Development of the End-Effector of a Picking Robot for Greenhouse-Grown Tomatoes. Applied Engineering in Agriculture, 29(6), 1001-1009. doi:https://doi.org/10.13031/aea.29.9913 Davidson, J., Silwal, A., Karkee, M., Mo, C., & Zhang, Q. (2016). Hand-picking dynamic analysis for undersensed robotic apple harvesting. Transactions of the ASABE, 59(4), 745-758. Davidson, J. R., Hohimer, C. J., & Mo, C. (2016). Preliminary Design of a Robotic System for Catching and Storing Fresh Market Apples. Ifac Papersonline, 49(16), 149-154. doi:10.1016/j.ifacol.2016.10.028 Fujita, M., Ikeda, S., Fujimoto, T., Shimizu, T., Ikemoto, S., & Miyamoto, T. (2018). Development of universal vacuum gripper for wall-climbing robot. Advanced Robotics, 32(6), 283-296. doi:10.1080/01691864.2018.1447238 FESTO (2021). Basic principles of vacuum technology. Retrieved from https://www.festo.com/net/SupportPortal/Files/286804/Basic_Vacuum_Technology_Prin ciples.pdf 138 Gerber, C. E. (1985). United States Patent No. 4501113. Gray, G. (2007). United States Patent No. 7540137 B2. Hemming, J., van Tuijl, B. A. J., Gauchel, W., & Wais, E. (2014, August). Field test of different end-effectors for robotic harvesting of sweet-pepper. In XXIX International Horticultural Congress on Horticulture: Sustaining Lives, Livelihoods and Landscapes (IHC2014): 1130 (pp. 567-574). Iida, F., & Laschi, C. (2011). Soft Robotics: Challenges and Perspectives. Procedia Computer Science, 7, 99-102. doi:https://doi.org/10.1016/j.procs.2011.12.030 Jou, J. H., Wu, M. H., Shen, S. M., Wang, H. C., Chen, S. Z., Chen, S. H., ... & Hsieh, Y. L. (2009). Sunlight-style color-temperature tunable organic light-emitting diode. Applied Physics Letters, 95(1), 184. doi:10.1063/1.3176217 Kalisky, T., Wang, Y., Shih, B., Drotman, D., Jadhav, S., Aronoff-Spencer, E., & Tolley, M. T. (2017, September). Differential pressure control of 3D printed soft fluidic actuators. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 6207-6213). IEEE. Kessens, C. C., & Desai, J. P. (2010, May). Design, fabrication, and implementation of self- sealing suction cup arrays for grasping. In 2010 IEEE International Conference on Robotics and Automation (pp. 765-770). IEEE. Kondo, N., Yata, K., Iida, M., Shiigi, T., Monta, M., Kurita, M., & Omori, H. (2010). Development of an End-Effector for a Tomato Cluster Harvesting Robot. Engineering in Agriculture, Environment and Food, 3(1), 20-24. doi:10.11165/eaef.3.20 Kourtev, H. (2018). A robust soft and vacuum hybrid end-effector and compliant arm for picking in clutter (Doctoral dissertation, Rutgers University-School of Graduate Studies). Lee, J. G., & Rodrigue, H. (2019). Efficiency of origami-based vacuum pneumatic artificial muscle for off-grid operation. International Journal of Precision Engineering and Manufacturing-Green Technology, 6(4), 789-797. doi:10.1007/s40684-019-00142-0 Lee, Y., Lan, C., Chu, C., Lai, C., & Chen, Y. (2013). A Pan–Tilt Orienting Mechanism With Parallel Axes of Flexural Actuation. IEEE/ASME Transactions on Mechatronics, 18(3), 1100-1112. doi:10.1109/TMECH.2012.2195192 Li, J., Karkee, M., Zhang, Q., Xiao, K., & Feng, T. (2016). Characterizing apple picking patterns for robotic harvesting. Computers and Electronics in Agriculture, 127, 633-640. doi:10.1016 139 Liu, J., Li, P., & Mao, H. (2015). Mechanical and kinematic modeling of assistant vacuum sucking and pulling operation oftomato fruits in robotic harvesting. Transactions of the ASABE, 58(3), 539-550. Lu, M. N., Su, T. S., Pylnev, M., Long, Y. S., Wu, T. C., Tsai, M. A., & Wei, T. C. (2021). Stepwise optimizing photovoltaic performance of dye‐sensitized cells made under 50‐lux dim light. Progress in Photovoltaics: Research and Applications, 29(5), 533-545. Lu, R., Li, Z., and Lammers, K. 2020. System and method for harvesting fruit. U.S. Patent Application #62,982,833. 2020 (pending). Macidull, J. C. (1972). United States Patent No. 3756001. Miller, J. T., & Wicks, N. (2018). Vacuum-Actuated Bending for Grasping. Robotics, 7(4), 73. doi:10.3390/robotics7040073 Murray, R. M., Li, Z., & Sastry, S. S. (1994). A Methematical Introduction to Robotic Manipulation (1st Edition ed.). Boca Raton: CRC Press. Nakamoto, H., Ohtake, M., Komoda, K., Sugahara, A., & Ogawa, A. (2018, October). A gripper system for robustly picking various objects placed densely by suction and pinching. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 6093-6098). IEEE. Parker, M., Unrath, R.C., Safely, C., Lockwood, D. (Jan.1, 1998) High Density Apple Orchard Management. NC State Extension Publications. Retrieved from https://content.ces.ncsu.edu/high-density-apple-orchard-management Pothula, A. K., Zhang, Z., & Lu, R. (2018). Design Features and Bruise Evaluation of an Apple Harvest and In-Field Presorting Machine. Transactions of the Asabe, 61(3), 1135-1144. doi:https://doi.org/10.13031/trans.12327 Puchalski, C., Brusewitz, G. H., & Slipek, Z. (2003). Coefficients of friction for apple on various surfaces as affected by velocity. Agricultural Engineering International: CIGR Journal. Robertson, M. A., & Paik, J. (2017). New soft robots really suck: Vacuum-powered systems empower diverse capabilities. Science Robotics, 2(9), 11. doi:10.1126/scirobotics.aan6357 Roper, T.R., Frank G.G. (1992). Planning and Establishing Commercial Apple Orchards in Wisconsin. University of Wisonsin- Extension Cooperative. A3560. Retrieved From Planning and Establishing Commercial Apple Orchards in Wisconsin (A3560) Scharfy, D., Boccali, N., & Stucki, M. (2017). Clean Technologies in Agriculture-How to Prioritise Measures? Sustainability, 9(8), 1303. doi:10.3390/su9081303 140 Shamshiri, R., Weltzien, C., Hameed, I., Yule, I., Grift, T., Balasundram, S., . . . Chowdhary, G. (2018). Research and development in agricultural robotics: A perspective of digital farming. International Journal of Agricultural and Biological Engineering, 11, 1-14. doi:10.25165/j.ijabe.20181104.4278 Shi, Y., Guo, Y., Liu, L., Zhao, J., Chen, J., & Cui, Y. (2016). Design of an End Effector For Crawling Roundlike Fruits. Academic Journal of Manufacturing Engineering, 14(1). Retrieved from https://www.researchgate.net/profile/Yinggang_Shi/publication/309635302_Design_of_a n_end_effector_for_crawling_roundlike_fruits/links/590d4f5b0f7e9b2863a1de3c/Design -of-an-end-effector-for-crawling-roundlike-fruits.pdf Shi, Y., Zhu, K., Zhai, S., Zhang, D., Liu, L., Zhao, J., . . . Cui, Y. (2018). Design of an Apple- Picking End Effector. Journal of Mechanical Engineering, 64, 1-9. doi:10.5545/sv- jme.2017.5084 Silwal, A., Davidson, J., Karkee, M., Mo, C., Zhang, Q., & Lewis, K. (2016). Effort towards robotic apple harvesting in Washington State. In 2016 ASABE Annual International Meeting (p. 1). American Society of Agricultural and Biological Engineers. Silwal, A., Davidson, J. R., Karkee, M., Mo, C., Zhang, Q., & Lewis, K. (2017). Design, integration, and field evaluation of a robotic apple harvester. Journal of Field Robotics, 34(6), 1140-1159. doi:10.1002/rob.21715 Tawk, C., Panhuis, M. I. H., Spinks, G. M., & Alici, G. (2018). Bioinspired 3D Printable Soft Vacuum Actuators for Locomotion Robots, Grippers and Artificial Muscles. Soft Robotics, 5(6), 685-694. doi:10.1089/soro.2018.0021 Water Pressure Sensor G1/4 1.2Mpa (2021). seeed: The IoT Hardware enabler. Retrieved from https://www.seeedstudio.com/Water-Pressure-Sensor-G1-4-1-2MPa-p-2887.html Whitesides, G. M. (2018). Soft Robotics. Angewandte Chemie (International ed.), 57(16), 4258- 4273. doi:10.1002/anie.201800907 You, K., & Francis Burks, T. (2016). Development of A Robotic Fruit Picking End Effector and An Adaptable Controller. Paper presented at the 2016 ASABE Annual International Meeting, St. Joseph, MI. http://elibrary.asabe.org/abstract.asp?aid=47426&t=5 Zhang, K., Lammers, K., Chu, P., Li, Z., & Lu, R. (2021). System design and control of an apple harvesting robot. Mechatronics, 79, 102644. Zhang, Z., Pothula, A. K., & Lu, R. (2018). A Review of Bin Filling Technologies for Apple Harvest and Postharvest Handling. Applied Engineering in Agriculture, 34(4), 687-703. doi:https://doi.org/10.13031/aea.12827 141 Zhang, Z., Zhang, Z. H., Wang, X. M., Liu, H., Wang, Y. J., & Wang, W. J. (2019). Models for economic evaluation of multi-purpose apple harvest platform and software development. International Journal of Agricultural and Biological Engineering, 12(1), 74-83. doi:10.25165/j.ijabe.20191201.4360 142