BIO-INSPIRED SOFT ROBOTS: DESIGN, MODELING, AND CONTROL By Xinda Qi A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of Electrical and Computer Engineering—Doctor of Philosophy 2024 ABSTRACT Soft robots are developed and studied for their safety and adaptability in various applica- tions. Compared to their rigid counterparts, soft robots can use their deformable bodies to adapt to challenging environments and tolerate collisions and inaccuracies. Natural animals, due to their intrinsic softness, have become popular bio-inspirations for many soft robots, which, in turn, could provide insights into biomechanics. Snakes, known for their adaptability and flexibility, inspire the development of limb- less mobile robots for tasks in complex environments. Inspired by snakes, we propose a novel pneumatic soft snake robot that uses traveling-wave deformation to navigate com- plex, constrained environments such as pipeline systems. The unique pneumatic system in the modular snake robot generates traveling-wave deformation with only four independent air channels. Experimental results show good agreement with finite element method (FEM) predictions and demonstrate the robot’s adaptability in complex pipeline systems. Addition- ally, a spiral-type soft snake robot is proposed for more robust locomotion in constrained environments, utilizing helix-like deformation for propulsion. Besides the locomotion in constrained environments, we develop a multi-material 3D- printed snakeskin with orthotropic frictional anisotropy, inspired by real snakeskin, to enable undulatory slithering of the robot on planar rough surfaces. This snakeskin comprises a soft base with embedded rigid scales, mimicking real snakeskin. The designs generate various frictional anisotropies that propel the robot during serpentine locomotion. Experiments show effective serpentine locomotion on artificial and outdoor surfaces like canvas and grass. Given the complexity of the dynamic model of the snake robot’s serpentine locomotion, a model-free reinforcement learning approach is chosen for integrated locomotion and naviga- tion. We propose Back-stepping Experience Replay (BER) to enhance learning efficiency in systems with approximate reversibility, reducing the need for complex reward shaping. BER is used in the soft snake robot’s locomotion and navigation task, with a dynamic simulator assessing the algorithms’ effectiveness. The robot achieves a 100% success rate in learning, reaching random targets 48% faster than the best baseline approach. In addition to mobile robots, bio-inspired soft robots have been proposed for robotic manipulators, enabling safe and robust interactions with humans and delicate objects. In- spired by octopus tentacles, we design a multi-section cable-driven soft robotic arm with novel kinematic modeling. An analytical static model captures the interaction between the actuation cable and the soft silicone body. Experiments show good agreement with model predictions and the soft robotic arm demonstrates high flexibility and a large workspace for potential human-machine interaction applications. Copyright by XINDA QI 2024 To my beloved family and friends. v ACKNOWLEDGEMENTS I would like to express my sincere gratitude to my advisor, Prof. Xiaobo Tan, for providing me the opportunity to be part of his team in Smart Microsystems Lab during my doctoral study. His valuable guidance and generous encouragement benefited me a lot in my research and will continue to benefit me in my future career. I would also like to thank my committee members and lab mates for their kind help during my doctoral study. I would also like to acknowledge the support from the funding agencies that made this work possible: the MSU Strategic Partnership (Grant 16-SPG-Full-3236), the National Sci- ence Foundation (Awards CNS 2125484, ECCS 2024649, and CNS 2237577), and the US National Institutes of Health (Grant 1UF1NS115817-01). vi TABLE OF CONTENTS CHAPTER 1 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 CHAPTER 2 A NOVEL PNEUMATIC SOFT SNAKE ROBOT USING TRAVELING-WAVE LOCOMOTION IN CONSTRAINED ENVIRONMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . CHAPTER 3 BIO-INSPIRED 3D-PRINTED SOFT SNAKESKINS ENABLE EFFECTIVE SERPENTINE LOCOMOTION OF A SOFT ROBOTIC SNAKE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . CHAPTER 4 BACK-STEPPING EXPERIENCE REPLAY WITH APPLICATION TO MODEL-FREE REINFORCEMENT LEARNING FOR A SOFT SNAKE ROBOT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 32 60 CHAPTER 5 DESIGN AND MODELING OF A SOFT SPIRAL SNAKE ROBOT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 CHAPTER 6 DESIGN AND NON-LINEAR MODELING OF A MODULAR CABLE-DRIVEN SOFT ROBOTIC ARM . . . . . . . . . . . . . . . 100 CHAPTER 7 CONCLUSION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 BIBLIOGRAPHY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 vii CHAPTER 1 INTRODUCTION Soft robots have been widely proposed for their adaptability, safety in interaction, and great potential applications in different fields. Unlike their rigid counterpart, soft robots are able to utilize their deformable bodies to adapt to different challenging environments and provide soft buffers to tolerant collisions and inaccurate interactions. Many designs of the soft robots were inspired by animals, which use soft actuation and have compliant structures. For example, for the soft mobile robots that work in complex and challenging geometries, the snakes could offer various inspirations for the locomotion method and structural designs. For the soft robotic arms, their designs and actuation mechanisms could be inspired by the anatomy of octopus tentacles. In this chapter, the development of soft snake robots, design of soft robots inspired by biological structures, reinforcement learning for soft robots, and development of soft robotic arms are discussed. The state of the art in those areas is reviewed, followed by a summary of the contributions we have made. 1.1 Soft Snake Robots in Constrained Environments 1.1.1 State of the Art In recent years, a number of robots have been developed by using inspiration from animals due to their advanced and robust locomotion capabilities. One class of such animals are snakes, which are capable of moving in most of the complex environments on earth. The slim body structure of the snakes enables them to travel in narrow, constrained environments. Snake-like robots are thus promising for applications such as pipeline inspection, search and rescue in rubbles, and minimally invasive surgeries. Existing snake robots have predominantly adopted rigid modules. One classic snake robot utilized multiple motors to deform the rigid body, which demonstrated several dif- ferent locomotion modes in traversing a complex terrain [122, 60, 121], climbing [120], and 1 inspecting inside a pipeline [93]. Another multi-section snake robot achieved crawling on land and swimming in water [21]. Despite these progresses, there are some limitations for rigid snake robots. In particular, the size of a rigid snake robot is limited by the motors and complex mechanisms, which increases its difficulty in traversing a narrow environment. To address the aforementioned concern, several soft snake robots driven by fluid power have been proposed. For example, soft snake robots with wheels were developed and their locomotion model was presented [78, 77, 65, 66, 67]. Soft snake robots can use the wave of their body to generate lateral undulation and side-winding locomotion. In addition, other than robots on wheels, soft snake robots can use the contact between the body and the environment to move forward by utilizing the anisotropic friction [13, 11]. Compared with a rigid snake robot, the size of a soft snake robot could be smaller and there is a large contact surface between the robot and environment. However, most of the reported soft snake robots require independent control of each module/section, with separate fluid channels/tubings for each module, which presents significant challenges in practical applications due to the resulting complexity in system hardware and control implementation. 1.1.2 Contribution In this work, we propose a novel pneumatic soft snake robot that exploits traveling- wave motion to move in complex, constrained environments such as a pipeline. The robot is modular, with a unique pneumatic system design that requires the use of only four air channels regardless of the number of modules. The robot is 3D-printed, and thus low-cost and easy to build. Finite element modeling of the bending behavior of each module is conducted in ANSYS. The dynamic behavior of the robot, consisting of six modules, is further modeled in SOFA. In particular, it is found that the locomotion speed of the robot increases with the actuation pressure and decreases with the friction coefficient. Extensive experimental results on a snake robot prototype show good agreement with model predictions. The robot also demonstrates the capability of moving in constrained pipeline environments, including traveling in pipes of different diameters and challenging geometry such as a sharp elbow. 2 1.2 Artificial Snake Skins for Locomotion of Snake Robots 1.2.1 State of the Art Bio-inspired structures and actuators are utilized extensively to achieve various functions for robots, such as grasping and locomotion[71, 85, 49]. Animals have been sources of inspi- ration for a myriad of mechanism designs: biomimetic shark skins[118], fibrillar gecko pad adhesive interfaces[9], robotic water striders[53], and legged and underwater robots[135, 104]. In particular, the softness of robots inspired by the soft structures of animals, makes such robots more robust to environmental challenges[29]. Numerous soft robots have been devel- oped for different applications, such as inspection, human-machine interaction, and medical surgeries[51], inspiring ideas for designing more robust, human-friendly robotic solutions for daily tasks. Snake robots have been developed to achieve simple but robust locomotion mimicking limbless animals[89, 122, 22]. They are of particular interest in several scenarios, like mini- mally invasive surgery and rescue tasks, because of their compact structure and flexibility, enabling them to adapt to different environments[30, 50]. As one common and efficient locomotion method, serpentine motion or slithering of snakes relies on traveling-wave defor- mation of their bodies and the orthotropic friction anisotropy (distinct frictional properties in longitudinal and lateral directions) generated by their skins[39]. During the undulation, different parts of the snake have different velocity directions, and the angles formed between local velocity directions and local body orientations are different. The friction on the ventral surface thus has a varying magnitude along the body due to the friction anisotropy, with a resulting integral of the friction generating the propulsion for the serpentine locomotion.16 This locomotion method allows bio-inspired robots to use slim structures to advance in com- plex environments and was initially achieved by rigid-link snake robots, the deformation of which was activated by motors[43, 70]. Recently, several soft snake robots were created by using soft bending actuators, which enhances their flexibility[78, 63]. To generate friction anisotropy, passive wheels were widely used in these robots, with rolling and sliding frictions 3 in the orthogonal directions. However, the use of passive wheels compromises the adapt- ability of soft robots and limits their utility in field applications. Therefore, developing a compact soft snakeskin with orthotropic friction anisotropy is of significant interest. There have been various studies on incorporating functional surfaces into mobile robots and utilizing special textures to meet the needs of versatile applications[68]. Kirigami skins, cylindrical scale arrays, and barbed and sawtooth-like structures were proposed to drive soft crawling robots and provided friction anisotropies[91, 12, 99, 69, 7, 96, 46, 108, 94]. Movable skins were also implemented to achieve agile locomotion for snake robots at the cost of com- plex mechanisms[72, 33]. In particular, notable examples of functional structures for friction anisotropy include hook-like structures[47], snakeskin-inspired clay-structure interfaces[44], and spiky skin structures[45]. There have also been studies focusing on friction attributes of surfaces and skins[24, 82]. The complex friction anisotropies of different real snakeskins were analyzed[125, 37], and it was shown that the transverse friction of a snake is higher than the longitudinal frictions on rough surfaces[39]. Several bio-inspired surfaces and sawtooth-like structures were designed and tested by emulating the microstructure of snakeskins and other rough surfaces[106, 36, 80, 107]. 1.2.2 Contribution In this work, we present a multi-material 3D-printed snakeskin with orthotropic friction anisotropy, which permits undulatory slithering of a soft snake robot on rough surfaces. Such a snakeskin is composed of a soft skin base and embedded rigid scales attached to the robot’s ventral surface. The bioinspired designs of scale shapes and arrangements lead effectively to various types of anisotropic friction, and provide means of switching robot’s locomotion direction to be either the same as or opposite to the propagation direction of the traveling-wave undulation. Furthermore, steering of locomotion can be achieved by applying additional pressure bias in one air path to break symmetry of body deformation. We also successfully demonstrate the snake robot’s mobility on various outdoor rough substrates, including concrete surfaces and a grass lawn, as well as pipes of different dimensions and 4 materials, for potential field applications. 1.3 Reinforcement Learning for Soft Robots 1.3.1 State of Art As a promising decision-making approach, reinforcement learning (RL) has drawn increas- ing attention for its ability to solve complex control problems and achieve generalization in both virtual and physical tasks, as evidenced in various applications, such as chess games [97], quadrupedal locomotion [61], and autonomous driving [42]. Considering the inherent infinite degrees of freedom of soft robots and their complicated interactions with environ- ments [55], RL approaches were adopted for the control of complex soft robotic systems, such as soft manipulators [105, 48] and wheeled snake robots [62]. As a typical challenge for RL, especially in tasks where complicated behaviors are involved (such as soft robot applications), the learning efficiency suffers from the relatively large search space and the inherent difficulties of the tasks, which usually requires delicate reward shaping [76] to guide the policy optimization and to constrain the learning directions or the behavior styles. The RL agents have to successfully reach their goals for efficient learning before getting lost in numerous inefficient failure trials. Multiple strategies were proposed to address the hard exploration challenge with sparse rewards, including improving the exploration techniques for more versatile trajectories from intrinsic motivations [79, 83, 19, 8], and exploiting the information acquired from the undesired trails [3, 31, 25]. Compatible with these techniques that might improve learning efficiency, the motivation of BER proposed for off-policy RL is the human ability to solve problems forward (from the beginning to goal) and backward (from the goal to the beginning) simultaneously, which is different from the standard model-free RL algorithms that mostly rely on forward explo- ration. For example, in proving a complicated mathematical equation, an effective method is to derive the equation from both sides where the information of both the left-hand side (beginning) and the right-hand side (goal) is utilized, to which the reasoning process and the mechanism of BER are similar. 5 1.3.2 Contribution In this paper, a BER algorithm is introduced that allows the RL agent to explore bidi- rectionally, which is compatible with arbitrary off-policy RL algorithms. It is applicable for systems with approximate reversibility and with fixed or random goal setups. After an eval- uation of BER with a toy task, it is applied to the locomotion and navigation task of a soft snake robot. The developed algorithm is validated on a physics-based dynamic simulator with a computationally efficient serpentine locomotion model based on the system charac- teristic. Comprehensive experimental results demonstrate the effectiveness of the proposed RL framework with BER in learning the locomotion and navigation skills of the soft snake robot compared with other state-of-the-art benchmarks, indicating the potential of BER in general off-policy RL and soft robot control applications. 1.4 Spiral Soft Snake Robot with Soft Actuators 1.4.1 State of Art Soft robotics has emerged as a rapidly growing field due to the advantages of flexibility, adaptability, and safe interaction with the environments and humans [10, 115], making them suitable candidates for applications in medical surgery [112], rescue operations [119], and environmental monitoring [38]. Mimicking the versatile and efficient locomotion of biological snakes, soft snake robots can navigate through constrained and complex environments that are challenging for their rigid counterparts. The inherent softness and passive flexibility [113] enable them to achieve high maneuverability and adaptability in diverse environments. The snake robots were proposed for their slim body shape and limbless locomotion method, making them a nice robotic platform for constrained environment inspections [122, 60]. Also, the simplicity of the snake’s body without a complex limb or wheel structure sim- plifies the robotic system design and also makes the system more robust compared with the other robot systems with complex propelling mechanisms [89, 120]. Thus, multiple-section snake robots were proposed to generate the movements of the snakes and achieve various locomotion in different scenarios. One classic type was the rigid multi-link snake robots, 6 where the deformation of the body was controlled by the embedded motors between the ad- jacent links [21, 78, 67]. By controlling the motion of the actuation motors between different links, the snake robots could generate different locomotion, such as serpentine, side-winding, and rolling for different environments and applications. Soft snake robots with spiral and helical locomotion patterns were introduced for improvements in navigation through irregu- lar surfaces and constrained environments. Twisted inflatable tubing was used to generate the helix shape of the robot for in-pipe locomotion [100]. Both soft snake robots [5] and rigid snake robots [123] are able to independently control the bending deformation of each segment to generate a spiral body for side-winding locomotion. However, the multiple actu- ation tubings of the soft snake robot with independent controls increases the structure and control complexity of the snake robot and limit its potential for tasks such as surgery. In addition to the slim and limbless body shape, the intrinsic softness and flexibility of the snake body also benefited the robot’s navigation in complex and challenging environments. By using the soft actuator to replace the rigid actuation mechanism, the snake robot obtained extra passive adaptability for the unexpected environment challenge. Several pneumatic- driven soft snake robots with bending actuators on the two body sides were proposed to generate serpentine locomotion on flat surfaces with the help of wheels or artificial snake skins [89, 78]. Besides, more complex soft snake robots [5] were designed with three bending actuators in one link to achieve more complex movement and locomotion, where each air chamber was controlled independently to achieve various gaits and movements on flat surfaces and a numerical method was used to analyze the locomotion of the snake robot [4, 6, 111]. For applications in constrained environments, a 3D printed soft robot that could generate 2D traveling-wave deformation was proposed for pipeline inspections [89], while a twisted locomotive device using a spiral gait was designed to act as an in-pipe mobile robot [101, 102, 103], where an analytical model estimated the helix shape of the twisted device. 7 1.4.2 Contribution In this work, we design a pneumatic modular snake robot that can generate a 3D helix- type traveling-wave deformation. Compared with other snake robots which could generate multiple 3D deformation modes, the proposed robot utilizes a specially designed embedded pneumatic system to simplify the robot structure by only using three independent input air paths, which improves the adaptation of the snake robot for the constrained environment and endows more potential of the robot for the miniaturization in medical applications. The proposed snake robot is modular and made of identical bending actuators that are driven by the three independent air channels, making the structure of the snake robot compact while the length of the snake robot can be tuned efficiently. The designed process for casting each section of the snake robot achieves an efficient and low-cost fabrication. The design of the bending actuator and casting method with silicone material improves the softness and flexibility of the snake robot in all transverse bending orientations, improving the adaptability of the snake robot for a complex pipeline system and on irregular terrains. 1.5 Cable-Driven Soft Robotic Arm Inspired by Octopus 1.5.1 State of the Art Soft robotic arms and manipulators have been widely proposed and developed for their various advantages, such as safe human-machine interactions, robustness, and flexibility[34, 128, 136, 56, 81]. Compared with their fully rigid counterparts, soft manipulators are able to utilize the compliance and softness of their body structures to adapt to external collisions and constraints, the potential threat to humans, while capable of accomplishing traditional manipulation tasks [58, 114, 124]. The advantages of the soft manipulators make them com- petitive candidates for applications involving the handling of delicate and complex objects, such as those in the food and medical industry [20, 26, 116]. Multiple structures and actuation methods have been developed for building soft robotics arms to achieve compliant and efficient deformation. For example, the fluid-driven method was widely used for soft actuators, where the fluid pressures inside actuator chambers were 8 controlled for generating elastic deformation[17, 110, 89]. Other types of soft actuators have also been constructed by replacing the fluid actuation with other actuation mechanisms[129, 2, 131, 54, 52], such as electrical polymers and shape memory materials. Besides, the cable- driven method was another popular actuation method thanks to its simplicity and high force-to-weight ratio[57, 27, 23], where embedded eccentric cables driven by external motors delivered torques to achieve deformation of the soft body. To control the deformation of the soft arm efficiently, models that capture the relationship between the actuation space and the task space of the robotic arms have been developed. This modeling process was generally complex and often dependent on different designs and actuation methods. Models for fluid-driven actuators were built based on static and dy- namic analysis[75, 127]. For cable-driven actuators, kinematic models based on geometry relationships were built for a simple cable actuation system[117]. Static models for the tendon-driven soft manipulator were also proposed to analyze the deformation of the elastic tendons[15, 14]. Besides, models were built for the coupling and decoupling cable system of multi-section soft manipulators[117, 92]. Piece-wise constant curvature models were widely utilized due to their simplicity, while other models, such as FEM models and the Cosserat rod model were proposed with better accuracy but higher complexity [74, 35, 109, 130]. 1.5.2 Contribution In this work, we propose a multi-section cable-driven soft robotic arm inspired by octopus tentacles with novel kinematic modeling. Each section of the modular manipulator is made of a soft tubing backbone, a soft silicon arm body, and two rigid endcaps, which connect adjacent sections and decouple the actuation cables of different sections. The soft robotic arm is made by casting after the rigid endcap structures were 3D printed, achieving low-cost and convenient fabrication. To capture an immerse effect of cables into the soft silicon arm body, which results from removing the intermediate rigid cable guide for better softness and safety, an analytical static model is built, the prediction of which shows great correspondence with experiment results. A kinematic model of the multi-section arm is then developed by taking 9 piece-wise constant curvature assumption for its motion planning algorithm. The proposed soft arm demonstrates high flexibility and a large workspace for its potential human-machine interaction applications. 10 CHAPTER 2 A NOVEL PNEUMATIC SOFT SNAKE ROBOT USING TRAVELING-WAVE LOCOMOTION IN CONSTRAINED ENVIRONMENTS In this chapter, we present a novel pneumatic soft snake robot which is able to navigate in constrained environments such as pipeline, by utilizing traveling-wave locomotion method which is inspired by the slithering movement of snakes. The soft snake robot is modular, and each section of which is made of Filaflex 82A via 3D printing and is assembled together after the fabrication. The soft snake robot is driven by air pressures inside different air channels which is controlled by using a pneumatic control board. The FEM simulation and the experiments validate the traveling-wave deformation of the snake robot and robot’s locomotion in constrained environments such as a complex pipeline system. 2.1 Design and Fabrication of the Snake Robot 2.1.1 Robot Body Design As a potential sensing platform for complex and constrained environments, the snake robot proposed in this work has a simple and compact structure made of soft material. There are no rigid joints on the robot, minimizing the risk of failure. Also, without tubing on the sides of the body, the simple shape of the snake robot makes it easy to traverse a constrained environment. The body of the soft snake, as illustrated by a prototype in Fig. 2.1, consists of multiple modular sections connected together (6 in the example here). Each section is a pneumatic bending actuator. The bending actuators share the same structure in general, but differ on the internal routing of air pathways. For each pneumatic bending actuator, there are four separate air chambers and they are located on two sides of the robot’s body, as shown in Fig. 2.2. The soft bellow surfaces of the snake body are used to amplify the deformation magnitude of the bending actuator and to improve the contact between the snake and complex working environments. 11 Figure 2.1 A soft snake robot prototype developed in this work. Figure 2.2 Pneumatic actuator (modular section) of the robot body. The four air chambers enable the actuator to have four different bending modes in the Z direction when different pairs of chambers are pressurized. The shapes of these bending modes are illustrated in Fig. 2.3. In particular, the actuator will bend in modes A and C when the chambers 1,2 and chambers 3,4 are actuated, respectively, and it will deform in modes B and D when chambers 1,4 and chambers 2,3 are actuated, respectively. This design provides the basis for the traveling-wave generation in the spatial domain, because the four bending modes comprise one period of the serpentine wave, as shown in Fig. 2.3. The robot is comprised of the modular actuators linked together, with the corresponding air pathways designed such that modes A - D are concatenated as illustrated in Fig. 2.3. The robot can be made longer by adding modules, as long as the periodic sequencing of modes A - D is preserved. In this work, a robot consisting of six bending actuator modules is considered, to 12 maintain at least two points of contact with the environment at any given time. Figure 2.3 Different bending modes of the actuators constitute a spatial serpentine wave. 2.1.2 Pneumatic System Design Different from previous works on soft snake robots, only four air paths are required in the proposed robot for the control of all the air chambers of the robot, regardless of the total number of bending actuator modules. This is important since it significantly reduces the complexity of hardware (valving, tubing) and control. To prevent external tubing from interfering with the movement of the snake robot (es- pecially in a constrained environment) and to minimize the total size of the robot, the air paths are designed to stay inside the snake body, as shown in Fig. 2.2. The four air paths are connected to the pneumatic source via tubing on the last bending actuator (the tail of the robot). The chambers and air paths are connected through several windows, which is the only difference between the four types of actuators, as shown in Fig. 2.2. Each air pathway is connected to one and only one of the chambers in each actuator. The actuators in the snake body are classified into four types which are denoted as 1st, 2nd, 3rd, 4th in order, as shown in Fig. 2.4, and they constitute one period of the snake body. The linkage relationships between the air paths and the chambers in different types of actuators are shown in Table 2.1. A sinusoidal traveling-wave equation, which is used to approximate the movement of the robot, is characterized by: z = A sin(ωt + kx) (2.1) 13 Figure 2.4 One period of the snake robot’s body. Table 2.1 Connection between Air Paths and Chambers in Different Actuators in One Period of the Robot Body. Air Paths 1 2 3 4 3 Chambers in different actuator 4 1 2 4th 3rd 2nd 1st 3rd 2nd 1st 4th 2nd 4th 1st 3rd 2nd 1st 3rd 4th which can be decomposed into the space domain and the time domain, separately:   z = A sin(ωt + θ1), when x is fixed  z = A sin(kx + θ2), when t is fixed (2.2) To illustrate the effect of the pneumatic system design of the robot, the snake robot’s body configuration is studied discretely when the 1st actuator is under the deformation mode A, B, C, D, which are the deformation modes at four representative times in one period of a traveling-wave, as shown in Fig. 2.5. The 1st actuator is under bending mode A when the air pathways 1 and 4 and the linked chambers are pressurized, as shown in Fig. 2.6. At the same time, the chambers 2 and 3 in the 2nd actuator are actuated, which make the 2nd actuator deform in mode B. As the Fig. 2.6 shows, the 3rd actuator and the 4th actuator deform in modes C and D under this condition, respectively. Thus, in the body length direction of the snake robot, the deformation modes A, B, C, D constitute one wavelength of the serpentine wave. Because of the periodicity of the snake robot modules, the rest of the body will continuously propagate the wave, which shows that the wave in space can be generated by the pneumatic system design under the condition that the 1st actuator has deformation mode A. 14 Figure 2.5 Deformation modes of the 1st actuator constituting a time-domain serpentine wave. Figure 2.6 Chambers linked to air path 1 (yellow) and air path 4 (green) in serpentine wave of the robot body. Similarly, it can be verified that when the 1st actuator deforms in modes B, C, D, the following actuators will deform accordingly in the phase-lagged modes to preserve a serpentine shape. This analysis shows that the pneumatic system design automatically decouples the traveling-wave generation into the spatial wave design, as shown in Fig. 2.3, and bending mode variations of one actuator in the time domain, as shown in Fig. 2.5. Such decoupling simplifies both the robot design and the control implementation. In addition, using just 4 air paths to control all the chambers in the long snake robot makes the system easy to fabricate and improves its operation robustness. 2.1.3 Prototyping and Fabrication In prior work on soft robots, casting is a frequently used method for fabrication because the material used in casting is soft and results in large deformation [32]. However, the casting method requires complex and accurate operations and it is difficult to fabricate complex internal structures of a soft body. In this work we demonstrate the fabrication of 15 the entire snake robot body, including its intricate air pathways, through 3D printing, which is another popular way to fabricate pneumatic soft actuator [132, 84]. The fused deposition modeling (FDM) 3D printer is selected to print the snake body because it is low-cost and it can print without support material by choosing proper parameters and designing a structure without large suspended areas. Although 3D printing with support material could have higher accuracy, unless the support material is soluble in solutions that do not affect the rest of the print, it would be extremely hard to remove the material and complete the fabrication, due to the complex structure inside the snake robot. Filaflex® 82A, a thermoplastic polyether-polyurethane elastomer (TPE) material, is used to print the body of the snake robot (i.e., the modular pneumatic actuators) because it has a sound balance between softness and elasticity. The dynamic requirement of the snake robot prevents the choice of a material that is too soft and exhibits excessive damping. The placement of the actuator and the printing procedure are shown in Fig. 2.7. Most of the major parts of the structure in the snake body (for example, the bellow walls on two sides of the body) can support themselves during the printing. However, there is still one upper surface that does not have enough support. So, the parameters including wall thickness, printing speed and extrusion multiplier are essential to avoiding under-extrusion for the last layers and preventing leakage problems. The 3D printer QIDI TECH ® is used for fabrication and some key parameters used in 3D printing of the soft snake robot are shown in Table 2.2. Figure 2.7 3D printing of the snake body: (a) Starting of the printing; (b) In printing; (c) Completion of the printing. The printer used in the fabrication has a build plate of 230×150 mm2, which limits the 16 Table 2.2 Key Parameters in FDM 3D Printing. Parameters Nozzle diameter Extrusion width Extruder temperature Extruder multiplier Printing speed Values 0.4 mm 0.45 mm 205 ◦C 1.3 900 mm/min length of the printed objects. Thus, the different actuators of the snake robot are printed separately and then connected in the designed order by short link tubing, which connects air paths of neighboring actuators, as shown in Fig. 2.8. Finally, the flexible glue Sikaflex® is applied to fix and seal the connections. Figure 2.8 Assembling process of different actuators of the snake body. 2.2 Control Scheme and Actuation System Control signals for the pressures in the four air channels are designed to achieve traveling- wave motion of the snake robot. One period of pressure signals is shown in Fig. 2.9, which are expressed as relative values with respect to the atmosphere. The bending modes of the 1st actuator when the air paths are actuated by the designed pressure control signals are shown in Fig. 2.10. At the time nT, (n+1/4)T, (n+2/4)T and (n+3/4)T, the actuator shows bending modes A, B, C, D, respectively, which satisfies the serpentine wave designed in the time domain shown in Fig. 2.5. After the pressure values at the four discrete times are designed, linear interpolation is used to generate piece-wise linear pressure signals. It is noticed that the pressure control signals have a quarter period phase lag sequentially 17 Figure 2.9 Pressure control signals for four air paths: (a) Air path 1; (b) Air path 2; (c) Air path 3; (d) Air path 4. Here n = 0, 1, 2. . . and T is the time period. Figure 2.10 Bending modes of 1st actuator in one period of traveling-wave. 18 from air path 1 to path 4, which influences the propagation direction of the traveling wave. If the control signals have a quarter period phase lead sequentially, the traveling wave will change its propagation direction to the opposite, which means that the snake robot can move forward and backward by altering the phase relationship between the pressure signals. Fig. 2.11 shows the schematic of the pressure control system. The Arduino Mega® is selected as the micro controller for the computation and communication of the system. PID controllers are used to track the desired pressure signals in a feedback system by modulating the PWM signals to four MOSFETs which are used for power amplification. The PWM signals are sent to solenoid valves (SMC VQ110U-5M) to control the airflow and the pressure in each of the four channels. Honeywell® silicon pressure sensors “ASDXAVX100PGAA5” are used to provide the feedback. Figure 2.11 Architecture of the pressure control board. 2.3 Bending Modes Analysis The finite element method (FEM) modeling and simulation are valuable tools for the analysis of soft robots [86, 98]. FEM modeling and simulation was conducted for static analysis of the four-chamber bending actuator module and a number of experiments were conducted to validate the FEM modeling and simulation. The bending modes of the actuators are the basic deformation elements for generating 19 traveling waves for the robot. Thus, FEM modeling and simulations were used to verify the bending modes of the actuators. The software ANSYS Workbench® and the toolbox “Static Structural” were used, where the material parameters were derived from the technical sheet of the printing material; see Table II, III. Because of the relative low strain estimated for the actuator’s deformation, a linear elastic model is used for the soft material, with Young’s modules of 12.5 MPa and Poisson ratio of 0.49. Table 2.3 Properties of the Soft Material Used for the Robot. Soft Material Density Stress at 20% elongation Stress at 100% elongation 6 MPa Stress at 200% elongation 10 MPa 45 MPa Tensile strength Filaflex 82A 1.12 g/cm3 2.5 MPa In the four bending modes, modes A and C are symmetric of each other, and so are modes B and D. Fig. 2.12 shows the simulated bending mode C and its experimental counterpart, where the applied pressures (for chambers 1 and 2) were 200 kPa. The bending angle θ between the horizontal axis and the tangent line of the actuator’s mid-line at the right end is used to compare the deformations of mode C in simulations and experiments, as shown in Fig. 2.13. Fig. 2.14 shows the corresponding comparison between simulations and experiments for mode D, where the pressures were applied in chambers 1 and 4. Because the bending angle θ in mode D cannot describe the deformation efficiently, the displacement d of the right end of the actuator in the -Z direction is used to compare the simulations and experiments, as shown in Fig. 2.15. From the simulations and the experiments, it is verified that by applying pressure in specific chambers, the bending modes A, B, C, D can be generated as designed. Overall the FEM modeling is shown to capture well the behavior of the experimental prototype. In particular, the simulated bending angle shows good agreement with the experimental measurement. For the displacement curve, the discrepancy between the simulation and 20 experimental results starts to show when the applied pressure is above 100 kPa. This might be attributed to the hyper-elasticity of the soft material. Figure 2.12 The simulation and experiment of bending mode C when the applied pressure is 200 kPa : (a) the deformation in simulation, where the color indicates displacement in the Z-direction; (b) the deformation in experiment. Figure 2.13 The simulation and experimental results of the bending angle of the actuator in mode C. 2.4 Locomotion Analysis of the Snake Robot FEM modeling and simulations were also conducted for dynamic analysis of the loco- motion of the full snake robot in a constrained environment. In particular, a pipeline was considered for the constrained environment, given that an important potential application of soft snake robots is pipeline inspection. A number of experiments were conducted to val- 21 Figure 2.14 The simulation and experiment of bending mode D when the applied pressure is 200 kPa : (a) the deformation in simulation, where the color indicates displacement in the Z-direction; (b) the deformation in the experiment. Figure 2.15 The simulation and experiment results of the displacement in -Z direction of the actuator in mode D. idate the simulation and test the robot’s performance and capability for traversing different pipeline environments. 2.4.1 Dynamic Simulation of Robot Locomotion in Pipelines The traveling-wave motion of the snake body generates the relative movement between the contact points of the snake robot and the environment, as discussed in the previous papers on traveling-wave locomotion [133, 16]. Thus, friction forces at the changing contact area of the snake robot drive the robot to move in the same direction of the traveling wave, as shown in Fig. 2.16. Like the dynamics of a rolling wheel, the driving force fd and the resistance force fr are both provided by friction. FEM modeling of the dynamic locomotion for the soft snake robot has been conducted, 22 Figure 2.16 The simulation and experiment results of the displacement in -Z direction of the actuator in mode D. which is able to accommodate complex collision and frictional contact. The FEM framework SOFA (Simulation Open Framework Architecture) has been used in dynamic simulation of soft robots [1, 28]. It is an efficient tool for physics simulation and meets the requirements of this work; in particular, it is amenable to dynamic FEM simulation and complex collision calculation. SOFA and the SOFT ROBOTS plugin were used for robot locomotion simulation in the constrained environment (pipeline). The basic configuration of SOFA simulation is shown in Fig. 2.17. The red points represent the contact nodes between the snake and the pipeline, and the yellow point is the node whose time-trajectory was studied. Figure 2.17 Configuration of the SOFA simulation of the snake robot. Fig. 2.18 shows the traveling-wave motion of the snake robot within a 2-inch pipe, under the designed structure and pressure control signals, where the control signals’ period was 1 s. The material properties and the gravity field were assigned, and the time step used in simulation was set to be 0.01 s. The yellow curve in Fig. 2.18 shows the trajectory of the studied point A, which confirms that the robot moves in the same direction as the 23 Figure 2.18 The moving snake robot with a traveling-wave locomotion in the SOFA simula- tion, viewed from the -Z direction. traveling-wave of the robot body. From the dynamic simulation in SOFA, the trajectory data of point A is extracted, as shown in Fig. 2.19 and Fig. 2.20, when the friction coefficient between the robot and the pipeline is 0.65 and the maximum pressure of the control signal is 138 kPa (20 psi). The displacement per period of the snake robot is used as the average speed and is derived from the data in Fig. 2.20, which is about 16 mm/s when the period of traveling-wave is 1s. These results suggest that SOFA can be used to simulate the dynamic locomotion and analyze the robot performance. Figure 2.19 The trajectory of point A in the XOY plane in the SOFA simulation. 24 Figure 2.20 The position of point A in the X direction in the SOFA simulation. 2.4.2 Experiments of Robot’s Locomotion with Different Parameters A number of experiments have been conducted to test the performance of the soft snake robot and to validate the SOFA analysis. The experiment setup for the snake robot’s loco- motion in a pipeline is shown in Fig. 2.21. A 2-inch clear PVC pipe was first selected as the constrained environment because of its small diameter and transparent body. The snake robot was placed in the pipeline in the same posture with the simulation so it could undulate in the horizontal plane. The speed of the robot was derived from the recorded video during the experiments. Figure 2.21 Experiment setup for the snake robot locomotion. A series of experiments was performed to verify the locomotion of the snake robot in pipelines and investigate the factors that influence the speed of the soft snake robot in the pipeline, with which the simulation results from SOFA were compared. The period T of signals and traveling-waves in all simulations and experiments was 1 s, which means that 25 their frequency was 1 Hz. The first factor that influences the speed of the snake robot v is the maximum value pm of the pressure signals. Fig. 2.22 shows the simulation results in SOFA and experiment results of the snake robot’s locomotion. The friction coefficient µ between the clear PVC pipe and the soft material was measured to be 0.65, which was obtained as the ratio between the tensile force when pulling the robot along the pipe at a constant speed and the normal force between the robot and the pipe. The error bars of the experiment represent the means and the standard deviations of three runs of the experiments. Figure 2.22 The robot’s speed in SOFA simulations and the experiments when friction coef- ficient µ is 0.65. In the experiments, the speed of the robot increased from 0 to 18 mm/s when the maxi- mum pressure pm was increased from 0 to 172 kPa (25 psi). The corresponding simulation in SOFA showed that the robot’s speed changes from 0 to 16 mm/s. The experiments also showed there was a dead zone in the relationships between pm and v. Despite these minor discrepancies, overall the agreement between the simulation and experimental measurement is good. Another important factor for speed is the friction coefficients between robot and pipelines, which were derived from the friction measurement tests. The relationships between v and pm under different µ are shown in Fig. 2.24, Fig. 2.25 and Fig. 2.26, where a slippery polyethy- lene surface (µ = 0.53), a paper surface (µ = 1.10), and a metalized moisture-resistant polyester surface (µ = 1.49), were used as the inner surface of the pipeline, respectively. Fig. 26 2.27 shows the relationship between µ and v in a 2-inch pipeline when the pm was 138 kPa (20 psi) in SOFA simulations and the experiments. Figure 2.23 Different inner surfaces of pipeline used in the experiments: (a) µ = 0.53; (b) µ = 1.10; (c) µ = 1.49. From the simulations and the experiments, it can be seen that the speed of the soft snake robot decreases with an increasing friction coefficient for the contact between the robot and the environment. On the other hand, a higher friction coefficient is expected to result in higher payload capacity (before slip happens). Therefore, optimization of the frictional contact has to be taken into account multiple objectives. Figure 2.24 The robot’s speed in SOFA simulations and the experiments when friction coef- ficient µ is 0.53. 2.4.3 Robot Locomotion in Complex Constrained Environments The soft and compliant body of the snake robot makes it possible to adapt to different environments. To confirm that, we have performed experiments with pipelines of different 27 Figure 2.25 The robot’s speed in SOFA simulations and the experiments when friction coef- ficient µ is 1.10. Figure 2.26 The robot’s speed in SOFA simulations and the experiments when friction coef- ficient µ is 1.49. Figure 2.27 The robot’s speed in SOFA simulations and the experiments as a function of friction coefficient, when pm is 138 kPa. diameters. Fig. 2.28, Fig. 2.29, and Fig. 2.30 show the locomotion of the snake robot in pipes of diameters of 1.625 inches, 2 inches, and 6 inches, respectively. The robot’s average 28 speeds in the straight parts of the three pipes were about 9 mm/s, 18 mm/s and 35 mm/s, respectively. These results show that the robot’s speed increases with the diameter of the pipe, which can be explained by the fact that a larger pipe imposes less restriction on the robot, resulting in larger amplitude of the traveling-wave. Figure 2.28 The locomotion of the snake robot in a 1.625-inch pipeline. Figure 2.29 The locomotion of the snake robot in a 2-inch pipeline. In addition, the soft snake robot was capable of passing the bending pipeline and even a sharp elbow, which revealed its robust locomotion in complex constrained environments and potential applications in pipeline inspection. Fig. 2.31 shows the locomotion of the snake robot in a complex pipeline system. 2.5 Discussion A novel pneumatic soft snake robot was proposed in this work, which used a traveling- wave locomotion method moving in complex and constrained environments such as a pipeline system. The pneumatic system and the control scheme design enabled the snake robot to 29 Figure 2.30 The locomotion of the snake robot in a 6-inch pipeline. Figure 2.31 The movement of the snake robot in a complex pipeline system of 2-inch pipelines. generate traveling-wave motion by only using four air paths, which simplified the body struc- ture greatly and improve the snake robot’s ability to move in constrained environments. In fabrication, 3D printing was used to build the whole body of the snake and made the robot robust, low-cost, lightweight and easy to build. FEM modeling and static structural simu- lation in ANSYS were conducted for bending mode verification, while dynamic simulation in SOFA was conducted for robot locomotion. Overall satisfactory agreement was achieved between model predictions and experimental measurements. Extensive experiments were conducted to verify the robot’s locomotion in pipelines and examine the relationship between the speed of the robot and the amplitude of pressure 30 signals pm under different frictional contact conditions. In a 2-inch PVC pipe, the speed of the snake robot using traveling locomotion reached 18 mm/s, for the maximum pressure of 172 kPa (25 psi) and the friction coefficient µ of 0.65. The speed of the robot increased with the maximum pressure, but decreased with an increasing friction coefficient (at least for the range of friction coefficient tested in this work). Furthermore, additional experiments showed that the proposed robot moved smoothly in pipes of different parameters and of complex geometry including sharp turns. For future work, the design of the robot will be optimized based on FEM analysis using ANSYS and SOFA, the validity of which has been shown in this work. Factors to be optimized include geometry and dimensions of individual actuator modules, stiffness and frictional properties of the material, and the control signals. Multi-objective optimization will be pursued, to incorporate objectives in speed, payload capacity, power consumption, and size. We will also pursue integration of a compact pneumatic source, control board, and batteries with the robot to enable untethered operation and will study its ability to navigate inclined pipes and move on flat surfaces. We will further explore applications of the soft snake robot, such as inspection of gas distribution pipelines. These pipelines have relatively small diameters, often in the range of 2–4 inches, which makes it practically infeasible to deploy traditional rigid robots efficiently. 31 CHAPTER 3 BIO-INSPIRED 3D-PRINTED SOFT SNAKESKINS ENABLE EFFECTIVE SERPENTINE LOCOMOTION OF A SOFT ROBOTIC SNAKE In this chapter, we present a bio-inspired multi-material 3D-printed soft snakeskin with friction anisotropy which enables the serpentine locomotion of the soft snake robot on rough surfaces. The proposed snake skin is make of a rigid material and a soft rubber-like material to imitate the rigid snakes scales embedded in soft snake skin. The snake skin is attached to the ventral surface of the soft robotic snake to generate friction anisotropy and propels the snake robot during the serpentine locomotion. The steering of the robot’s locomotion can be achieved by applying auxiliary pressure bias to specific air channels for navigation. Various experiments show the effective serpentine locomotion of the robotic snake with the snakeskin on different surfaces, including artificial rough surfaces and outdoor rough substrates such as concrete ground and grass lawn. 3.1 Design and Fabrication of the Soft Robotic Snake In this work, inspired by the snake scales on soft skin base and by the structures with friction anisotropies [80, 107], we present a novel type of soft snakeskin with orthotropic friction anisotropy, which enables the serpentine locomotion of a pneumatic soft snake robot on rough surfaces when the robot’s traveling-wave deformation is activated. Compared with other common barbed or sawtooth-like structures [69, 7, 96, 46, 108, 94] which generate friction anisotropies in opposite directions, the combination of the soft skin and rigid scales in the proposed design achieved an orthotropic friction anisotropy on a highly deformable substrate. The softness of the skin base provided robustness and compliance to soft robots and the rigidity of the scales improved the anisotropies and wear resistance. In addition, compared with the passive wheels [43, 70] for the soft snake robot’s serpentine locomotion, the snakeskin was more compact and robust to adapt to different environments. The proposed soft snakeskins consisted of a soft base skin and embedded rigid scales (Fig. 3.1E). The rigid scales on skins, resembling the snake scales, provided friction anisotropy, while the 32 soft base skin provided the flexibility to cling on to the deformable robot body (Fig. 3.1F). A modular pneumatic soft snake robot [89] acted as the robot body that could generate traveling-wave deformations, to which the snakeskin was adhered by using silicone glue (Fig. 3.1A). The soft snake robot was composed of six soft pneumatic bending actuators and each of them contained four separated air chambers (Fig. 3.1D). There were four air paths in the robot and each of them was connected to one air chamber in each bending actuator. The linkages between the air paths and the chambers in actuators were carefully designed to generate wave-like bending deformation (Fig. 3.1B). Then, the traveling-wave deformation of the robot could be achieved by using the time-varying sinusoidal pressures with phase differences in the four air paths (Fig. 3.1C) that were generated by a pressure control board (Fig. 3.2). The traveling-wave’s propagation direction could be controlled by switching between phase-lagged and phase-advanced pressures. On the rough substrates, once the traveling-wave deformation of the snake robot with snakeskins was generated, the anisotropic friction between the substrates and the skins would propel the robot to achieve serpentine locomotion like real snakes. To apply the designated air pressures in different air paths that were linked to the air chambers of the soft snake robot, a pressure control board was used, whose architecture is shown in Fig. 3.2. An Arduino Mega 2560 microcontroller was used to generate controlling PWM signals and send them to the MOSFETs for power amplification. Then the PWM signals that came from the MOSFET were sent to the solenoid valves (VQ110U-5M, SMC) to control the air flows and the air pressures in the robot’s air paths. Pressure sensors (ASDXAVX100PGAA5, Honeywell) were used to measure the air pressures and send the feedback signals to the microcontroller to form a feedback control loop. PID controllers were implemented in the microcontroller for the feedback control to generate the desired pressure inputs in the four air paths in the snake robot to achieve different traveling-wave deformations. As two main parts of the soft snake robot prototype, the soft pneumatic snake robot 33 Figure 3.1 Bio-inspired soft snakeskin and the robot. (A), Soft snake robot with the proposed snakeskins (ventral side up). (B), Air chambers linked to four air paths: air path 1 (blue), air path 2 (yellow), air path 3 (green), and air path 4 (red) in the robot’s body with wave- like bending deformation. (C), Pressures with amplitude pm in four air paths to generate traveling-wave deformation, which generates the bending deformation in (B) at time tB. (D), One of the pneumatic bending actuators of the robot body. (E), The proposed snakeskin with a soft base and rigid scales. (F), The soft snakeskin clung to the soft actuator under bending deformation. body and the soft snakeskins were fabricated separately for the convenience of snakeskin replacement in testing the performance of various types of snakeskins. The soft pneumatic snake robot body was made of Filaflex 82A, a kind of thermoplastic polyether-polyurethane elastomer (TPE) material, and was produced by a fused deposition modeling (FDM) 3D printer (QIDI TECH I). The 3D printing parameters were carefully 34 Figure 3.2 Architecture of the pressure control board to generate the pressure input to the snake robot. tuned to achieve airtight elastic material printing for complex structures of the robot body, and the parameters were elaborated in our previous paper [89]. The six segments of the snake robot body were printed separately (Fig. 3.3A) and were then concatenated together by using short soft tubing for air channel connection, where a flexible glue Sikaflex was used for sealing and adhesion (Fig. 3.3B). The snakeskins were also fabricated with 3D printing (Fig. 3.3E) and an Objet Connex 350 3D printer was used, which could construct the complex objects layer by layer by jetting different photopolymerizable materials in one layer that was cured by the subsequent UV light irradiation. The 3D printer’s capability of printing multiple different materials for one part made the printer a convenient choice for the proposed snakeskins, which were composed of a soft skin base and rigid scale arrays implanted in the skin layer. Objet Tango Black, a soft rubber-like material, was selected to print the soft skin layer, and Objet Vero White, a rigid plastic material, was selected to print the rigid scale arrays. After cleaning the soft resin support material used in this 3D printing process, different types of snakeskins were obtained, which were compliant with the snake robot’s bending deformation. Before the snake body and snakeskins were combined, the belly of the snake robot was first modified into a flat surface. The unevenness of the snake robot’s belly could arise from 35 Figure 3.3 Fabrication method and materials for the snake robot prototype. (A) 3D-printing of the segments of the soft snake robot body. (B), Assembling of the soft snake robot body by using soft tubing and flexible glue. (C), Using flat surfaces and silicone glue to build a flat snake ventral surface. (D), The cured silicone glue forming a flat snake ventral surface after the support blocks are removed. (E), 3D-printing of different types of snakeskins. (F), Adhering snakeskins to the soft snake robot body with a flat ventral surface by using silicone glue. the assembly of different body segments and the curing process of the flexible glue. A flat belly surface of the snake robot would ensure the contact between each part of the snakeskin along the snake body and the substrate during the serpentine locomotion and thus improve its efficiency. To construct a flat ventral surface, a casting method was used and Ecoflex 36 00-30 was selected to be the construction material (Fig. 3.3C). First, several 3D-printed plastic blocks of the same size were distributed on a flat surface and the whole snake robot was placed on these blocks that had flat top surfaces. Then, the silicone glue Ecoflex 00-30 was added in the gaps that were between the blocks and the ventral surface of the snake robot. The silicone glue would not leak away from the gaps completely because of its high viscosity and surface tension. After the curing of the silicone glue, the supporting blocks were removed, and a flat soft surface of the snake belly was obtained after trimming (Fig. 3.3D). Finally, the soft snakeskins were adhered to each segment of the snake robot’s body by using Ecoflex 00-30, which completed the fabrication of the soft snake robot prototype (Fig. 3.3F). 3.2 Results 3.2.1 Orthotropic Friction Anisotropy of the Scales on the Snakeskins The robotic snakeskin proposed in this work adopted a hybrid structure: a soft skin base with rigid embedded scales. Three different shapes were proposed for the outer edge of the scales: triangular, circular, and trapezoidal (Fig. 3.5A). Each type of scale had width w = 5 mm, length l = 6 mm, and an opening angle α = 30 degrees between the skin base surface and scale surface. The presence of the opening angle allowed overlapping scales to cover the entire snakeskin and improved the flexibility of the skin in the vertical direction for uneven substrates. Furthermore, to secure the connection between the rigid scales and the soft base, an anchor structure was used at the inner end of the scale that was implanted in the soft skin base (Fig. 3.5A). Experiments were conducted to investigate the friction properties between the scales and different rough surfaces and examine the feasibility of using the proposed scales to generate orthotropic friction anisotropy. A cotton canvas layer and a paper layer, respectively, were used as the substrate surfaces for these experiments. To test and validate the orthotropic anisotropic friction between the proposed scales and the substrates (canvas and paper layers), an experimental setup for the friction force tests was designed (Fig. 3.4B). The canvas layer 37 used in this work was “Canvas Duck Natural” from “James Thompson”, and the paper layer used was “Slip Resistant Paper/Poly Drop Cloth” from “Everbilt” that had a blue paper top layer and white plastic fabric bottom layer. A linear stage driven by a step motor was used to drive the support structures that were linked to one end of the force sensor. Another end of the force sensor was connected to two sides of the test sample to pull it forward at a constant speed of 10 mm/s during the friction force experiments. The test sample consisted of a sample body, which acted as the structure linked to the force sensor, and a replaceable bottom, to which different scales were attached (Fig. 3.4A). The sample body and the replaceable bottom were 3D-printed separately by using the material “Objet Vero White” and were assembled and fixed by using two screws. The separation between the sample body and the replaceable bottom allowed the relative rotations between them and permitted the adjustment of the angle θ to get the distribution of the friction force of the scales in all directions. Fig. 3.4C showed the force balance of the test sample during a friction force experiment when the linear stage pulled the test sample to move forward. Force sensors were used to measure the pulling force P and the gravity force G, which were equal to the friction force f and the normal force N , respectively. Then, by changing the angle θ in different experiments, the distribution of the friction force fθ of the scales on different substrates with different θ was measured in the experiments. Fig. 3.4D showed one measurement of fθ during one experiment, where it could be noticed that the measured data at the beginning and the end of the measurement were different from the values when the friction force became stable. Thus, the mean friction force in this experiment was calculated by: fθ (cid:80) fθ∈S |S| fθ = (3.1) In the experiments, by choosing different angles θ between the scale direction and the moving direction of the scale samples pulled by a linear stage, the friction forces fθ between scales and substrates were measured when the moving directions of the scales were fixed (Fig. 38 3.5B). In particular, two different moving directions (along the fibers, which is smooth, versus transverse to the fibers, which is rough) on the canvas surface were selected because of the pronounced roughness anisotropy of the canvas. Figure 3.4 Experimental setups for friction force experiments. (A), The test sample with a replaceable bottom, to which the scales were attached, allowing the adjustment of angle θ. (B), The experimental setup for characterizing the friction force between the scales and the substrate. (C), The force balance of the test sample with scales in constant-speed friction force experiments, where the N , P , f , and G denote the normal pressure force, pulling force, friction force, and gravity, respectively. (D), Measurement of the friction force fθ during one experiment (circular scale on paper, θ being 90 degrees), where the S denotes the measurement set. 39 Figure 3.5 Friction anisotropy of scales of different shapes on various substrates. (A), Snake scales of different shapes: triangular, circular, and trapezoidal, with length l, width w, and opening angle α. (B), Substrates for experiments: paper layer, canvas layer when the scales moved in the smooth direction, and canvas layer when the scales moved in the rough direction, where v, θ, fθ denote the speed, angle, friction force, respectively. (C), Coefficients of friction µθ between scales and substrates as a function of the angle θ, where the columns and rows of the subfigures correspond to the scale types and the substrates in experiments, respectively. (D), Orthotropic friction anisotropy (F A∥) and reversed friction anisotropy (F A⊥) of different scales on the paper substrate. (E), Orthotropic and reversed friction anisotropy of different scales on the canvas substrate in the smooth direction. (F), Orthotropic and reversed friction anisotropy of different scales on the canvas substrate in the rough direction. 40 After the friction forces fθ between the scales and substrates were measured, the coeffi- cients of friction µθ of the scales on the corresponding substrates were calculated by: µθ = fθ N (3.2) where is the mean of fθ, and N is the normal force between the scales and the substrates. The distributions of µθ for the three types of scales on different substrates are shown in Fig. 3.5C, where the columns and rows of the subfigures correspond to the scale types and the substrates used in the experiments, respectively. Furthermore, based on the µθ distributions, the friction anisotropies of scales on substrates were calculated by: F A⊥ = F A∥ = µ0◦ µ90◦ µ0◦ µ180◦ (3.3) where F A⊥ and F A∥ are the orthotropic and the reversed friction anisotropy, respectively, µ0◦, µ90◦, µ180◦ are the µθ of the scales when θ is 0, 90 degrees, 180 degrees, respectively. The friction anisotropies of the scales on different substrates are shown in Fig. 3.5, D to F. In the experiment, it was shown that the proposed bio-inspired scales were able to gener- ate both orthotropic and reversed friction anisotropies on rough surfaces (Fig. 3.5, C to F). Most distributions of µθ for scales on rough substrates exhibited clear orthotropic anisotropy characterized by elliptical shapes, generally showing that µθ decreased with the angle θ be- tween 0 and 90 degrees and increased with the angle θ between 90 degrees and 180 degrees. Also, the distributions showed that µθ with θ between 180 and 360 degrees was symmetric to that between 0 and 180 degrees, which was resulted from the geometry symmetry of the proposed snake scales. In particular, µθ of the triangular scales remained almost the same when θ was between 90 and 180 degrees, which might be due to the shape of the contact areas (could be assumed as an ellipse) between the scales and the rough substrates. The eccentricity of the ellipse contact area between the triangular scale and substrates was much smaller than that of the circular or trapezoid scale because of their edge shape. Thus, this 41 more isotropic contact area of the triangular scale resulted in its almost constant µθ with θ between 90 and 180 degrees, where the mechanical interlocking effect was not dominating. The friction anisotropies between the scales and each substrate were also compared (Fig. 3.5, D to F). On all substrates, the orthotropic friction anisotropy of the scale increased when the shape changed from triangular to circular to trapezoidal, while the reversed friction anisotropy of the scale increased when the shape changed from circular to trapezoidal to triangular. It needs to be mentioned that the triangular scale exhibited the highest reversed friction anisotropy on all substrates, which might be attributed to a stronger mechanical interlocking effect brought by its sharper tip compared with edges of other scales. 3.2.2 Baseline Locomotion of Snake Robot with Isotropic Skins The previous literatures [39] showed friction anisotropy was necessary for snakes to achieve efficient serpentine locomotion. A foam layer (Super-Cushioning Polyurethane Foam Sheet, McMaster-Carr) was thus used as an isotropic snakeskin to provide a baseline for ex- amining the serpentine locomotion of the snake robot. The foam layer was much softer than the robot body and was adhered to the ventral surface of the robot by using a thin acrylic adhesive layer (Fig. 3.7A). In baseline experiments, the aforementioned paper and canvas surfaces were used, and pressures (Fig. 1C) with a period T of 1 s were used to generate the robot’s traveling-wave deformation. The coefficient of friction between the foam and pa- per substrate and that between the foam and canvas substrate were measured as 1.104 and 1.417, respectively, by using the same method as adopted for the measurement of friction properties for the proposed scales. During the locomotion experiments of the snake robot on the different substrates, 25 markers were attached evenly along the top of the snake robot to monitor the positions of different parts of the snake robot under a motion capture system (Fig. 3.6A). The motion capture system used in this work was “Opti-track”, including a set of infrared cameras and the related software, which could track the 3D positions of each marker in the projection space. Then, based on the position data of markers, the snake robot’s center of mass position, 42 main direction, and the local curvatures were calculated to provide useful information for the locomotion analysis of the snake robot and the conclusions. Figure 3.6 Experimental setups for locomotion experiments of the snake robot. (A), The motion capture system tracks the position data of the markers on the snake robot during the locomotion experiments. (B), The local curvature of the bending actuator of the snake robot body can be calculated by the position data of three markers. The position of the center of mass of the snake robot was calculated by: X = Y = 1 n 1 n n (cid:88) i=1 n (cid:88) i=1 Xi Yi (3.4) where X and Y are the X and Y position of the robot’s center of mass, respectively, Xi and Yi are the X and Y position of the ith marker, respectively, and n is the total number of markers on the snake robot body. Then, the main direction of the snake robot could be derived as: ϕi = arctan ϕ = 1 n − 1 Yi+1 − Yi Xi+1 − Xi n (cid:88) ϕi i=1 (3.5) where ϕ is the main direction of the snake robot and ϕ1 was the angle between the X-axis and the connection line that connected the ith marker and the (i + 1)th marker. Based on the tracking data of the robot’s center of mass position and the main angle, the speed and the trajectory of the soft snake robot could be calculated and obtained for further analysis. 43 The speed v and the angular speed ω of the robot measured in one experiment could be calculated by: (cid:113) (∆X)2 + (∆Y )2 ∆t ∆ϕ ∆t v = ω = (3.6) where ∆X and ∆Y are the displacements of the center of mass of the robot in X and Y directions during a time interval ∆t when the robot’s locomotion is stable, ∆ϕ is the changing of the main direction of the robot during the time interval ∆t . The error bars in the experiment results of v and ω represent the means and the standard deviations of v and ω of three runs for each locomotion experiment, respectively. Besides, the local curvatures of the snake robot could also be calculated by using the position tracking data of the markers. For the modular snake robot, each of the bending actuator (modular segment) in the robot could be considered as the combination of two curved parts, and each of the parts could be approximated by a curve with a constant curvature that could be calculated by using the position data of three markers (Fig. 3.6B). The curvature of the right half part of the bending actuator in Fig. 3.6B was calculated by using an osculating circle: κAC = 1 R R = |AC| (cid:112)2(1 − cos ∠AOC (3.7) where κAC is the curvature of the right half of the bending actuator (curvature of curve AC), R is the radius of the osculating circle O, |AC| is the distance between the marker A and C. Then, to solve the radius R, the angle ∠AOC could be calculated by: ∠AOC = 2π − 2∠ABC (3.8) where the angle ∠ABC and the distances between the markers could be derived as: 44 ∠ABC = arccos |AB|2 + |BC|2 − |AC|2 2|AB||BC| |AB|2 = (XA − XB)2 + (YA − YB)2 |BC|2 = (XB − XC)2 + (YB − YC)2 |AC|2 = (XA − XC)2 + (YA − YC)2 (3.9) where |AB| is the distance between the marker A and B, |BC| is the distance between the marker B and C, XA, XB, XC and YA, YB, YC are the X positions and Y positions of the marker A, B, and C during the locomotion experiments, respectively. Based on the above algorithm, the curvatures κi for each curve on the snake robot could be obtained. Then, the local maximums κi,m of the curvatures κi were found by using the function “argrelextrema” in Python (Fig. 3.7D). Then, the maximum curvature of the robot’s traveling-wave deformation κm in one experiment could be calculated by: κm = 1 2a 2a (cid:88) ( i=1 1 |Sl| (cid:88) κim) κim∈Sl (3.10) where a is the number of the bending actuators of the robot body, Sl is the measurement set of the κim in the time interval ∆t during the locomotion experiment. |Sl| is the size of the set Sl. The robot’s maximum curvature κm was used to describe the amplitude of the traveling- wave deformation of the robot, the larger the κm was, the larger the deformation was. The error bars in the experiment results of κm represent the means and standard deviations of κm of three runs for each locomotion experiment. In the baseline locomotion experiments, it was shown that the traveling-wave deformation of the robot was successfully activated by the pressure inputs, and the snake robot moved slowly because of the lack of friction anisotropy (Fig. 3.7, B and C). It was also shown that the robot’s maximum curvature κm, which was correlated with the traveling-wave’s amplitude, increased with the pressure amplitude pm of the pressure inputs (Fig. 3.7E). In addition, κm on the canvas was smaller than that on the paper when the same pm was used; this could be attributed to the larger resistance between the foam skin and the canvas substrate, 45 Figure 3.7 Snake robot with isotropic ventral surfaces and the baseline experiments. (A), Soft snake robot with isotropic foam skins on the ventral surface. (B), Movement of the snake robot on the canvas substrate when the pressure amplitude pm was 138 kPa. (C), Movement of the snake robot on the paper substrate when pm was 138 kPa. (D), The curvature κ and its local maximums of one segment of the snake robot on the canvas when pm was 138 kPa. (E), The relationships between pm and the maximum curvature κm in the robot’s locomotion experiments on two substrates. (G), The relationships between pm and the robot’s speed v in the robot’s locomotion experiments on two substrates. which showed that the interactions between the robot and substrates could influence the deformation of the robot itself. Moreover, the experimental results showed that the robot’s speed v increased with pm, with a maximum speed of 5.67 mm/s when pm was 172 kPa (25 psi) and the substrate was the paper (Fig. 3.7F). Note that the forward movement could be partially attributed to the propelling effect of the pneumatic input, which transferred some momentum to the robot. 3.2.3 Serpentine Locomotion of the Snake Robot with Snakeskins Following the experiments with the isotropic foam skin, the proposed soft snakeskins were implemented on the snake robot to generate serpentine locomotion, where the substrates, pressure inputs (except the phase relations of the pressures), experimental setups, and data processing methods were the same as those used in the baseline experiments. In particular, we could tune the phase relation of the pressure inputs (i.e., phase-lag or phase-lead) to generate traveling-wave deformation that propagates either forward or backward along the 46 robot’s body. Note that the orthotropic anisotropic friction of the snakeskin on substrates, a key factor in generating serpentine locomotion, was determined by both the shape of the scales (Fig. 3.5) and the arrangement direction of the scales on the skin base (Fig. 3.9, A to C). By using the µθ distributions of the scales and their specific arrangements on the soft base, the snakeskin’s coefficient of friction µs θ could be calculated by taking the mean of µθ of every scale on that snakeskin in the θ direction (Fig. 3.8). The distributions of coefficient of friction of snakeskins µs θ were calculated by using µtheta of the rigid scales and the arrangement of the scales on the skins. The snakeskin with orthogonally arranged circular scales on the paper substrate was used as an example to illustrate the calculation method (Fig. 3.8). For the proposed symmetric snakeskins, three polar coordinates, Σ0, Σ1, and Σ2, were defined, whose reference directions were the skin direction, the main direction of scales of the first row, and the main direction of the scales of the second row, respectively. Then, by assuming that the normal pressure force between the skin and the substrate during contacts and robot’s locomotion was distributed evenly on each scale on the skin, and based on the µθ1 distribution between the circular scale of the first row and paper (Fig. 3.8B), and the µθ2 distribution between the circular scale of the second row and paper (Fig. 3.8C), µs θ could be calculated by: µs θ = θ1 + µ0 µ0 θ2 2 (3.11) where µ0 θ1 is the distribution of coefficient of friction between the circular scales of the first row and the paper substrate in the coordinate Σ0, µ0 θ2 is the distribution of coefficient of friction between the circular scales of the second row and the paper substrate in the coordinate Σ0. Similarly, µs θ of other skins with different scale arrangements and shapes on different substrates could be calculated. Particularly, two different µθ values were measured between each skin and the canvas substrate along two perpendicular directions (smooth and rough directions) due to the can- vas’s roughness anisotropy. Thus, to derive µs θ on the canvas substrate, these two different µθ values were averaged into a single µθ first to describe the general friction properties of 47 the scales on canvas substrate, which could be calculated by: µθ(canvas) = µθ(smooth) + µθ(rough) 2 (3.12) where µθ(canvas) is the µθ value of the scales on the general canvas surface, µtheta (smooth) is µtheta of the scales on the canvas moving along the smooth direction, µθ(rough) is µθ of the scales on the canvas moving along the rough direction. Then, µθ(smooth) was used as µθ of the scales on the canvas substrate and used for the calculation of the µs θ value for different skins on the canvas substrate. Figure 3.8 Diagram to show the calculation of the coefficient of friction of snakeskins. (A), A snakeskin with orthogonally arranged circular scales where the skin coordinate was Σ0, the coordinate for the scales of the first row was Σ1, and the coordinate for the scales on the second row was Σ2. (B), The distribution of coefficient of friction µθ1 between the circular scale of the first row and the paper substrate. (C), The distribution of coefficient of friction µθ2 between the circular scale of the second row and the paper substrate. (D), The distribution of coefficient of friction µs θ between the skin with orthogonally arranged circular scales and the paper substrate. Furthermore, the friction anisotropies of the skins on different substrates could be cal- culated by utilizing the µs θ distributions of the skins. The orthotropic fiction anisotropies of 48 the skins with reversely and orthogonally arranged scales on different substrates could be calculated by: F As ⊥ = µs 90◦ µs 0◦ (3.13) where F As ⊥ is the orthotropic friction anisotropy of skins on different substrates, µs 0◦ is the θ value of the skins on different substrates when θ is 0, µs µs 90◦ is the µs t heta value of the skins on different substrates when θ is 90 degrees. The orthotropic fiction anisotropies between the skins with collaterally arranged scales and different substrates could be calculated by: µs 180◦ µs 0◦ ⊥ is the orthotropic friction anisotropy of the skin with collaterally arranged scales (3.14) F As ⊥ = where F As on the substrates, µs 90◦ is the µs θ value of the skin on substrates when θ is 90 degrees, µs 180◦ is the µs θ value of the skin on substrates when θ is 180 degrees. Different equations were used to evaluate orthotropic friction anisotropies for skins with differently arranged scales because: first, the propagation directions of the traveling-wave were different for the robot with differently arranged skins in the experiments; second, we would like to keep all the anisotropies greater than one for more intuitive comparations among them. To examine the locomotion performance of the robot with various snakeskins, three skins with different scale arrangements were tested first, where the circular scales, which resemble real snake scales the most, were adopted. The reversed scale arrangement (RA), collateral scale arrangement (CA), and orthogonal scale arrangement (OA) for the snakeskins were designed and used (Fig. 3.9, A, B, and C, resp.). All three arrangements had two rows of scales and adopted a mirror symmetry. The orientation difference between the scales of the two rows was 180 degrees, 0, and 90 degrees for RA, CA, and OA, respectively. The µs θ distributions of different snakeskins on the paper substrate were presented beneath the skin designs (Fig. 3.9, A to C). It was shown that the snakeskin with RA and CA had orthotropic anisotropies, where the largest µs θ was on 90 and 180 degrees direction, respectively; but for the snakeskin with OA, the distribution of µs θ was nearly isotropic. 49 Figure 3.9 Serpentine locomotion of the snake robot with differently arranged snakeskins with circular scales. (A), The coefficient of friction µs θ of the snakeskin with reversed scale arrangement (RA) and circular scales on the paper substrate. (B), The µs θ of the snakeskin with collateral scale arrangement (CA) and circular scales on the paper. (C), The µs θ of the snakeskin with orthogonal scale arrangement (OA) and circular scales on the paper. (D), Locomotion of the robot with RA on the canvas substrate when the pressure amplitude pm was 138 kPa. (E), Locomotion of the robot with CA on the canvas when pm was 138 kPa. (F), Locomotion of the robot with OA on the canvas when pm was 138 kPa. (G), Locomotion of the robot with RA on the paper substrate when pm was 138 kPa. (H), Locomotion of the robot with CA on the paper when pm was 138 kPa. (I), Locomotion of the robot with OA on the paper when pm was 138 kPa. In the locomotion experiments, it was shown that the snakeskins with RA and CA both successfully generated the serpentine locomotion of the robot on the substrates, while the 50 snakeskin with OA could not drive the robot efficiently (Fig. 3.9, D to I). It was also shown that the direction of the robot’s serpentine locomotion was opposite to (the same as, resp.) the traveling-wave propagation direction, when the skin with RA (CA, resp.) was adopted. This reverse of locomotion direction with respect to the traveling-wave direction was at- tributed to the difference of the orthotropic anisotropy in the two snakeskins, of which the directions of the largest µs θ were orthogonal to each other. As an intermediate design between the snakeskins with RA and CA, the snakeskin with OA generated inefficient locomotion, neither moving forward nor backward with respect to the traveling-wave direction. It was also noticed that the robot had a slow lateral sway during the locomotion, which might be attributed to the imperfection in the 3D printing of the soft snake robot. Quantitative results of the serpentine locomotion of the robot with different circular scale arrangements are further presented (Fig. 3.10, A to F). It was shown that both the locomotion speed v and the curvature κm increased with pm, and the maximum speed reached about 31 mm/s when the snakeskin with RA was used on the canvas, with pm of 172 kPa (25 psi). It was also noticed that κm of the robot with RA was smaller than those with CA and OA, which was attributed to the stronger lateral resistance provided by the snakeskin with RA. For the robot with RA and CA (Fig. 3.10, D and E), it was shown that both the speed v and its increasing rate (with respect to pm) increased with pm within the experiment range; and that the speed v was larger on the canvas than that on the paper when the same snakeskin and pm were used, which could be attributed to the higher orthotropic friction anisotropy between the skins and the canvas. In addition, the robot with skins with OA was much slower than the robot with other scale arrangements (Fig. 3.10F), which might be attributed to the much weaker friction anisotropy of this type of snakeskin. To further examine the impact of the scale shape, we compared the robot’s locomotion performance when using the triangular-, circular- and trapezoidal-shaped scales with either RA or CA on the snakeskins (Fig. 3.11, C and F). The orthotropic friction anisotropies of snakeskins F As ⊥ are shown when different scale shapes were adopted (Fig. 3.11, A and B). 51 Figure 3.10 Quantitative locomotion results of the robot with differently arranged snakeskins with circular scales. (A), The relationships between the pressure amplitude pm and the maximum curvature κm of the snake robot with reversed scale arrangement (RA) on two substrates. (B), The relationships between pm and κm of the snake robot with collateral (C), The relationships between pm and κm scale arrangement (CA) on two substrates. of the snake robot with orthogonal scale arrangement (OA) on two substrates. (D), The relationships between pm and locomotion speed v of the robot with RA on two substrates. (E), The relationships between pm and v of the robot with CA on two substrates. (F), The relationships between pm and v of the robot with OA on two substrates. In the experiments for different scale shapes, it was shown that the robot’s speed v, as well as its increasing rate, increased with pm given the same snakeskin and substrate. Among all the snakeskin designs, the one with RA and trapezoidal scales had the best locomotion performance, where the speed v increased from 0 to 37 mm/s when pm increased from 0 to 172 kPa (25 psi). Among the snakeskins with RA, the skin with trapezoidal scales generated the fastest serpentine locomotion for given pm and substrate, followed by the skin with circular scales, and then by the skin with triangular scales (Fig. 6, D and E), showing consistency with their F As ⊥ on substrates; i.e., the skin with larger generated faster locomotion. For the skins with CA, generally, the same trend still held (Fig. 3.11, G and H). However, it was noticed that the snakeskin with triangular scales generated the fastest locomotion when pm was 69 kPa (10 psi) and 103 kPa (15 psi) on the canvas, which might be attributed to the 52 Figure 3.11 Serpentine locomotion results of the robot with reversely and collaterally ar- ranged scales of different shapes. (A), Orthotropic friction anisotropies (F As ⊥) of snakeskins with reversed scale arrangement (RA) and different scales on two substrates. (B), F As ⊥ of snakeskins with collateral scale arrangement (CA) and different scales on two substrates. (C), Illustration of snakeskins with RA and different scales. (D), The relationships between pressure amplitude pm and locomotion speed v of the robot with RA and different scales on the canvas substrate. (E), The relationships between pm and v of the robot with RA and different scales on the paper substrate. (F), Illustration of snakeskins with CA and different scales. (G), The relationships between pm and v of the robot with CA and different scales on the canvas. (H), The relationships between pm and v of the robot with CA and different scales on the paper. more prominent mechanical interlocking effect between the triangular scales and the canvas in those conditions. 3.2.4 Steering Locomotion of the Snake Robot with Snakeskins The steering capability of the soft snake robot with snakeskins was further studied follow- ing the verification of the effectiveness of the proposed snakeskins in generating serpentine 53 locomotion. In the steering experiments, snakeskins with RA and CA and circular scales were used. To generate turning, single-channel biased sinusoidal pressure inputs (Fig. 3.12, B and C) with pm of 138 kPa (20 psi) and T of 1 s were used, replacing the unbiased pres- sure inputs (Fig. 3.12A) used in the previous locomotion tests. As a result, the induced asymmetric bending deformation led to a laterally unbalanced traveling-wave (Fig. 3.12, E and F), resulting in a moment for the turning of the robot. The additional biased pressure pbias was applied to the 2nd air path and the 4th air path to generate left turn and right turn, respectively. Their turning directions were opposite because the air chambers linked to the 2nd and 4th air paths had laterally mirror-symmetric positions in the robot body (Fig. 3.1B). The trajectories of the center of mass of the robot with biased pressure inputs were shown in Fig. 3.13, A, B, C, and D, for snakeskins with RA and CA, on the canvas and the paper, respectively. The angular velocities ω of the robot in all these cases were shown in Fig. 3.13, E and F, where positive (negative, resp.) ω denotes turning right (left, resp.). In the experiments, it was shown that the robot turned left and right on both substrates when the pbias was applied to the 2nd air path and 4th air path, respectively, and the mag- nitude of ω increased with pbias in the 2nd or 4th air path in the experiment range. The trajectory symmetry of the robot with the same pbias in the 2nd and 4th air path was ob- served, and it was also shown that the snake robot went almost straight when pbias was 0. However, it was noticed that the left turning was almost always faster than the right turning given the same pbias and substrate, which might be attributed to the imperfection in the 3D printing and the assembly process. It was also observed that the robot on the canvas generally turned faster than that on the paper given the same skin and pbias. The robot with RA generally turned faster than that with CA given the same substrate and pbias, with the exception when the paper substrate and the bias pressure pbias in the 4th air path were applied. 54 Figure 3.12 Pressure inputs and deformation modes for the steering of the snake robot. (A), Unbiased pressure inputs with amplitude pm for forward locomotion. (B), Pressure inputs with pressure bias pbias on the 2nd air path. (C), Pressure inputs with pbias on the 4th air path. (D), The balanced traveling-wave deformation of the robot. (E), The laterally unbalanced traveling-wave deformation of the robot for turning left, consisted of the bending deformation induced by the pbias in the 2nd air path and the balanced traveling-wave deformation. (F), The laterally unbalanced traveling-wave deformation of the robot for turning right, consisted of the bending deformation induced by the pbias in the 4th air path and the balanced traveling- wave deformation. 3.2.5 Outdoor and In-pipe Tests of the Soft Snake Robot The soft snake robot with the proposed snakeskins was also tested in three outdoor environments, including two concrete surfaces with different roughness levels and a grass lawn. Pressure inputs with pm of 172 kPa (25 psi) and pbias of 0 were used to generate straight serpentine locomotion and the robot’s speed was estimated from the recorded videos. The snakeskins with RA and trapezoidal scales (Fig. 3.14, A to C) and the skins with CA and trapezoidal scales (Fig. 3.14, D to F) were implemented in the outdoor tests. Besides, three different pipes were used to test the robot’s locomotion ability in constrained environments, where the skins with CA and trapezoidal scales (Fig. 3.14, G to I) were implemented and pressure inputs with pm of 138 kPa (20 psi) and pbias of 0 were used. The experiments showed that the proposed snakeskin could help the robot adapt to various surfaces and achieve serpentine locomotion. Consistent with the experiments with 55 the canvas and paper surfaces, the robot’s locomotion directions on both concrete surfaces were opposite to (the same as, resp.) the traveling-wave propagation directions, when using snakeskins with RA (CA, resp.). The locomotion speed reached about 21mm/s and 23 mm/s on the smoother and rougher concrete surfaces, respectively, for skins with RA, and reached about 15 mm/s and 18 mm/s on the two concrete surfaces, respectively, for skins with CA. Moreover, we found that in the grass, the locomotion direction was always the same as the traveling-wave propagation. The speed of the robot with RA was about 2 mm/s, slower than the 5 mm/s speed of the robot with CA. This behavior might be attributed to the interactions between the grass and the sides of the robot body, which could propel the robot in the direction of the traveling-wave.11 In the pipe inspection experiments of the robot, it was also shown that the locomotion direction was the same as the traveling- wave propagation, when the interaction between the environments and the sides of the robot dominated. The speed of the snake robot with the proposed skins reached about 10 mm/s and 14 mm/s in a 2-inch PVC pipe and a 6-inch PVC pipe, respectively. Furthermore, the snake robot demonstrated its locomotion robustness by passing a narrow bottleneck in a soft bending pipe although the snake robot became slower when passing the challenging geometry (bottleneck). 3.3 Discussion In this work we designed novel soft snakeskins to equip soft snake robots, leading to efficient serpentine locomotion on rough substrates. Integrating a soft base and rigid scales provided a balance between stretchability and friction anisotropy of the snakeskin, and fa- cilitated large bending deformation of the robot. Moreover, multi-material 3D printing was used to achieve fast prototyping of the snakeskins, making the whole robot low-cost and easy to build. We systematically varied robot design (e.g., scale shape and arrangement) to measure the locomotion efficiency, and compared it with the baseline experiments. The robot’s serpentine locomotion reached 37 mm/s when the amplitude of pressure input pm was 172 kPa (25 psi) and the snakeskins with reversely arranged trapezoidal scales were used 56 on the canvas substrate. Also, we were able to alter the scale arrangements to change the locomotion direction to be either the same as or opposite to the traveling-wave propagation direction of the induced body undulation. Furthermore, we successfully demonstrated that the steering of the snake robot could be achieved by controlling the bias of the pressure input, and the robot could slither over and move on various outdoor surfaces. For future work, we will further improve the skin design by optimizing the number of rigid scales and by optimizing the materials of the skin base and rigid scales. In particular, while the softness of the skin base is required for the snakeskin not to constrain the body undulation, a too soft skin base would not be able to effectively transmit the friction force from the scales to the body. We will also pursue the integration of a miniaturized pneumatic system with the proposed soft snake robot with snakeskins to realize untethered operation. In addition, we will test the proposed snakeskins on snake robots with other actuation mechanisms such as electroactive polymers. Finally, we will explore applications for the soft snake robot with snakeskins, such as inspection of unpiggable pipelines. 57 Figure 3.13 Trajectory of the center of mass of the snake robot in the steering experiments, and the angular velocities of the robot in different settings. (A), The trajectory of the robot’s center of mass, with snakeskins with reversed scale arrangement (RA) and circular scales, on the canvas substrate. (B), The trajectory of the robot’s center of mass, with snakeskins with RA and circular scales, on the paper substrate. (C), The trajectory of the robot’s center of mass, with snakeskins with collateral scale arrangement (CA) and circular scales, on the canvas. (D), The trajectory of the robot’s center of mass, with snakeskins with CA and circular scales, on the paper. (E), The angular velocity ω of the robot with RA and circular scales on both substrates, where pbias is the pressure bias. (F), The ω of the robot with CA and circular scales on both substrates, where pbias is the pressure bias. 58 Figure 3.14 Locomotion of the snake robot on different outdoor substrates. (A), The lo- comotion of the robot with reversed scale arrangement (RA) and trapezoidal scales on the smoother concrete surface. (B), The locomotion of the robot with RA and trapezoidal scales on the rougher concrete surface. (C), The locomotion of the robot with RA and trapezoidal scales in the grass. (D), The locomotion of the robot with collateral scale arrangement (CA) and trapezoidal scales on the smoother concrete surface. (E), The locomotion of the robot with CA and trapezoidal scales on the rougher concrete surface. (F), The locomotion of the robot with CA and trapezoidal scales in the grass. (G), The locomotion of the robot with CA and trapezoidal scales in a 2-inch PVC pipe. (H), The locomotion of the robot with CA and trapezoidal scales in a 6-inch PVC pipe. (I), The locomotion of the robot with CA and trapezoidal scales in a 2-inch soft bending pipe with a narrow bottleneck in the middle. 59 CHAPTER 4 BACK-STEPPING EXPERIENCE REPLAY WITH APPLICATION TO MODEL-FREE REINFORCEMENT LEARNING FOR A SOFT SNAKE ROBOT In this chapter, a model-free reinforcement learning approach is proposed for an integrated locomotion and navigation control of the pneumatic soft snake robot presented in chapter 3, which efficiently controls the snake robot from the start point to a random distant target de- spite the soft robot’s intricate nature and complex interactions with environments. A novel technique, Back-stepping Experience Replay (BER) is proposed to enhance the learning ef- ficiency of off-policy reinforcement learning (RL) algorithms for systems with approximate reversibility. This method utilizes the back-stepping transitions to construct reversed trajec- tories that reach the goal and ease the sparse reward challenge. BER addresses inaccuracies of the back-stepping transition in the training process by a purification of the replay ex- perience. With the help of BER, the RL agent is able to learn an efficient controller that drives the soft snake robot from the start to the target in a dynamic simulation to assess the efficiency of BER, in which the robot demonstrates successful learning (100 % success rate) and achieve an average speed 48 % faster than that of the best baseline RL algorithm. 4.1 Back-stepping Experience Replay 4.1.1 Background 4.1.1.1 Reinforcement Learning A standard RL formalism is adopted where an agent (e.g. a robot) interacts with an environment and learns a policy according to the perceptions and rewards. In each episode, the system starts with an initial state s0 with a distribution of p(s0), and the agent observes a current state st ∈ S ⊆ Rn in the environment at time step t. Then, an action at ∈ A ⊆ Rm is generated to control the agent based on the current policy π and st. Afterward, the system evolves to a new state st+1 based on the action and transition dynamics p(·|st, at), and a reward rt = r(st, at, st+1) is collected by the agent for the learning before the termination of 60 the episode. During the training process, the RL agent learns an optimal policy π∗ : S → A mapping states to actions that maximize the expected return. The return is defined as the accumulated discounted reward Rt = (cid:80)∞ i=t γi−tri, where γ is a discount factor. The state value function V π(st) = E(Rt|st) represents the expected return starting from state st following the current policy π, and the action value function Qπ(st, at) = E(Rt|st, at) represents the expected return starting from the state st with an immediate action at by following the current policy π. All optimal policies π∗ share the same optimal Q-function Q∗, according to the Bellman equation [73]: Q∗(st, at) = Es′∼p(·|st,at) (cid:20) r(st, at, s′) + γ max a′∈A (cid:21) Q∗(s′, a′) (4.1) 4.1.1.2 Deep Q-Networks and Deep Deterministic Policy Gradient DQN is a model-free, off-policy RL approach suitable for agents operating in discrete action spaces [73]. It typically employs a neural network Q to approximate the optimal Q-function Q∗, selecting optimal actions: a∗ = arg maxa∈A Q(st, a). Exploration is often facilitated by the ϵ-greedy algorithm. To stabilize training, a replay buffer stores transition data (st, at, rt, st+1) and is used to optimize Q with a loss L = E(Q(st, at) − yt), where the target yt is calculated by using a periodically updated target network Qtarg: yt = rt + γ maxa∈A Qtarg(st+1, a), and using transitions in the replay buffer. DDPG [59] is an off-policy RL algorithm that simultaneously learns a Q-function and a policy. DDPG interweaves the learning process of an approximator to Q∗, with an approxi- mator to select a∗, offering a unique adaptation for continuous action scenarios. 4.1.2 Algorithm for BER The above classical off-policy RL algorithms often face challenges with systems char- acterized by sparse rewards or challenging tasks with rewards hard to reshape. In such scenarios, RL agents rarely achieve informative standard forward explorations due to a low success rate in reaching goals in complex problems without precise guidance [3]. To address these challenges, we propose a novel Back-stepping Experience Replay (BER) algorithm for 61 tasks with different goals (Alg. 1), designed to enhance the learning efficiency of off-policy RL algorithms. This is achieved by incorporating exploration methods in both forward and backward directions. The BER algorithm requires at least an approximate reversibility of the system. This means that from a standard transition (st, at, st+1), a back-stepping transition (st+1, (cid:101)at, st) can be constructed, which is similar to a real transition (st+1, (cid:101)at, sb,t) in the environment, i.e., sb,t ≈ st. The action in the back-stepping transition is calculated as (cid:101)at = f (st, at, st+1), where function f is dependent on the environment. The approximate reversibility is evalu- ated by a small upper bound K for all transitions during back-stepping: ∥sb,t − st∥ ≤ K · ∥st+1 − st∥, K < 1 (4.2) There exists a perfect reversibility when K = 0 with a probably complex function f , while an approximate reversibility might be achieved with a slightly larger K and a simpler and solvable function f . It is important to balance the accuracy and computational efficiency of f for effectively constructing back-stepping transitions that preserve enough information for learning. The idea of BER is simple yet effective: instead of solely relying on forward explorations (navy blue solid line in Fig. 4.1) from initial states to goals, which depend heavily on the randomness of forward trajectories to reach these goals, RL agents also navigate backward from the goals to the initial states in the tasks (sky blue solid line in Fig. 4.1). The standard transitions are sampled from the standard forward and backward exploration trajectories (solid lines in Fig. 4.1), where the initial states of themselves are included. Then, the back-stepping transitions are calculated based on the standard transitions to constitute the reversed trajectories (dashed lines in Fig. 4.1), where the virtual goals are set to be the original initial state in their corresponding standard trajectories, such that the reversed tra- jectories are guaranteed to reach their virtual goals and contribute to the learning efficiency. During the explorations, the standard and the back-stepping transitions are collected and stored in separate replay buffers for training. A strategy St is used to sample the transitions 62 Figure 4.1 Illustration of the Back-stepping Experience Replay, with navy and sky blue solid lines representing forward and backward explorations, respectively. from the standard replay Rf with a probability Pt,f and from the back-stepping replay Rb with a probability Pt,b, where Pt,f + Pt,b = 1. For a system with imperfect reversibility, Pt,b gradually drops to zero to purify the transition set for training because of the inaccurate back-stepping transition. The details of BER are shown in Alg. 1. It should be noticed that the operator ⊙ between the states and the goals also indicates the modification of the sequential data (e.g., the history data) when the back-stepping transitions are constructed. The BER accelerates the estimation of Q-functions of the RL agent by using the re- versed successful trajectories to bootstrap the networks. One interpretation of BER is a bi-directional search method for standard off-policy RL approaches, with a higher conver- gence rate and learning efficiency. The purification strategy of the transitions for training needs to be carefully tuned (e.g., tuning the probabilities Pt,f , Pt,b) and might be combined with other exploration techniques, to reach an accurate policy learning in the end and avoid the limitations brought by the bi-directional search method, e.g., non-trivial sub-optimum. In the practical learning tasks, the accuracy and the complexity of the function f : (cid:101)at = f (st, at, st+1), which calculates the actions (cid:101)at in the back-stepping transitions (st+1, (cid:101)at, st), need to be balanced. An accurate f yields better reversibility (with smaller K in Eq. (2)) with more accurate back-stepping transitions and brings less bias and noise, while f itself 63 Algorithm 4.1 Back-stepping Experience Replay (BER). Given: − An off-policy RL algorithm A. − A probability Pb triggering backward trial. − A strategy St for sampling transitions in replays Require: − Approximate reversibility of the system Initialize A Initialize replay buffers Rf and Rb for epoch = 1 → M do Sample a goal g with an initial state s0. Forward trial starts for t = 0 → Tend − 1 do Sample an action at using the policy of A: at ← π(st ⊙ g) Execute action at, observe new state st+1 end for t = 0 → Tend − 1 do rt := r(st, at, st+1, g) Store transition (st ⊙ g, at, rt, st+1 ⊙ g) in Rf ▷ standard experience replay Construct a back-stepping transition: rb,t := r(st+1, (cid:101)at, st, s0) Store transition (st+1 ⊙ s0, (cid:101)at, rb,t, st ⊙ s0) in Rb end Forward trial ends Backward trial starts with Pb Swap the goal g and the initial state s0: s0, g = g, s0 Repeat line 6 - line 16 Backward trial ends for t = 1 → N do ▷ e.g. DDPG ▷ e.g. initialize networks ▷ e.g. ⊙ → diff, concat ▷ BER Sample a mini-batch B from the replay buffers {Rf , Rb} using St Perform one step of optimization using A and mini-batch B end end could be computationally expensive or even unsolvable. On the other hand, a moderate relaxation of the accuracy of f might boost the efficiency of the calculation of back-stepping transitions, when the larger bias and the noises brought by the approximate reversibility (with larger K) are managed by the purification mechanism in BER. 64 4.1.2.1 A case study of BER To illustrate the effectiveness and generality of BER, a general binary bit flipping game [3] with n bits was considered as an environment for the RL agent, where the state was the bit value array s = {si}n i=1 ∈ S, si ∈ {0, 1}, and the action was the index of the chosen bit a ∈ {1, ..., n} = A that was flipped. It was noticed that the game was completely reversible and (cid:101)at = f (at) = at for any time step and transition. The initial state s0 ∈ S and the goal g ∈ S were sampled uniformly and randomly, with a sparse non-negative reward: rt(s, a) = −[s ̸= g]. The game is terminated once s = g. A simple ablation study was designed where a DQN and a DQN with BER were used for training when n = 4, 6, 8. The fully activated backward exploration and the use of back- stepping transitions were stopped after 1k epochs directly. The experimental result (Fig. 4.2) showed that BER facilitated an effective and efficient policy learning for a general DQN approach, and contributed more when the problem became more complex (i.e., n was larger). Figure 4.2 Training experiments of the bit flip game with different algorithms and state dimensions. (A) Returns; (B) Success rates. 4.2 BER in Model-free RL for a Soft Snake Robot In this section, a locomotion and navigation task for a compact pneumatic soft snake robot with snake skins in our previous works [90, 88] is utilized to further evaluate the effectiveness and efficiency of BER with a model-free RL approach, where the robot learns 65 both movement skills and efficient strategies to reach different challenging targets. 4.2.1 Soft Snake Robot and Serpentine Locomotion Compared with soft snake robots where each air chamber was controlled independently [64], in this paper, a more compact soft snake robot with snake skins [90] is considered. There are only four independent air paths to generate the traveling-wave deformation of the robot, which enables the robot to traverse complex environments more easily by reducing the number of pneumatic tubing. The body of the robot consists of six bending actuators and each actuator is divided into four air chambers (Figs. 4.3A, 4.3D) that connect to four air paths (Fig. 4.3B). Four sinusoidal waves with 90-degree phase differences and the same amplitude can be used as references of pressures in air paths to generate traveling-wave deformation (Fig. 4.3C), when the biases of waves induce unbalanced actuation for steering of the robot. Figure 4.3 The overview of the soft snake robot with skins. (A) The soft snake robot with soft snakeskins; (B) The connection between air chambers and air paths; (C) The actuation pressures for air paths; (D) The structure of one bending actuator; (E) The structure of soft snakeskin; (F) The simulation (sim) and experimental (exp) results of the trajectory of the COM of the snake robot on a rough paper surface. Serpentine locomotion is adopted for the movement of the soft snake robot, where the anisotropic friction between the snake skins and the ground propels the robot during the traveling-wave deformation [40]. The artificial snake skins are designed with a soft substrate and embedded rigid scales (Fig. 4.3E); see [88] for more details. 66 To describe the serpentine locomotion of the robot, the dynamic model in [40] is adopted, where the body of the robot is modeled as an inextensible curve in a 2D plane with a total length L and a constant density ρ per unit length. The position of each point on the robot at time t is defined as: X(s, t) = (x(s, t), y(s, t)) (4.3) where s is the curve length measured from the tail of the robot. By utilizing a mean-zero anti-derivative I0 [41] (I0[f ](s, t) = (cid:82) s 0 ds (cid:82) s (cid:82) L 0 ds′f (s′, t)), the position X(s, t) and the orientation θ(s, t) (the angle between 0 f (s′, t)ds′ − 1 L the local tangent direction and the X -axis of the inertial frame) of each point are described as a function of the position X(t) and orientation θ(t) (Fig. 4.4) of the center of mass (COM) of the robot: X(s, t) = X(t) + I0[Xs](s, t) θ(s, t) = θ(t) + I0[κ](s, t) (4.4) (4.5) where Xs = (cos θ, sin θ) and κ(s, t) is the local curvature. X(t) = 1 L (cid:82) L 0 X(s, t)ds, θ(t) = (cid:82) L 0 θ(s, t)ds. The curvature κ(s, t) is related to the local pneumatic pressure via: 1 L κ(s, t) = Kb · ∆p(s, t) (4.6) where Kb is the proportional constant and ∆p(s, t) is the pressure difference between the two air chambers at point s. The anisotropic friction ff ric between the snake skins and the ground is described as a weighted average of the independent components in different local directions (forward ˆf , backward ˆb, transverse ˆt):    ff ric = −ρg(µt( ˆu · ˆt)ˆt + µl( ˆu · ˆf ) ˆf ) µl = µf H( ˆu · ˆf ) + µb(1 − H( ˆu · ˆf )) (4.7) where ˆu represents the direction of the local velocity, µf , µb, and µt are the friction coefficients of the snakeskin in ˆf , backward ˆb, and ˆt directions, respectively. H(x) = (1 + sgn(x))/2, where sgn is the signum function. 67 Figure 4.4 The illustration of the soft snake robot with serpentine locomotion approaching a target. The dynamics of each point of the snake robot is determined by Newton’s second law: ρ ¨X = ff ric + finte (4.8) where finte is the internal force in the robot body, which includes internal air pressure, bending elastic force, etc., with observations: (cid:82) L 0 (X(s, t)−X(t))×finte = 0. 0 finte = 0 and (cid:82) L Finally, the dynamics for the COM of the robot are derived using the equation (4.3)-(4.8) with the observations of finte; see [41] for more details. Based on the dynamic model of the robot, which simplifies a dynamic system for all points of the robot to a single dynamic system for the COM of the robot, a simulator is de- signed with proper discretizations and numerical techniques for RL training. The simulation results matched the experimental results [88] of the soft snake robot when different pressure biases were applied for the robot’s steering (Fig. 4.3F), where the wavy trajectories in the experiments were attributed to the limited number (25) of the tracking markers in the tests. 4.2.2 RL formulation of Locomotion and Navigation of the Robot In this work, the locomotion and navigation of the soft snake robot are formulated as a Markov Decision Process (MDP) M and solved with a model-free RL. The M is defined as a tuple M = (A, S, R, T , γ): 1. Action space: Compared with a random Central Pattern Generator (CPG) [62], more constrained sinusoidal waves are used to generate a smoother traveling-wave 68 deformation of the robot for better locomotion efficiency. Besides, the learned controller of the robot is limited to avoid high-frequency pressure changes, i.e., the RL agent is only able to generate an action to change the parameters of the waveform at the beginning of each actuation period [0, T ] that is same as the period of the sinusoidal waves, and one episode consists of multiple connected actuation periods. The sinusoidal pressure pi for i-th channel of the robot is designed as: pi = pm sin (c · 2π T tr + (i − 1) · π 2 ) + bi,pre + (bi − bi,pre) tr T (4.9) where tr ∈ [0, T ] is the relative time in one actuation period. pm and bi ∈ [0, bm] are the fixed magnitude and bias of the sinusoidal waves for the i-th channel, respectively, i ∈ {1, 2, 3, 4}. bi,pre is a one-step history of the wave bias bi for the i-th channel with bi,pre = 0 at the initial state. c ∈ {−1, 1} is a variable to control the propagation direction of the traveling-wave deformation and thus can change the movement direction of the robot. The action space A of the RL agent for locomotion and navigation of the robot is designed as: a = {ba,1, ba,2, c} ∈ A (4.10) where bi’s are constructed by ba,1 ∈ [−bm, bm] and ba,2 ∈ [−bm, bm]:    b1, b3 = max(0, ba,1), − min(0, ba,1) b2, b4 = max(0, ba,2), − min(0, ba,2) (4.11) At the beginning of each actuation period, based on the current policy, the RL agent observes the state and generates an action, which specifies the waveform of the pressures in that period to propel the snake robot. The wave design guarantees the continuity of the pressures across different actuation periods to avoid impractical sudden changes in the pressures and the robot’s body shape. 69 2. State space: A goal-conditioned state is used for the learning of the RL agent for adapting to different random targets. Specifically, a relative representation of the snake robot’s position and orientation with respect to the target is used as part of the state (Fig. 4.4): s = {∆X, ∆Y, ∆θ, ba,1,pre, ba,2,pre} ∈ S (4.12) where ∆X = xg − X, ∆Y = yg − Y denote the relative position of the target to the COM of the snake robot, ∆θ = θg − θ ∈ (−π, π] represents the relative direction of the target to the main direction of the robot, and θg = arctan(∆Y /∆X) is the angle between the line from the COM of the robot to the target and the X -axis, ba,1,pre and ba,2,pre are two-step histories of the action ba,1 and ba,2, respectively, with an initial setting of {0, 0}. The velocities of COM of the robot are not included as part of the state because the value of the Froude number F r [41] in serpentine locomotion of the snake robot is small, indicating that the frictional and gravitational effects dominate the inertial effect. Two-step histories (longer than one step) are introduced to compensate for the omission of the velocity state. 3. Reward function: The reward function r is pivotal for the RL agent to learn the desired behaviors. The training objective in this work is to drive the COM of the snake robot to reach a random target as soon as possible, with a preference for serpentine locomotion where the robot approaches the target along its main direction. Therefore, the reward assigned to the agent at time t is designed as: rt =    w1 w1 ∆Lt ∆L0 ∆Lt ∆L0 + w2 + w2 2∆θr,t π 2∆θr,t π + Rg, ∆Lt ≤ ϵ , else (4.13) where w1 and w2 are non-positive coefficients, Rg is a large sparse positive success reward once the COM of the robot enters a neighborhood of the target with a radius of ϵ. ∆Lt = (cid:112)∆X 2 is the distance between the COM of the robot and target t + ∆Y 2 t 70 at time t, and ∆Lt = ∆L0 when t = 0. The deflection ∆θr,t ∈ [0, π/2] is used in the reward to allow the robot to approach the target in a backward direction as well: ∆θr,t =    |∆θt|, −π/2 ≤ ∆θt ≤ π/2 π − |∆θt|, else (4.14) 4. Transition probabilities: The transition probability, T (s′|s, a), characterizes the underlying dynamics of the robot system in the environment. In this study, we do not assume any detailed knowledge of this transition probability while developing our RL algorithm. However, it is noticed that some tests of the system are utilized in validat- ing the function f to construct back-stepping transitions with acceptable reversibility, which distinguishes this approach from pure model-free approaches. 4.2.3 Experiments of RL algorithms 4.2.3.1 Experimental Setups The RL experiments for the locomotion and navigation of the snake robot were conducted in a customized dynamic simulator which was developed based on the aforementioned ser- pentine locomotion model. The soft snake robot had a length of 0.5 m with a linear density of 1.08 kg/m. The frictional anisotropy between the snake skins and the ground was set as µf : µb : µt = 1 : 1 : 1.5, and the maximum of the pressure bias bm was set as the same as pm = 276 kPa. The proportional constant Kb between the applied pressure difference and the curvature was set as 0.058 kPa·m. The period of the actuation and the sinusoidal waves was 1 s. The serpentine locomotion of the soft snake robot demonstrated approximate reversibil- ity (Fig. 4.5A) in extensive simulations when the function f was designed as: (cid:101)at = f (at) = {ba,1, ba,2, −c} when at = {ba,1, ba,2, c}. The trajectories in extensive simulation results sug- gested a small K < 1 (in Eq. (6.2)) for locomotion and navigation of the soft snake robot when the above function f was used. 71 Figure 4.5 (A). The approximate reversibility of the movement of the soft snake robot with snake skins. (B). The fixed target and the sampling range of the random targets. The soft snake robot was initialized in the simulator by using a horizontal static curved shape ((X, Y ) = (0, 0), θ = 0) with zero-value action histories and a target (with neigh- borhoods: ϵ = 0.03 m), whose control policies were learned by using BER (with DDPG) and several state-of-art benchmark algorithms, including DDPG, HER [3], and PPO [95]. The number of total training epochs was 10,000 and the strategy to sample the transitions was Pt,b = 0.5e−0.002i when the index of epoch i ≤ 2500, Pt,b = 0 when i > 2500, and Pt,f = 1 − Pt,b, Pb = Pt,b. The coefficients of the reward were selected as ω1 = 0.15, ω2 = 1 (while the choice of weights influences the learning performance, it was observed not to alter the general trend in performance comparison among the algorithms), and the termination condition for one episode was either the COM of the robot entering a neighborhood of the target and receiving a success reward (Rg = 50) or the exploration time exceeding 150 s. The return, success rate, average distance (the averaged ∆Lt/L0 for each time step t), and average deflection (the averaged ∆θr,t for each time step t) were used to evaluate the algorithms during the training, with moving-window averaging for training with different seeds (lwindow = 50 epochs). Three training experiments with different random seeds (for parameter initialization) were conducted to evaluate each algorithm, where the solid line and the shaded area showed the mean and the standard deviation, respectively (Figs. 6.6 and 6.8). An AMD 9820X processor with 64 GB memory and Ubuntu 18.04 was used for the training. 72 Figure 4.6 Experimental results of the training for locomotion and navigation of the soft snake robot with one fixed target (0, 0.5 m). (A) Returns; (B) Success rates; (C) Average distances; (D) Average deflections. 4.2.3.2 Locomotion and Navigation with a Fixed Target The performance of the algorithms was initially evaluated on the locomotion and naviga- tion task of the robot, targeting a challenging fixed point (xg, yg) = (0, 0.5 m) (Fig. 4.5B). The experiment results of the training showed that both DDPG and BER were able to solve the task and learn policies to reach the fixed target successfully, while HER had worse sta- bility and PPO was unable to solve the task within the epoch limitation (Fig. 4.6). It was also shown that BER had a faster convergence rate and better stability compared with other baseline algorithms. The evolution of the maximum Q-value at different locations for the algorithms during the training process (with the same seed) revealed the underlying mechanism and the ad- vantage of BER (Fig. 4.7). It was shown that the effective Q-values in the training with BER were estimated from both the start and the target locations, expediting the successful explorations and the convergence of the estimation. The BER learned a more informative Q-value distribution after 500 epochs than that of the baseline DDPG after 1000 epochs. The final Q-value distribution of BER was also more accurate than that of the baseline DDPG, manifested by their shapes and the positions of the Q-value’s peaks. 4.2.3.3 Locomotion and Navigation with Random Targets A locomotion and navigation task of the soft snake robot with random targets was then explored by using different RL algorithms, where a half ring was used to randomly sample the target because of the system symmetry: g ∈ {(d, α) | d ∈ [0.3, 1], α ∈ [0, π]} (Fig. 4.5B). 73 Figure 4.7 Evolution of the maximum Q-value at different locations during the training (from left to right: initial state, epoch 100, 500, 1k, 10k), with blue arrows illustrating the directions of explorations. (A) Training with BER; (B) Training with DDPG. Besides, a strategy was designed where the targets were sampled uniformly from gradually expanding areas for the i-th training epoch within the total n epochs: g ∈ {(d, α) | d ∈ [0.3, 1], α ∈ [0, ψ] ∪ ( π 2 − ψ, π 2 + ψ] ∪ (π − ψ, π]}, ψ = π 4n2 i2. The training results revealed that BER outperformed all other tested benchmarks (Fig. 4.8). BER achieved the highest return and success rate during training, exhibiting more sta- ble behavior and a smaller average deflection. In contrast, the baseline DDPG’s performance declined when introduced to a variety of targets, despite its strong early-stage performance. HER struggled to learn to reach targets in different areas, indicating that the increasing of additional inefficient goals would not improve its performance but induce undesired behav- iors, whereas PPO gradually learned an effective policy, a process that benefited from the random-goal training setup involving progressively changing targets. The robot’s trajectories further demonstrated BER’s efficiency (Fig. 4.9), where con- trollers with median success rates from each algorithm were used for control. A video for these experiments in a simulator can be viewed at https://youtu.be/Z0da6rVu9j8. Three representative targets were tested: (−0.8 m, 0.1 m) for moving backward, (0, 0.5 m) for 74 Figure 4.8 Experimental results of the training for locomotion and navigation of the soft snake robot with random targets. (A) Returns; (B) Success rates; (C) Average distances; (D) Average deflections. moving towards a lateral target, (0.8 m, 0.1 m) for moving forward. The BER controller successfully and smoothly guided the robot to all targets. In contrast, the DDPG and HER controllers exhibited inefficient oscillations, possibly due to less accurate Q-function estima- tion. While the PPO controller managed to reach all targets, it also displayed oscillation and adopted a sub-optimal policy for the forward target (0.8 m, 0.1 m). The quantitative results of the algorithms (Table 4.1) were the average values tested by using the controllers trained with different seeds, and using 50 random targets sampled from the half-ring area (Fig. 4.5B). The average velocity (vavg = ∆L0/tep, tep: episode length) indicated the efficiency of the learned controllers. Notably, the average velocity of the robot with the BER controller (0.0169 m/s) was approximately 48% faster than that of the DDPG baseline (0.0114 m/s), and significantly higher compared to other benchmarks. Besides, compared to other algorithms, BER not only learned an efficient controller based on the primary reward (highest average deflection: 0.2920 rad) but was also able to sacrifice the secondary reward to some extent (second highest average distance: 0.4002 m/m) for better performance. The success rate of BER reached 100% while the other baselines did not exceed 65%, which exhibited the advantage of BER in the locomotion and navigation learning of the soft snake robot. 4.3 Discussion A novel technique, Back-stepping Experience Replay, was proposed in this work, which exploited the back-stepping transitions constructed by using the standard transitions in 75 Figure 4.9 Trajectories of the COM of the soft snake robot by using the controllers learned by different algorithms. (A) Trajectories with a backward target where the relationships between positions and time are shown in (D), (G); (B) Trajectories with a lateral target where the relationships between positions and time are shown in (E), (H); (C) Trajectories with a forward target where the relationships between positions and time are shown in (F), (I). Table 4.1 Testing performance comparisons of different algorithms. Metrics PPO HER DDPG BER Average velocity (m/s) 0.0061 0.0080 0.0114 0.0169 Average distance (m/m) 0.6241 0.5202 0.3903 0.4002 Average deflection (rad) 0.4049 0.6702 0.3915 0.2920 Success rate (%) 64.44 43.33 61.11 100 both forward and backward exploration trajectories, improving the learning efficiencies in off-policy RL algorithms for the approximate reversible systems. The BER was compatible with arbitrary off-policy RL algorithms, demonstrated by combining with DQN and DDPG in a bit-flip task and locomotion and navigation task for a soft snake robot, respectively. A model-free RL framework was proposed for locomotion and navigation of a soft snake 76 robot as an application of the proposed BER, where a conventional locomotion model for real snakes was adopted to describe the serpentine locomotion of the soft snake robot and to design a simulator for learning. An RL formulation for locomotion and navigation of the soft snake robot was built based on the characteristics of the robot. Extensive experiments showed that the proposed RL approach was able to learn an efficient controller that drove the soft snake robot approaching fixed or even random targets by using serpentine locomotion. For the tasks with random targets, the controller learned by using BER achieved a 100 % success rate and the robot’s average speed was 48 % faster than that of the best baseline RL benchmark. For future work, we will apply the proposed RL approach with BER to a physical soft snake robot system, to explore the simulation-to-reality gap and minimize such a gap using techniques like [134]. It is also noted that we did not consider obstacles in the environment in the current work. We plan to investigate extending the proposed approach to such cases. In addition, we will also study the influence of the function f and the approximate reversibility of general systems (e.g. robotic arms) on BER, and analyze the convergence properties of BER for proper state-of-the-art off-policy RL algorithms. 77 CHAPTER 5 DESIGN AND MODELING OF A SOFT SPIRAL SNAKE ROBOT In this work, we propose a pneumatic soft snake robot that is able to generate a spiral traveling wave deformation with only three independent actuation air paths for locomotion in constrained environments and on flat terrains. The proposed snake robot is modular and each section of the robot has three evenly distributed pneumatic bending actuators. The air chambers in bending actuators share the same structure and materials to keep symmetry for actuation at different angles. With the proposed pressure control method, a spiral traveling- wave deformation is generated, propelling the soft snake robot in constrained environments and on the ground. A kinematic model is built to analyze the locomotion of the soft spiral snake robot in constrained environments. Various experiments show the effectiveness of the locomotion of the spiral snake robot in different environments. 5.1 Design and Fabrication of the soft spiral snake robot The proposed snake robot is modular and consists of multiple identical soft-bending actu- ators that can bend in different orientations with different bending angles. In the prototype of the soft snake robot, a design with six sections is used to generate spiral deformation, which provides enough contact points with constrained environments like a pipeline and with flat surfaces like a ground for stable interactions and support, respectively. Figure 5.1 Prototype of a 6-section spiral snake robot. 78 A pneumatically driven soft actuator is used for the bending actuator of the proposed soft snake robot. There are three evenly distributed air chambers in the bending actuator, which allows the actuator to deform in different orientations (Fig. 5.2). To improve the bending efficiency of the actuator, a bellow shape is used for the outer structure of the actuator to reduce the bending stiffness and introduce high structural anisotropy for each air chamber. Also, the axial deformation of the actuator is limited to improve the bending efficiency by using a central flexible tubing as the displacement-constrained structure. Figure 5.2 Structure for a bending actuator with billowing shape of the proposed soft snake robot. After the design of the single bending actuator, a casting method is used to fabricate the actuator with the complex shape (Fig. 5.3). Silicone glue Ecoflex 00-45 (Smooth-On, Inc.) is proposed to be used for the soft body of the bending actuator. Because of the symmetry of the bending actuator, three identical air chambers of the actuator are fabricated separately and then assembled together for a complete section. The casting molds for the actuator of the robot are 3D-printed. Then, the silicone glue is used as the casting material before the cured parts are removed from the molds (Fig. 5.3A- B). Another set of molds is used to complete the cured parts into a complete air chamber (Fig. 5.3C). After the air chambers of one section of the snake robot are obtained, they are assembled together and combined by using the same casting silicone glue for a single bending section of the snake robot (Fig. 5.3D). Then, different sections of the soft snake robot are 79 assembled together, where soft tubing is used as the connection between the air chambers of different sections, and silicone glue is used to seal the system (Fig. 5.3E). Figure 5.3 The fabrication process for the soft snake robot with the pneumatic bending actuator of the bellow shape. (A). Casting molds for the air chambers of a bellow shape. (B). A cured incomplete air chamber for the soft section. (C). Encapsulating the air chamber by using a mold. (D). Combining and bonding the air chambers and flexible tubing for one soft section of the robot. (E). Connecting the air chambers in different sections to assemble the different sections for a complete soft snake robot. Figure 5.4 Connections of the air chambers between adjacent sections of the snake robot. The connection between the air chambers of adjacent sections is designed to generate a spiral deformation shape with proper pressure inputs. The air chambers in each section will 80 be connected directly to the air chambers of the next section in the clockwise or counter- clockwise direction to generate a left-handed or right-handed spiral curve, respectively (Fig. 5.4). The relative twist angle between the linked air chambers in adjacent sections is set to be 120 degrees in this design for the generation of the spiral deformation. Finally, three independent air tubing are used to connect the three air chambers of the section on the tail of the snake robot and the pneumatic actuation board, achieving a simplified pneumatic system for the soft snake robot. Thus, when the proper input pressures are applied, a spiral curve for the whole robot is generated (Fig. 5.6). It is also noticed that the generated spiral curve has a central axis because of the structure and actuation symmetry. Figure 5.5 The generated helix-like deformations for the robot with different pressure inputs in one same channel. (A). When a single section has a 120-degree bending angle. (B). When a single section has a 90-degree bending angle. 5.2 Modeling of the Snake Robot 5.2.1 Static modeling of the Pneumatic Bending Actuator The proposed snake robot is made of multiple identical bending actuators with same bending angle and different bending orientations. To describe the deformation of the pro- posed snake robot, a static model is built for the deformation of one single section. The model is used to describe the relationship between the actuation pressures in different cham- bers and the bending angle θ and orientation ϕ of the single section (Fig. 5.6). Considering 81 the flexible tubing embedded in the bending actuator limiting its axial deformation, only the bending actuation is considered in this modeling part. Figure 5.6 The bending deformation of a single section on the robot. Considering the adaptive locomotion task of the soft robot in complex environments, an efficient modeling approach is adopted along with some simplifing. The piecewise constant curvature (PCC) assumption is used for the shape modeling and an Euler-Bernoulli beam principle with pure moment input is used: κ = 1 R = M EI (5.1) where κ and R are the curvature and the radius of backbone of the single section. M is the total moment applied by the actuation pressures. E is the modulus of elasticity and I is the moment of inertia of the single section. It is also assumed that EI is a constant for different bending orientation and would be calibrated in the experiments. The components of bending moment generated by Chamber 1 can be calculated: Mx1 = My1 = (cid:90) r2 (cid:90) 2π/3 r1 0 (cid:90) r2 (cid:90) 2π/3 r1 0 P1r2 sin ψdψdr P1r2 cos ψdψdr 82 (5.2) (5.3) where Mx1 and My1 are the moment components of Chamber 1 around X axis and Y axis, respectively. r1 and r2 are the radii shown in Fig. 5.6. P1 is the pressure applied in Chamber 1. Similarly, the moment components for Chamber 2 (Mx2, My2), and for Chamber 3 (Mx3, My3) can be computed. The total moment Mtotal, bending curvature κ, bending angle θ and bending orientation ϕ can be derived: Mtotal = (cid:118) (cid:117) (cid:117) (cid:116)( 3 (cid:88) i=1 Mxi)2 + ( 3 (cid:88) i=1 Myi)2 κ = θ = 1 EI l EI Mtotal Mtotal ϕ = arctan (cid:80)3 (cid:80)3 i=1 Myi i=1 Mxi (5.4) (5.5) (5.6) (5.7) where l is the length of the backbone of the single section. 5.2.2 Shape Modeling of the Whole Robot After the model between pressures in different chambers and the deformation of a single bending section is acquired, we then build the geometric model of the central spine of the robot based on the PCC assumption and the bending deformation of different sections. It is noticed that the different sections share the same bending angle θ and bend in different planes due to the pnuematic system design of the robot. Here, we use the transformation matrix to calculate the global coordinates of each point on the snake robot based on the local frames (Fig. 5.7) and the local transformations. Specifically, the initial frame Σ0,s is defined such that the origin of Σ0,s is located at the one end of the robot. The X-axis of Σ0,s is tangent to the robot and the first section of the robot lies in the first quadrant in the XOY plane of Σ0,s. The global frame g is defined such that its X-axis shares the same direction as the central axis of the robot. 83 Figure 5.7 Global and local frames of the soft spiral snake robot. Σ0,s and Σ0,e denote the start and the end frames of the first section of the robot, respectively. Σ1,s and Σ1,e denote the start and the end frames of the second section of the robot, respectively. The frame g denotes the global frame where gx aligns with the central axis of the robot. 5.2.2.1 Modeling within Each Single Section In this section, a local model is built to describe the geometric shape of the single section with respect to its local frame Σi,s (Fig. 5.7) by using the following assumption: A5.1. The snake robot has piece-wise constant curvatures. (PCC). A5.2. There is no pressure attenuation across the soft snake robot, and the magnitudes of the bending curvatures for different sections are the same. The coordinates of the points on the i-th section of the robot can be described as: X i−1,s i (s) =                   xi(s) yi(s) zi(s) 1 84 = Hi(s, κi)X i−1,s i,0 (5.8) where X i−1,s i,0 is the original point of the frame Σi−1,s and X i−1,s i,0 = [0 0 0 1]T . The variable s is the curve length from the origin of Σi−1,s to the studied points. κi is the curvature of the i-th section and X i−1,s i (s) is the position of the studied point in the i-th section in the frame Σi−1,s. Hi(s, κi) is the homogeneous transformation matrix for ith section: Hi(s, κi) =     Rz(κi, s) pi(κi, s) 0 1   =          c(κis) −s(κis) 0 s(κis) κi s(κis) c(κis) 0 1−c(κis) κi 0 0 0 0 1 0 0 1          (5.9) where Rz(·) is the rotation matrix round local Z-axis, pi is an in-plane translation. s(·) and c(·) denote the sin(·) and cos(·), respectively. The tangent vector for each point on i-th section is also described based on its base frame Σi−1,s for part of the in-section modeling:  T i−1,s i (s) = Rz(κis)T i−1,s 0 = s(κis) c(κis) c(κis) −s(κis) 0      0 0  0      1   1     0       0 (5.10) where T i−1,s 0 is the tangent vector of the origin point of Σi−1,s. 5.2.2.2 Modeling in Base Frame of the Snake Robot After the shape model of each section with respect to their local base frame is derived, we then transform the shape to the base frame of the whole snake robot Σ0,s, represented by the points in the base frame Σ0,s. The point coordinates and the local tangent vector of the points on i-th section in the base frame Σ0,s of the robot can be derived: X 0,s i (s) = H 0,s i−1,sX i−1,s i (s) T 0,s i (s) = R0,s i−1,sT i−1,s 2 (s) 85 (5.11) (5.12) where H 0,s i−1,s = H 0,s 1,s H 1,s 2,s ...H i−2,s i−1,s = i−1 (cid:89) j=1 Hj(l, κj)Htw,j(γ) R0,s i−1,s = i−1 (cid:89) j=1 Rz(κjl)Rx(γ) (5.13) (5.14) Hj(l, κj) is calculated by Eq. (5.9). Htw,j(γ) is a transformation matrix to describe the     Rx(γ) 0 0 1 rotation between two adjacent sections: Htw,j(γ) =  . κj is the curvature of the j-th section and γ is the twist angle between the linked air chambers in the adjacent section. Rz(·) and Rx(·) are the rotation matrix around Z-axis and X-axis, respectively. By using this shape model, the spatial shape and local tangent vector of the soft spiral snake robot are described in its base frame Σ0,s and can be simulated in MATLAB (Fig. 5.8). Figure 5.8 Geometric shape of a soft spiral snake robot with six sections. 5.2.2.3 Calculation of the Central Axis (Global Frame g) of the Snake Robot After the shape of the snake robot is fully described, the direction and position of the cen- tral axis of the soft spiral snake robot are calculated, which are important for the locomotion analysis of the robot. 86 For the proposed pneumatic system design, the bending curvature κi and twist angle γ of all different sections are the same at different time steps and with different actuation conditions. Therefore, it is inferred that there exists a central axis of the snake robot, for which all sections share a rotation symmetry. Figure 5.9 Illustration of the tangent vectors of at the start of each section with respect to the central axis of the robot. It can be shown that the spiral shape (Fig. 5.9) of the snake robot has the following properties: P5.1. The angles between the section planes and the central axis are the same at each time: Σi,s,z · gx = C1(t) (5.15) where Σi,s,z denotes a unit vector along the Z-axis of the frame Σi,s for i-th section. P5.2. The angles between the tangent vectors of the start of each section and the central axis, are the same at each time: Ti · gx = C2(t) (5.16) P5.3. The angle between the tangent vector at the starts of two adjacent sections are the same at each time: Ti · Ti+1 = C3(t) (5.17) Based on these properties and the structure symmetry of the robot, it can be deduced that the tangent vectors at the start points of different sections are distributed on a cone 87 surface (Fig. 5.9). The direction of gx is then derived as: gx = (T3 − T2) × (T2 − T1) ||(T3 − T2) × (T2 − T1)|| (5.18) It is easy to calculate gx for a special case when T3 = T1: gx = (T1 + T2)/||(T1 + T2)||. In this way, gx is presented in the base frame Σ0,s. After gx is calculated, we can then calculate the position of the origin of the global frame Og in the base frame Σ0,s. By using the PCC assumption (piece-wise constant curvature), each section can be considered as an arc, and thus in the perspective of gx, it becomes a symmetric part of an ellipse (orange curve in Fig. 5.10). Figure 5.10 Spiral snake robot (A). in a pipe and (B). on the ground with a view along its central axis. For simplicity, gy is defined as a unit vector from the origin of frame g to the origin of frame Σ0,s to describe the frame g (Fig. 5.10). In order to calculate the transformation between the base frame Σ0,s and the global frame g, we design an intermediate frame Σinte (Fig. 5.11), which is constructed by rotating the base frame Σ0,s around its origin and make the x-axis of Σinte parallel with gx. The rotation of from Σ0,s to Σinte can be present in a quaternion form and then is converted to a rotation matrix: R0,s inte = quat_to_rotm(q) q = cos( α 2 ) + sin( α 2 )β · i 88 (5.19) Figure 5.11 Shape of the spiral snake robot with a view along its central axis. (A). The global frame g and the start frame Σ0,s of the first section. (B). The global frame g and the intermediate frame Σinte. where α = arccos(gx · Σ0,s,x) is the angle between gx and Σ0,s,x, β is the rotation axis vector: β = gx × Σ0,s,x. Then, the start points Pi of different sections are projected onto YOZ plane of Σinte: X inte Pi = H inte 0,s X 0,s Pi (5.20) where H inte 0,s =    Rinte 0,s 0  0  . 1 With X inte Pi , i = 1, 2, 3, we can compute the origin of frame g and gy in Σinte. Specifically, we can calculate out the center of circle prescribed by P1, P2, P3, which is the origin of frame g in the YOZ plane of Σinte (Fig. 5.11). Finally, we obtain the transformation matrix between the intermediate frame Σinte and the frame g. A simple algorithm is shown below: 1. Convert X inte Pi into 2D space on YOZ plane of Σinte. 2. Calculate the center of circle X inte Og = [0 Og,y Og,z]T prescribed by P1, P2, P3. ginte y = −X inte Og || . ||X inte Og 3. Derive the transformation matrix between the frame Σinte and the frame g: H inte g =    ginte x ginte y y X inte ginte Og 0 1    (5.21) 89 where ginte x = [1 0 0]T , and gz = gx × gy. Finally, the transformation matrix between the base frame Σ0,s and frame g is obtained: X g i (s) = H g 0,sX 0,s i (s) (5.22) where H g 0,s = H g inteH inte 0,s . Combined with Eq. (5.8) and Eq. (5.11), the shape model of the robot in the frame g can be calculated based on the bending curvature κ and the twist angle γ between adjacent sections. 5.2.3 Kinematic Modeling of the Robot in Constrained Environments A kinematic model of the proposed soft spiral snake robot in a cylinder environment is derived based on the shape modeling in the previous section. For simplicity, we assume: R = Rb + r (5.23) where R is the radius of the cylinder environment, Rb is the radius of the encapsulating cylinder surface of the backbone of the robot, and r is the radius of the backbone of the bending actuator. The radius of the encapsulating cylinder environment (Fig. 5.10) of the snake robot R is: R = max(||gx × X g i (s)||) + r, s ∈ [0, l], i ∈ {1, ...n} (5.24) where i is the index of the sections, l is the length of one section of the robot. Furthermore, based on the structure symmetry of the bending actuator and robot shape, it is inferred that the maximum distance between each section and the central axis of the robot is the distance between gx and the midpoint of the section: R = ||gx × X g 0 ( l 2 )|| + r (5.25) From the next locomotion analysis (Sec. 5.3) and the shape of the robot, the snake robot in the cylinder environment has a spiral advancing locomotion method, and the direction of the local movement is derived as: θ = arccos(Σ0,s,x · gx) (5.26) 90 where θ is the angle between the local velocity of the contact point and the central axis gx, Σ0,s,x is a unit vector along X axis of frame Σ0,s. Its component along gx propels the robot to move along the cylinder environment such as a pipeline, and its component perpendicular to gx makes the snake robot have a rotation in the cylinder environment around gx. Thus, the velocity v of the proposed snake robot along the cylinder environment (e.g. pipeline) is derived as: v = 2πr T cos θ (5.27) where r is the radius of the bending actuator, and T is the period of the actuating air pressures in the chambers. 5.3 Actuation Scheme of the Air Pressures and Locomotion Analysis in a Local Perspective After a spiral shape of the robot is achieved by using multiple linked bending actuators, time-varying pressure inputs are designed to generate a traveling-wave for the spiral defor- mation. To stabilize the radius of the encapsulating cylinder of the spiral snake robot, the pressure signals are designed so that the bending orientation ϕ has a linear relationship with time, while the bending angle remains fixed at a constant value. It could be formulated as an optimization problem: min P1,P2,P3 ||P1 + P2 + P3||2 s.t. P1 ≥ 0; P2 ≥ 0; P3 ≥ 0 Mtotal (5.28) θ = θtar = l EI ϕ = 2π T t = arctan (cid:80)3 (cid:80)3 i=1 Myi i=1 Mxi where P1, P2, P3 are the pressure in Chamber 1, 2, 3, respectively. θtar is the target bending angle for a single section, and T is the period of the actuation signal. It is straightforward to calculate the solution of the optimization problem, where the sig- nals are show in Fig. 5.12. A PID controller is implemented in a micro-controller (Arduino) 91 Figure 5.12 Actuation signals for pressures in different channels. A. Pressure signals for moving forward. B. Pressure signals for moving backward. to achieve closed-loop control of the air pressures in different air paths by using pressure sensors. During the traveling-wave generation, the general spiral shape of the snake robot does not change because the bending angles of different sections stay the same with respect to time. However, the bending orientation of the sections with respect to the backbone curve changes, which influences the relative positions between the air chambers in one section with respect to the backbone of that section (Fig. 5.13). Therefore, the contact points between the snake robot and the external environment like pipe or ground will change periodically around the sections of the snake robot and thus propel the snake robot. The propelling force direction of the robot is decided by the moving direction of the contact points between the snake robot and the environment, which could be approximated by the perpendicular direction of the tangent direction of the backbone of the robot at the contact point. 5.4 Simulation and Experimental Results 5.4.1 Simulation of the Shape of the Snake Robot The shape of the snake robot and the encapsulating cylinder of its spine are calculated and simulated based on the proposed model, which also serves to validate the assumptions we made in previous sections. Fig. 5.14 shows a simulation result of the spine of the soft 92 Figure 5.13 Locomotion analysis for the soft snake robot. The relative position of air cham- bers changes with respect to the shape of the bending actuator during the traveling-wave generation. spiral snake robot with its center axis and encapsulating cylinder surface, when the bending angle for each actuator is 60 degrees and the twist angle γ is 120 degrees. Figure 5.14 A. The shape of the soft spiral snake robot with the central axis and encapsu- lating cylinder. B. View of the robot along the central axis. The central axis of the snake robot and its encapsulating cylinder surface for the spine of the proposed snake robot are shown in Fig. 5.14A. It is also shown that the contact points between the robot and the cylinder environment are located in the middle of each section, as shown by the red markers in Fig. 5.14B. 93 Figure 5.15 The shape of the soft spiral snake robot with fixed twist angle (120 degrees) and different bending angle for one section: A. 60 degrees, B. 90 degrees, C. 120 degrees, D. 150 degrees. Figure 5.16 The shape of the soft spiral snake robot with fixed bending angle (90 degrees) and different twist angle: A. 30 degrees, B. 70 degrees, C. 110 degrees, D. 150 degrees. 94 Both the bending angle α of the actuator and the twist angle γ between the linked chamber in adjacent sections affect the shape of the robot and its kinematics. The bending angle α is controlled by the actuation pressures and the twist angle γ is fixed during the assembling process of the snake robot. Fig. 5.15 shows the robot shape with a fixed γ (120 degrees) and different α. Fig. 5.16 shows the robot shape with a fixed α (90 degrees) and different γ. The quantitative results of the shape of the soft spiral snake robots are further analyzed by using the radius of their encapsulating cylinder surface for evaluation. Fig 5.17 shows the relationship between α of actuators and the radius of the encapsulating cylinder R when the γ is fixed (120 degrees). It is shown that R increases with an increase of α when γ is fixed. Figure 5.17 The relationship between the bending angle of actuators and the radius of the encapsulating cylinder when the twist angle is fixed (120 degrees). Fig 5.18 shows the relationship between the twist angle γ and the radius of the encap- sulating cylinder R when the bending angle α is fixed (90 degrees). It is shown that R decreases with the increasing of γ when α is fixed. 5.4.2 Locomotion of the Soft Snake Robot in a Pipe Experiments of the proposed snake robot in a pipe were used to validate the locomotion of the snake robot in a constrained environment. A clear 2-inch pipe was used as the cylinder- constrained environment of the proposed robot. The period of the actuation signal T was set to be 6 s. 95 Figure 5.18 The relationship between the twist angle between actuators and the radius of the encapsulating cylinder when the bending angle is fixed (120 degrees). Fig. 5.19 shows the forward locomotion of the snake robot in the pipe (with pressures of Fig. 5.12A), and it is observed that the snake robot had a forward motion along the pipe, as well as a rotation motion in the pipe. The interactions between the robot body and the inner surface of the pipe propelled the robot and achieved an spiral locomotion in the pipe. It was also noticed that the interactions between the robot and the pipe always happened at the midpoint of each bending actuator, which was consistent with the prediction of the robot model. Figure 5.19 Locomotion of the proposed snake robot in a 2-inch pipe. The backward locomotion of the snake robot in the pipeline could also be achieved by using actuation signals with phase lead (Fig. 5.12B), which is shown in Fig. 5.20. Besides, it was also shown that the soft snake robot was able to pass challenging geometries smoothly in a complex pipe system. Fig. 5.21 shows the locomotion of the proposed soft snake robot 96 passing a sharp elbow. Figure 5.20 Backward locomotion of the proposed snake robot in a 2-inch pipe. Figure 5.21 The proposed snake robot passing a sharp turn in a 2-inch pipe. 5.4.3 Locomotion of the Soft Snake Robot on Flat Surfaces Another set of experiments showed that the proposed snake robot was not only able to move in a constrained environment, but also on flat surfaces without other additional anisotropic structures, demonstrating its locomotion adaptability in versatile environments. 97 Fig. 5.22 shows the forward locomotion of the snake robot on a flat surface with phase- lag actuation signals (Fig. 5.12A), while Fig. 5.23 shows the backward locomotion of the robot with phase-lead actuation signals (Fig. 5.12B). It was shown that the snake robot had both velocities in the longitudinal direction and in the traverse direction, with respect to the center axis of the robot. The local rolling effect between the robot body of the surface propelled the robot on the flat surfaces without anisotropic structure. Figure 5.22 Forward locomotion of the proposed snake robot on a flat surface. 5.5 Discussion In this work, we designed a soft spiral snake robot that was able to generate a spiral traveling-wave deformation for locomotion in different environments by only using three independent air paths. Pneumatic bending actuators of a bellow shape and three evenly distributed air chambers were designed for each section of the modular snake robot and the related casting fabrication method was also developed. The actuation strategy of the pres- sures in three air paths was proposed to generate the traveling-wave deformation in 3D space. A model was developed to describe the shape of the soft spiral snake robot and its kinematics in cylinder environments like pipelines. Simulation results showed the relationship between the bending and twist angles of the actuators and the radius of its encapsulating cylinder 98 Figure 5.23 Backward locomotion of the proposed snake robot on a flat surface. surface, and the experiment results showed the locomotion performance of the proposed snake robot in a pipe and on flat ground. For future work, more experiments for the snake robot will be conducted to validate the kinematic model and the adaptability of the snake robot in complex pipeline systems and on flat surfaces. In addition, we will explore the miniaturization solutions for the proposed soft spiral snake robot for more challenging constrained environments. 99 CHAPTER 6 DESIGN AND NON-LINEAR MODELING OF A MODULAR CABLE-DRIVEN SOFT ROBOTIC ARM In this chapter, a multi-section cable-driven soft robotic arm inspired by octopus tentacles is presented. The soft robotic arm is modular and each section of the soft arm is made of rigid 3D-printed structures and silicone rubber. The soft arm is actuated by using a cable system to imitate the longitudinal muscles of octopus tentacles, whose guide channels are designed so that the deformation of each section can be independently controlled. A static model for the soft arm is proposed by considering the immersive effect between the actuation cables and the soft silicone rubber body of the soft arm. The experiments show good agreement with the model prediction compared with the baseline model and also demonstrate the high flexibility of the soft robotic arm. 6.1 Design and Fabrication of the Soft Robotic Arm In this study, we propose a decoupled modular cable-driven soft robotic arm made by 3D printing and casting and develop a novel analytical static model that considers the prominent immerse effect of the cable on the soft body of the proposed soft robotic arm, which was not included by most of the other analytic modeling works. Many biological structures and mechanisms inspired the designs of robotic systems, and the development of robotics also provides insights and bio-physical prototypes for bio-mechanics [126, 18, 87, 51]. In particular, the design of our soft robotic arm was inspired by the muscle anatomy of octopus tentacles, where the longitudinal muscle and transverse muscles were utilized for bio-inspiration. Specifically, the structure of a single section of the robotic arm consisted of three parts: a flexible backbone, a soft silicone body, and two rigid caps (Fig. 6.1B). A piece of soft tubing was selected as the backbone for its high bending flexibility and low stretchability to constrain the section length. Two rigid endcaps were attached to the ends of the backbone, which acted as connectors between sections and anchor points for cables. A complex anchor 100 structure was designed for a solid connection between the endcaps and the soft body. The soft silicone body was made from casting with three evenly embedded fiber-reinforced cable guides and fluid cavities. The actuation cable in the cable guides provided contraction force as longitudinal muscles while the potential fiber-reinforced actuators in fluid cavities was able to provide elongation force to emulate the effects of transverse muscles. The modular multi-section soft robotic arm consisted of arbitrary identical soft sections with the embedded cable system. The connection of the endcaps was able to generate pathways between the cable guides and the backbone tubing (Fig. 6.1C-D). The actuation cable, whose one end was fixed on the endcap, passed the cable guide in one section and went into the backbone tubing through the pathways before it was attached to the corresponding driving motor (Fig. 6.1E). During the bending motion of the multi-section robotic arm, the backbone tubing kept almost a constant length, separating deformation of one section and the cable lengths for other sections and thus achieving decoupling between different sections of the robotic arm. The proposed soft robotic arm was modular and consisted of multiple sections. For each section, the fabrication process could be separated into two steps: 3D printing and casting (Fig. 6.2). The rigid endcaps and the casting mold for the soft sections were first printed by a high-precision Object Connex 350 3D printer, which could construct complex structures layer by layer by jetting photopolymerizable materials that were cured by subsequent UV light. The material Objet Vero White was used in the 3D printing process for the rigid parts. The casting molds for the soft section including the rods for creating the fluid cavities were also fabricated by using the same 3D printing method. After all the molds and rigid parts for the soft section were prepared, the flexible backbone tubing was connected with the two rigid endcaps before assembling with the enclosure structures of the casting mold, which included a sole plate and three separate enclosure walls for easy removal of the molds after casting. Then, a high-strength Kevlar thread (High-Strength High-Temperature Thread, McMaster-Carr) was used to create a coil layer around the rods for the fluid cavities and 101 Figure 6.1 Structures of the soft robotic arm. (A). A two-section modular soft robotic arm. (B). The structure for one section of the robotic arm. (C). Connection of two endcaps. (D) One endcap at the tip side of the section. (E). Cable paths for different sections of the soft robotic arm. the cable guides. The rods with the fiber coils were then inserted into the assembled casting mold with the help of locating holes on the two endcaps to complete the mold assembly. The silicon glue Ecoflex 00-00 was then used for the construction of the soft body of the arm section in the casting process and was injected into the mold from the auxiliary hole of one endcap. After the curing of the silicone glue, the rods for the cavities were extracted from the mold and were then inserted back with a silicone glue covering to construct an extra layer of silicone over the fiber coil for protection. Finally, all the mold pieces and the seal taps were removed and one section for the soft robotic arm was fabricated. After each section of the robotic arm was prepared, the actuation cables were first assembled with 102 Figure 6.2 Fabrication process for one section of the soft robotic arm. (A). 3D printing of the rigid parts. (B). Assembling of mold for one section. (C). Twisting fiber coils around the rods for cavities. (D). Inserting the rods for cavities into the mold. (E). Injecting silicone glue into the mold and curing. (F). Adding another layer of silicone glue to the inner surface of the cavities. (G). Removing the mold and getting one section of the soft robotic arm. different sections, whose one end was fixed on the endcaps by using an anchor piece, and the other end was attached to the driving pulley. Screw connections were then applied to link different sections and the base for the whole robotic arm to complete the fabrication and assembling process of the prototype of the proposed soft robotic arm. 6.2 Modeling for the Soft Robotic Arm The kinematic model for the multi-section soft robotic arm was separated into two parts: a static model for a single section, which mapped actuation cable length to the bending configuration of a single section, and a kinematic model for multi-section arm, which char- acterizes the relationship between the bending configurations for all sections and the task space variables (end position of the arm). 6.2.1 Statics of a Section Driven by a Single Cable The static model for the single section of the robotic arm was built based on static analysis of the section bending deformation. Before studying a section with multiple actuation cables, 103 the model for a single cable-driven section was built and an arbitrary bending configuration was analyzed (Fig. 6.3A). The cable and the support of the section provided external forces. To simplify the static analysis, two assumptions have been made: 1. The backbone (dash line) of the soft section had a constant length, 2. The backbone and the cable (red line) had constant curvatures and perfect symmetries. Figure 6.3 Modeling for one section of the arm driven by multiple cables. (A). Bending configuration for one section of the arm driven by multiple cables. (B). External moments applied by five cables to the soft section in the lateral direction P . The bending angle of the backbone ϕ(s) and its curvature κc were described as: ϕ(s) = κc = s Rc dϕ ds = 1 Rc (6.1) where sc is the backbone length and Rc is the radius of the backbone. The cable curvature κg and the bending angle θ(s) were described similarly: θ(s) = s/Rg, κg(s) = 1/Rg, where Rg is the radius of the actuation cable, and s is the cable length. The following assumption was also made: 3. The friction between the cable and the soft body of the arm was negligible. Thus, the transverse force density ρ between the cable and the soft body was derived as: dFρ = T dθ dFρ ds ρ = = T κg 104 (6.2) where T is the tension of the cable and Fρ is the transverse force between the cable and the soft body. Next, the transverse forces ρe(s) in the base frame e was calculated by using a rotation matrix Re a:     −ρ   0 ρe(θ) = Re a  Re a(θ) =   cos θ − sin θ sin θ cos θ    (6.3) The total transverse force Feq between the cable and the soft body could then be obtained by the following integration: Feq = = (cid:90) ϕb−θ0 θ0 (cid:90) ϕb−θ0 θ0  ρe(s)ds ρe(s) dθ κg = T   − sin (ϕb − θ0) + sin θ0 cos (ϕb − θ0) + cos θ0 (6.4)    where ϕb is the bending angle of the section and θ0 is the incident angle of the cable (Fig. 6.3A). The contraction force FT applied by the cable to the soft section was calculated as:     FT = Re a   0 −T   = T   sin (ϕb − θ0) − cos (ϕb − θ0)   (6.5) Thus, the external force balance of the section was built, and the support force was 105 derived as: (cid:88) F = Fr + Feq + FT = 0 Fr = −Feq − FT     − sin (ϕb − θ0) + sin θ0 − sin (ϕb − θ0)   + T     (6.6) cos (ϕb − θ0) − cos θ0 − cos (ϕb − θ0)   = −T  = T      − sin θ0 cos θ0 Next, the external moment balance of the section was analyzed with respect to base point O (Fig. 6.3B). The arm rT (Fig. 6.2B) for FT was derived as:       rT = = −d − (Rd − Rd cos ϕb) Rd sin ϕb −d − 2Rd sin2(ϕb/2) 2Rd sin (ϕb/2) cos (ϕb/2)       (6.7) where Rd is the radius of the cable when the transverse deformation of the cable is not considered. The arm req (Fig. 6.3C) for Feq (which was located on the mirror symmetric axis of the bending section) was calculated by utilizing the geometry relationship:    req = −d − Rd sin2(ϕb/2) Rd sin(ϕb/2) cos(ϕb/2)    (6.8) Then, the external moment balance of the section was then built, and the support moment is derived: (cid:88) M = Mr + Meq + MT = 0 Meq + MT = req × Feq + rT × FT (6.9) = T d cos θ0 Mr = −T d cos θ0 106 Next, the incident angle of the cable θ0 (Fig. 6.3D) was calculated as: ∆y = (Rc − d) sin α = Rg sin(α − θ0) θ0 = α − arcsin((1 − κcd) κg κc sin α) (6.10) where α = ϕb/2 and d is the distance between the incident point of the cable and the base point O of the section. Then, an assumption was made to describe the relationship between κg and κc: 4. The maximum transverse deformation of the cable was proportional to the linear density of the transverse force applied by the cable. Thus, the following equations were derived based on geometry relationship (Fig. 6.3D): ∆h = |BC| = Rd(1 − cos α) − Rg(1 − cos(α − θ0)) (6.11) ρ = Kc∆h where ∆h is the maximum transverse deformation of the cable and Kc is the coefficient in the linear relationship between ∆h and ρ. After the external loading conditions was analyzed, an further assumption was made for the bending deformation of the soft section: 5. The soft section of the arm had a linear bending elasticity with no hysteresis. As a result, the relationship between the bending deformation and the external torque was derived by introducing a bending stiffness Kb: −Mr = Kbκc (6.12) Finally, by using the equations (6.2), (6.9), (6.10), (6.11), (6.12) and the geometry rela- tionships, the kinematic model for a single soft section driven by a single cable was derived 107 as: Kbκc = T d cos θ0 θ0 = α − arcsin((1 − κcd) κg κc sin α) T = Kc κg {( 1 κc − d)(1 − cos α) − 1 κg (1 − cos(α − θ0))} l = Rg(ϕb − 2θ0) − 1 κg (Lκc − 2θ0) (6.13) where L is the length of the soft section’s backbone. 6.2.2 Statics of a Section Driven by Multiple Cables After the model for one section driven by one cable was obtained, the model for multiple cable actuation was introduced to capture more complex bending deformation and larger workspace. A similar approach was used for building the multiple cable-driven model: a static analysis for an arbitrary bending configuration with multiple cable actuation assuming there was no slack for all the cables. Figure 6.4 Modeling for one section of the arm driven by multiple cables. (A). Bending configuration for one section of the arm driven by multiple cables. (B). External moments applied by five cables to the soft section in the lateral direction P . The bending configuration for the soft section and its bending plane were defined by the bending angle ϕb and the bending orientation γ (Fig. 6.4A). A neutral curve plane was defined so that it was perpendicular to the bending plane and included the backbone. The cables were indexed counterclockwise and the direction of the x axis of the base frame a pointed to the 1st cable. The angle between the 1st cable direction and the bending direction 108 was derived: βi = 2π(i − 1) n − γ i = 1, 2, 3 · · · (6.14) The distance between the incident point of the ith cable and the natural plane was calculated as: di = d cos βi i = 1, 2, 3 · · · (6.15) In the bending plane of ith cable (Fig. 6.4A), which was parallel to the bending plane of the backbone and included the ith cable, the external force condition was analyzed as the single cable driven case:  Fr,i = Ti   − sin θ0,i cos θ0,i    Mr,i = −Tidi cos θ0,i θ0,i = α − arcsin((1 − κcdi) κg,i κc sin α) Ti = Kc κg,i {( 1 κc − di)(1 − cos α) − 1 κg,i (1 − cos(α − θ0,i))} (6.16) where Ti, θ0,i, and κg,i are the cable tension, incident angle and curvature of ith cable, respectively, and Fr,i and Mr,i are the external force and moment in frame e. Then, the elastic deformation of the section was calculated: M = n (cid:88) i=1 −Mr,i M = Kbκc (6.17) Furthermore, there was no bending deformation in the direction that perpendicular to the bending direction, which means the total external moment in the lateral direction was zero: i=1 where Mp,i is the external moment applied by the ith cable in the lateral direction. n (cid:88) Mp,i = 0 (6.18) 109 The arm of the external forces for Mp,i was the distance between the bending place of ith cable and the bending plane of the backbone (Fig. 6.4B), i.e.: rp,i = d sin βi (6.19) Thus, the lateral moment Mp,i applied by the ith cable was derived as: Mp,i = FT,i × rp,i + (cid:90) ϕb−θ0 θ0 ρe i (s) × rp,ids = −Fr,i × rp,i (6.20) Then, based on equations (6.16), (6.17), (6.18), (6.20) and the geometry relationships, the kinematic model for a soft section of the arm actuated by multiple cables was derived as: n (cid:88) i=1 n (cid:88) i=1 Tid cos θ0,i cos βi = Kbκc Tid cos θ0,i sin βi = 0 θ0,i = α − arcsin((1 − κcdi) κg,i κc sin α) Ti = Kc κg,i {( 1 κc − di)(1 − cos α) − 1 κg,i (1 − cos(α − θ0,i))} li = Rg,i(ϕb − 2θ0,i) = 1 κg,i (Lκc − 2θ0,i) (6.21) 6.2.3 Modeling for Multiple-Section Soft Arm For the multi-section soft robotic arm, the kinematic model between the task space (end position) and the bending configuration for each section was then built by using translation homogeneous matrix. Specifically, considering the thickness of the rigid endcaps, each section of the arm can be divided into three parts: straight, bending, and straight. The translation matrix T i,s i,e from the end frame Σi,e to the base frame Σi,s for the ith section (Fig. 6.5A) was derived as: 110          1 p0 Rz(γ) 0 Ry(ϕb) p Rz(−γ) 0 T i,s i,e =       0 1 0     1     0 1 0   1 p0 0 1   1    (6.22) where p = [1 − cos ϕb/κc 0 sin ϕb/κc]T is the in-plane displacement of the bending part, p0 = [0 0 h]T is the displacement of the straight part and h is the thickness of the endcaps. Rz(γ), Ry(ϕb), and Rz(−γ) are the 3D rotation matrices around the z-axis, y-axis and z-axis for the angle γ, ϕb, and −γ, respectively. Figure 6.5 Modeling for the multi-section soft robotic arm. (A). Variables of bending config- uration for one section. (B). Local frames for different sections. (C). The inverse kinematics solver for the bending configurations of different sections based on the reference of the end position. (D). Open loop control of the soft robotics arm based on the proposed model. Then, the translation matrix T 1,s n,e from the end Σn,e to the base Σ1,s of the whole arm, and the end position of the arm p1,s t in Σ1,s (Fig. 6.5B) were calculated: T 1,s n,e = n (cid:89) i=1 T i,s i,e (6.23) p1,s t = T 1,s n,e · pn,e t where pn,e t = [0 0 0 1]T . The Jacobian matrix J for the end position of the arm with respect to the variables of bending configuration of each section was calculated as:   J =   Jv 0   = ∂p1,s t ∂q (6.24) 111 where q = [γ1 κc,1 · · · γn κc,n]T is the variables of bending configuration of each section. The calculated Jacobian is omitted for brevity and a small value δϕ is added to ϕ when ϕ → 0 for numerical stability. At least two sections are required for the arm to provide redundancy for tracking the end positions. Once the Jacobian matrix was obtained, a general inverse velocity kinematics for the soft arm can be acquired by using the pseudo inverse of the Jacobian matrix J † v : ˙q = J † v V + (I − J † v Jv) ˙q0 (6.25) v = J T J † v (JvJ T v + k2I)−1 where I is an identity matrix and k is a small number for a damping solution and V = ˙xd(t) is the velocity vector of the tracking trajectory xd(t). ˙q0 is set to zero for the minimum energy criteria. By using a numerical method to integrate the velocities, the references for the bending configuration variables were calculated: q(tk+1) = q(tk) + J † v (q(tk)) · ˙xd(tk) · ∆t (6.26) where q(tk) and q(tk+1) denote q in time step tk and tk+1, respectively. Furthermore, a closed-loop control was implemented (Fig. 6.5C) to eliminate the tracking error accumulated by the numerical integration in the inverse kinematics solver: ˙q = J † v (q)( ˙xd + Ke) (6.27) where e is the tracking error and K is a positive diagonal gain matrix. Thus, the kinematic model between the task space and the bending configuration for different sections was developed, which further derived the complete kinematic model for the proposed soft robotic arm when combined with the kinematic model for a single soft section. 112 6.3 Results 6.3.1 Baseline model for the soft robotic arm Extensive experiments have been conducted to validate the proposed kinematic model. Before the experiments, a baseline model for the arm was built without considering the transverse deformation of the cable during the bending configuration. The multi-section model parts of the baseline model and the proposed static model were the same (Fig. 6.1E). Rather than static analysis, the baseline model for the single section of the arm was derived based on geometry analysis of the cable and the bending section. The curvature κ(g, i) and the length li of the ith cable in a section (blue curve in Fig. 6.3A) was derived as: 1 κc = 1 κg + di li = Rg,i · ϕb = κc κg,i L (6.28) where di is derived from equation (6.15), κc and L are the curvature and the length of the backbone, respectively. Thus, the baseline kinematic model for the soft robotic arm was built by combining equation (6.28) and the model for the multi-section soft robotic arm. 6.3.2 Parameter Identification of the soft robotic arm Before the experiments, the model parameters of the arm were identified. The geometric parameters (L, d) were measured directly from the prototype. The bending stiffness Kb for a single section of the arm was calculated by using equations (6.9) and (6.12) assuming θ0 ≈ 0 when ϕb was small, where T and ϕb were measured by a force sensor and a motion capture system, respectively. During the experiments for the soft robotic arm, 5 optical markers were attached at the end of the robotic arm for monitoring the bending configuration and the end position of the robotic arm under a motion capture system (Fig. 6.7). The motion capture system used in the experiments was “Opti-track”, including a set of infrared cameras and the related software, which could track the position and orientation of the “Rigid Body” consisting of the attached markers on the robotic arm and provide accurate information for the bending 113 angle ϕb, bending orientation γ, and the position for the end of the robotic arm. Figure 6.6 Overall configuration of the experimental setups for the soft robotic arm. In the calibration experiments of the parameter Kb, a force meter was used to measure the cable tension T manually when the bending angle ϕb was recorded by the motion capture system (Fig. 6.7). Thus, Kb can be calculated by using the equation (6.9) and (6.12) assuming θ0 ≈ 0 when ϕb was small. Figure 6.7 Diagram of the calibration experiments for the parameter Kb. In the calibration experiments of the parameter Kc, the cable length was controlled by using a stepper motor and the bending angle ϕb was recorded by the motion capture system. The actuation cable was driven by a 3D-printed pulley which was attached to a stepper motor (NEMA-17, Adafruit) (Fig. 6.8A). The stepper motor was controlled by a micro- controller (An Arduino Mega 2560) in an open loop control (without encoder feedback) with 114 the help of a stepper motor driver (Adafruit Motor Shield V2) for power amplification (Fig. 6.8B). Then, via serial communication, the microcontroller was communicated with a main controller (laptop), where the analytical models were computed before the reference for the stepper motor was sent out. Figure 6.8 (A). Cable actuation mechanism for the soft robotic arm. (B). Diagram of the control and actuation system for the soft robotic arm. Thus, the relationship between the cable contraction lengths ∆l and ϕb was obtained in the experiments for one section driven by a single cable to identify Kc by using equation (4.13). The calibration result showed that, compared with the baseline model, the pro- posed kinematic model was better at describing the nonlinear relationship between the cable actuation and the bending angle for the soft robotic arm (Fig. 6.9B). 6.3.3 Experiments for a single section of the soft robotic arm After the model parameters were identified, Extensive experiments were conducted to compare the accuracy of the baseline model and the proposed static model, where an open loop control without feedback was used (Fig. 6.5D). A single section of the robotic arm was used to track different bending angles ϕb in specific bending orientations γ by using different models (Fig. 6.9A). In the experiments, it was shown that the tracking accuracy of the proposed static model was better compared to the baseline model in the experimental range (Fig. 6.9C-G), indi- cating the importance of considering the transverse deformation of the cable in the proposed robotic arm. The tracking error of the ϕb and γ kept small for different bending config- urations when the proposed model was used, which might be attributed to the structure 115 inaccuracy and the hysteresis of the soft material. The ϕb tracking error of the baseline model increased together with the target ϕb when the target γ was fixed, while the γ track- ing error of the baseline model almost kept constant despite the changing of the target ϕb when the target γ was fixed. Figure 6.9 Experiment results for a single section of the soft robotic arm. (A). The bending configuration of one section in experiments. (B). The relationship between cable contraction ∆l = L − l and bending angle ϕb with a single cable actuation. (C). Tracking different bending angle ϕb when γ = 0◦ using the baseline model (BA) and the proposed model (ST). (D). Tracking different bending angle ϕb when γ = 15◦. (E). Tracking different bending angle ϕb when γ = 30◦. (F). Tracking different bending angle ϕb when γ = 15◦. (G). Tracking different bending angle ϕb when γ = 60◦. Moreover, it was also shown that in the experiment range, ϕb tracking error of the baseline model increased with larger target γ when the target ϕb was fixed. The maximum ϕb tracking error increased from about 12 degrees to about 37 degrees when the target γ increased from 0 116 to 60 degrees, respectively. The γ tracking error of the baseline model increased from about 0 to about 16 degrees when the target γ increased from 0 to 30 degrees, respectively, and then decreased to near 0 when the target γ decreased to 60 degrees. It was noticed that for both models, the γ tracking error approached 0 when target γ was 0 and 60 degrees, which was attributed to a single effective cable contraction and two effective cable contraction with same contraction length, respectively. Then, a trajectory tracking experiment for the end of the single section was conducted where the trajectory reference was included in the workspace of the single section. The experiment showed that the tracking error of the single section by using the proposed model was smaller than that of the baseline model (Fig. 6.10A), and the soft section had flexible and versatile bending configurations (Fig. 6.10B-C). Figure 6.10 Experiment results for a single section of the soft robotic arm tracking a circular trajectory. (A). Trajectories of the end position of the single section by using the baseline model (BA) and the proposed model (ST). (B). Movement and bending configurations of the single section by using BA. (C). Movement and bending configurations of the single section by using ST. 117 6.3.4 Experiments for a two-section soft robotic arm A multi-section arm was assembled and utilized for the comparison of the baseline model and the proposed model, and its performance was further evaluated for potential applications. For simplicity, a two-section arm was assembled and controlled to track a circular trajectory within its workspace by using the baseline and the proposed model. The experiment results showed that compared with the baseline model, the tracking error of the proposed model was smaller, and the trajectory of the proposed model was closer to the reference (Fig. 6.11A-C). Compared with the tracking error by using a single section, the tracking error of the two-section soft arm was larger, which might be attributed to the error accumulation of multiple sections and a more prominent gravity influence for the base section of the soft arm. In addition, the two-section arm was controlled to track a straight trajectory within its workspace by using the different models, where the tracking error of the proposed model was smaller than the baseline model (Fig. 7D-F). In summary, the extensive experiments showed the advantage of the proposed kinematic model as compared to the geometric baseline model and validated the flexibility and dexterity of the proposed soft robotic arm. 6.4 Discussion In this work, we designed an octopus-inspired soft robotic arm and developed a related kinematic model to characterize its flexible movements. The modular design of the soft arm enabled longer arm prototypes and permitted a decoupling cable actuation system for different sections that simplified the modeling. The hybrid fabrication method of 3D printing and casting achieved low-cost and easy-to-build prototypes. An analytical static model was built to capture the transverse deformation of the cable during actuation for better accuracy, while a kinematic model between bending configuration for a single section and task space for the whole arm was built for motion planning. Extensive experiments were conducted to validate the proposed model and a geometric baseline model was used for comparison. The results of tracking experiments for a single sec- 118 Figure 6.11 Experiment results for a two-section soft robotic arm. (A). Trajectories of the end position of the arm tracking a circular path by using the baseline model (BA) and the proposed static model (ST). (B). Movement and bending configurations of the arm tracking a circular path by using BA. (C). Movement and bending configurations of the arm tracking a circular path by using ST. (D). Trajectories of the end position of the arm tracking a straight path by using BA and ST. (E). Movement and bending configurations of the arm tracking a straight path by using BA. (F). Movement and bending configurations of the arm tracking a straight path by using ST. 119 tion of the soft arm showed an obvious advantage and smaller tracking error of the proposed model over the baseline model in terms of bending angle, orientation, and the end position of the arm, which was also validated by a two-section arm in its tracking experiments for circular and straight trajectories that revealed the dexterity of the proposed soft arm. For future works, we will further optimize the analytical model for the proposed soft robotic arm by considering the gravity influence and hysteresis effect of the soft material before we apply a closed-loop control for more accurate applications. We will also explore integrated embedded sensors for the soft robotic arm which can provide bending configuration data of the arm for the closed-loop controls. Besides, we will embed fluid-driven actuators in the arm’s fluid cavities to build a stiffness control mechanism for applications with heavier workload. Finally, we will explore applications for the soft robotics arm such as medical and harvesting devices. 120 CHAPTER 7 CONCLUSION This work aims to provide new bio-inspired soft robotic solutions for complex and auto- matic applications such as rescue, invasive surgery, and human-machine interactions. The novel and compact robotic systems and platforms developed in this work were inspired by snakes and octopus tentacles and contributed to the field of soft robotics in multiple ways. First, inspired by the movement of real snakes in nature, we developed a novel pneu- matic soft snake robot using traveling-wave locomotion in a constrained environment. The pneumatic system and the control scheme design enabled the snake robot to generate the traveling-wave deformation in 2D space by only using four air paths, which made the robot more compact and robust for inspection in constrained environments. The 3D printing method was used to achieve a low-cost and lightweight fabrication for the complete soft snake robot. FEM modeling was used to simulate the bending deformation and the locomo- tion of the robot in ANSYS and SOFA, respectively, before the validation of experiments. The experiments showed good agreement with the model prediction and the speed of the proposed snake robot reached 18 mm/s with a maximum pressure of 172 kPa (25 psi) and a friction coefficient of 0.65 in a 2-inch pipe. Furthermore, additional tests showed that the snake robot moved smoothly in pipes of different diameters and of complex geometry including sharp turns. After the locomotion in constrained environments, the serpentine locomotion of the snakes was studied, based on which a novel soft snakeskin was designed and mounted on the robot. The proposed snakeskin balanced the stretchability and friction anisotropy by using a soft skin base and embedded rigid scales, which was achieved by using a multi-material 3D printing method. The snakeskin was combined with the robot to generate serpentine locomotion on a canvas with a speed of 37 mm/s when the pressure magnitude was 172 kPa (25 psi) after the validation of its friction anisotropy. The performance of different scale shapes and arrangements was evaluated in the experiment for the best design parameters. 121 Furthermore, it was also demonstrated that the steering of the robot could be achieved and the robot could slither over various outdoor surfaces. A model-free reinforcement learning approach was used to control the snake robot in a locomotion and navigation task because of the complexity of the dynamic model of the serpentine locomotion of the soft snake robot. In this work, we proposed a new technique, Back-stepping Experience Replay, which is compatible with arbitrary off-policy RL algo- rithms, to enhance learning efficiency. BER helped reduce the complexity in reward shaping for systems with approximate reversibility, and was used in the application of locomotion and navigation tasks of the soft snake robot, where a dynamic simulator was developed to assess the efficiency of the algorithms. With the help of BER, the robot demonstrated a 100 % success rate for the tasks with random targets and achieved an average speed 48 % faster than that of the best baseline approach. Furthermore, we proposed a soft snake robot that could generate a 3D spiral traveling wave deformation by only using three air paths, which could propel the robot in both con- strained environments and flat surfaces without the usage of artificial snakeskin. A bending actuator was adopted as the building block of the modular snake robot and the related cast- ing fabrication method was developed. The actuation strategy of the pressures in air paths was proposed to generate the traveling wave and the locomotion methods of the snake robot in constrained environments and flat surfaces were analyzed. The proposed soft snake was not only able to move in constrained environments adaptably, but also able to move on flat surfaces without the usage of an auxiliary snakeskin. Finally, we designed an octopus-inspired soft robotic arm along with a static model describing the relationship between the arm deformation and the cable actuation. The soft robotic arm was modular and fabricated with 3D printing and casting. A model was built to capture the transverse deformation of the cable during the actuation while a kinematic model was used to describe the configuration of the soft robotic arm. Extensive experiments showed good agreement with the predictions of the proposed model compared with a geometric 122 baseline model and also demonstrated the flexibility and dexterity of the proposed soft robotic arm. 123 BIBLIOGRAPHY [1] [2] Jérémie Allard, Stéphane Cotin, François Faure, Pierre-Jean Bensoussan, François Poyer, Christian Duriez, Hervé Delingette, and Laurent Grisoni. Sofa-an open source framework for medical simulation. In MMVR 15-Medicine Meets Virtual Reality, vol- ume 125, pages 13–18. IOP Press, 2007. Iain A Anderson, Todd A Gisby, Thomas G McKay, Benjamin M O’Brien, and Emilio P Calius. Multi-functional dielectric elastomer artificial muscles for soft and smart ma- chines. Journal of applied physics, 112(4):041101, 2012. [3] Marcin Andrychowicz et al. Hindsight experience replay. Advances in Neural Infor- mation Processing Systems, 30, 2017. [4] Dimuthu DK Arachchige, Dulanjana M Perera, Sanjaya Mallikarachchi, Iyad Kanj, Yue Chen, Hunter B Gilbert, and Isuru S Godage. Dynamic modeling and validation of soft robotic snake locomotion. arXiv preprint arXiv:2303.02291, 2023. [5] Dimuthu DK Arachchige, Dulanjana M Perera, Sanjaya Mallikarachchi, Iyad Kanj, Yue Chen, and Isuru S Godage. Wheelless soft robotic snake locomotion: Study on sidewinding and helical rolling gaits. In 2023 IEEE International Conference on Soft Robotics (RoboSoft), pages 1–6. IEEE, 2023. [6] Dimuthu DK Arachchige, Dulanjana M Perera, Sanjaya Mallikarachchi, Iyad Kanj, Yue Chen, and Isuru S Godage. Wheelless soft robotic snake locomotion: Study on sidewinding and helical rolling gaits. arXiv preprint arXiv:2303.02285, 2023. [7] [8] Henry C Astley, Jennifer M Rieser, Abdul Kaba, Veronica M Paez, Ian Tomkinson, Joseph R Mendelson, and Daniel I Goldman. Side-impact collision: mechanics of ob- stacle negotiation in sidewinding snakes. Bioinspiration & Biomimetics, 15(6):065005, 2020. Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, An- dew Bolt, et al. Never give up: Learning directed exploration strategies. arXiv preprint arXiv:2002.06038, 2020. [9] Michael D Bartlett, Andrew B Croll, Daniel R King, Beth M Paret, Duncan J Irschick, and Alfred J Crosby. Looking beyond fibrillar features to scale gecko-like adhesion. Advanced Materials, 24(8):1078–1083, 2012. [10] Antonio Bicchi, Giovanni Tonietti, and E Piaggio. Design, realization and control of soft robot arms for intrinsically safe interaction with humans. In Proc. IARP/RAS Workshop on Technical Challenges for Dependable Robots in Human Environments, pages 79–87, 2002. 124 [11] Callie Branyan, Chloë Fleming, Jacquelin Remaley, Ammar Kothari, Kagan Tumer, Ross L Hatton, and Yiğit Mengüç. Soft snake robots: Mechanical design and geo- metric gait implementation. In 2017 IEEE International Conference on Robotics and Biomimetics (ROBIO), pages 282–289. IEEE, 2017. [12] Callie Branyan, Ross L Hatton, and Yiğit Mengüç. Snake-inspired kirigami skin for IEEE Robotics and Automation Letters, lateral undulation of a soft snake robot. 5(2):1728–1733, 2020. [13] Callie Branyan and Yiğit Menğüç. Soft snake robots: investigating the effects of gait parameters on locomotion in complex terrains. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1–9. IEEE, 2018. [14] David B Camarillo, Christopher R Carlson, and J Kenneth Salisbury. Configuration tracking for continuum manipulators with coupled tendon drive. IEEE transactions on robotics, 25(4):798–808, 2009. [15] David B Camarillo, Christopher F Milne, Christopher R Carlson, Michael R Zinn, and J Kenneth Salisbury. Mechanics modeling of tendon-driven continuum manipulators. IEEE transactions on robotics, 24(6):1262–1273, 2008. [16] Li Chen, Shugen Ma, Yuechao Wang, Bin Li, and Dengping Duan. Design and mod- elling of a snake robot in traveling wave locomotion. Mechanism and Machine Theory, 42(12):1632–1642, 2007. [17] Xiaoqian Chen, Xiang Zhang, Yiyong Huang, Lu Cao, and Jinguo Liu. A review of soft manipulator research, applications, and opportunities. Journal of Field Robotics, 39(3):281–311, 2022. [18] Hyouk Ryeol Choi, Kwangmok Jung, Sungmoo Ryew, Jae-Do Nam, Jaewook Jeon, Ja Choon Koo, and Kazuo Tanie. Biomimetic soft actuator: design, modeling, control, and applications. IEEE/ASME transactions on mechatronics, 10(5):581–593, 2005. [19] Leshem Choshen, Lior Fox, and Yonatan Loewenstein. Dora the explorer: Directed outreaching reinforcement action-selection. arXiv preprint arXiv:1804.04012, 2018. [20] Matteo Cianchetti, Tommaso Ranzani, Giada Gerboni, Iris De Falco, Cecilia Laschi, and Arianna Menciassi. Stiff-flop surgical manipulator: Mechanical design and ex- In 2013 IEEE/RSJ international perimental characterization of the single module. conference on intelligent robots and systems, pages 3576–3581. IEEE, 2013. [21] Alessandro Crespi and Auke Jan Ijspeert. Amphibot ii: An amphibious snake robot that crawls and swims using a central pattern generator. In Proceedings of the 9th international conference on climbing and walking robots (CLAWAR 2006), number CONF, pages 19–27, 2006. 125 [22] Alessandro Crespi and Auke Jan Ijspeert. Online optimization of swimming and crawl- ing in an amphibious snake robot. IEEE Transactions on robotics, 24(1):75–87, 2008. [23] Tao Deng, Hesheng Wang, Weidong Chen, Xiaozhou Wang, and Rolf Pfeifer. Develop- ment of a new cable-driven soft robot for cardiac ablation. In 2013 IEEE International Conference on Robotics and Biomimetics (ROBIO), pages 728–733. IEEE, 2013. [24] PL Dickrell, SB Sinnott, DW Hahn, NR Raravikar, LS Schadler, PM Ajayan, and WG Sawyer. Frictional anisotropy of oriented carbon nanotube surfaces. Tribology Letters, 18:59–62, 2005. [25] Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. Advances in Neural Information Processing Systems, 32, 2019. [26] Alessandro Diodato, Margherita Brancadoro, Giacomo De Rossi, Haider Abidi, Diego Dall’Alba, Riccardo Muradore, Gastone Ciuti, Paolo Fiorini, Arianna Menciassi, and Matteo Cianchetti. Soft robotic manipulator for improving dexterity in minimally invasive surgery. Surgical innovation, 25(1):69–76, 2018. [27] Weiqiang Dou, Guoliang Zhong, Jinglin Cao, Zhun Shi, Bowen Peng, and Liangzhong Jiang. Soft robotic manipulators: Designs, actuation, stiffness tuning, and sensing. Advanced Materials Technologies, 6(9):2100018, 2021. [28] Christian Duriez and Thor Bieze. Soft robot modeling, simulation and control in real- time. In Soft Robotics: Trends, Applications and Challenges: Proceedings of the Soft Robotics Week, April 25-30, 2016, Livorno, Italy, pages 103–109. Springer, 2017. [29] N Elango and Ahmad Athif Mohd Faudzi. A review article: investigations on soft materials for soft robot manipulations. The International Journal of Advanced Manu- facturing Technology, 80:1027–1037, 2015. [30] Ismet Erkmen, Aydan M Erkmen, Fumitoshi Matsuno, Ranajit Chatterjee, and Tet- sushi Kamegawa. Snake robots to the rescue! IEEE Robotics & Automation Magazine, 9(3):17–25, 2002. [31] Meng Fang, Cheng Zhou, Bei Shi, Boqing Gong, Jia Xu, and Tong Zhang. Dher: Hind- sight experience replay for dynamic goals. In International Conference on Learning Representations, 2018. [32] Kevin C Galloway, Panagiotis Polygerinos, Conor J Walsh, and Robert J Wood. Me- chanically programmable bend radius for fiber-reinforced soft actuators. In 2013 16th International Conference on Advanced Robotics (ICAR), pages 1–6. IEEE, 2013. [33] Junyao Gao, Xueshan Gao, Wei Zhu, Jianguo Zhu, and Boyu Wei. Design and research of a new structure rescue snake robot with all body drive system. In 2008 IEEE In- 126 ternational Conference on Mechatronics and Automation, pages 119–124. IEEE, 2008. [34] Zheyuan Gong, Xi Fang, Xingyu Chen, Jiahui Cheng, Zhexin Xie, Jiaqi Liu, Bohan Chen, Hui Yang, Shihan Kong, Yufei Hao, et al. A soft manipulator for efficient delicate grasping in shallow water: Modeling, control, and real-world experiments. The International Journal of Robotics Research, 40(1):449–469, 2021. [35] Stanislao Grazioso, Giuseppe Di Gironimo, and Bruno Siciliano. A geometrically exact model for soft continuum robots: The finite element deformation space formulation. Soft robotics, 6(6):790–811, 2019. [36] J Hazel, M Stone, MS Grace, and VV Tsukruk. Nanoscale design of snake skin for reptation locomotions via friction anisotropy. Journal of biomechanics, 32(5):477–484, 1999. [37] A Hisham et al. On surface structure and friction regulation in reptilian limbless locomotion. Journal of the mechanical behavior of biomedical materials, 22:115–135, 2013. [38] Jesse Horne, Lauren McLoughlin, Elizabeth Bury, Amanda S Koh, and Evan K Wujcik. Interfacial phenomena of advanced composite materials toward wearable platforms for biological and environmental monitoring sensors, armor, and soft robotics. Advanced Materials Interfaces, 7(4):1901851, 2020. [39] David L Hu, Jasmine Nirody, Terri Scott, and Michael J Shelley. The mechanics of slithering locomotion. Proceedings of the National Academy of Sciences, 106(25):10081– 10085, 2009. [40] David L Hu, Jasmine Nirody, Terri Scott, and Michael J Shelley. The mechanics of slithering locomotion. Proceedings of the National Academy of Sciences, 106(25):10081– 10085, 2009. [41] David L Hu and Michael Shelley. Slithering locomotion. In Natural locomotion in fluids and on surfaces: swimming, flying, and sliding, pages 117–135. Springer, 2012. [42] Min Hua, Dong Chen, Xinda Qi, Kun Jiang, Zemin Eitan Liu, Quan Zhou, and Hong- ming Xu. Multi-agent reinforcement learning for connected and automated vehicles control: Recent advancements and future prospects. arXiv preprint arXiv:2312.11084, 2023. [43] Kang Huang, Ke Shao, Shengchao Zhen, and Hao Sun. A novel approach for mod- eling and tracking control of a passive-wheel snake robot. Advances in Mechanical Engineering, 9(3):1687814017693944, 2017. [44] Lin Huang and Alejandro Martinez. Load transfer anisotropy at snakeskin-inspired 127 clay-structure interfaces. In IFCEE 2021, pages 119–129. 2021. [45] Sichuan Huang, Yong Tang, Hosain Bagheri, Dongting Li, Alexandria Ardente, Daniel Aukes, Hamidreza Marvi, and Junliang Tao. Effects of friction anisotropy on upward burrowing behavior of soft robots in granular materials. Advanced Intelligent Systems, 2(6):1900183, 2020. [46] Naim Md Lutful Huq, Md Raisuddin Khan, Amir Akramin Shafie, Md Masum Billah, and Syed Masrur Ahmmad. Motion investigation of a snake robot with different scale geometry and coefficient of friction. Robotics, 7(2):18, 2018. [47] Zhongying Ji, Changyou Yan, Shuanhong Ma, Stanislav Gorb, Xin Jia, Bo Yu, Xiao- long Wang, and Feng Zhou. 3d printing of bioinspired topographically oriented surfaces with frictional anisotropy for directional driving. Tribology International, 132:99–107, 2019. [48] Rianna Jitosho, Tyler Ga Wei Lum, Allison Okamura, and Karen Liu. Reinforcement learning enables real-time planning and control of agile maneuvers for soft robot arms. In Conference on Robot Learning, pages 1131–1153. PMLR, 2023. [49] Karen Junius, Marta Moltedo, Pierre Cherelle, Carlos Rodriguez-Guerrero, Bram Van- derborght, and Dirk Lefeber. Biarticular elements as a contributor to energy effi- ciency: biomechanical review and application in bio-inspired robotics. Bioinspiration & biomimetics, 12(6):061001, 2017. [50] Tetsushi Kamegawa, Taichi Akiyama, Satoshi Sakai, Kento Fujii, Kazushi Une, Eitou Ou, Yuto Matsumura, Toru Kishutani, Eiji Nose, Yusuke Yoshizaki, et al. Development of a separable search-and-rescue robot composed of a mobile robot and a snake robot. Advanced Robotics, 34(2):132–139, 2020. [51] Sangbae Kim, Cecilia Laschi, and Barry Trimmer. Soft robotics: a bioinspired evolu- tion in robotics. Trends in biotechnology, 31(5):287–294, 2013. [52] Yoonho Kim, German A Parada, Shengduo Liu, and Xuanhe Zhao. Ferromagnetic soft continuum robots. Science Robotics, 4(33):eaax7329, 2019. [53] Je-Sung Koh, Eunjin Yang, Gwang-Pil Jung, Sun-Pill Jung, Jae Hak Son, Sang-Im Lee, Piotr G Jablonski, Robert J Wood, Ho-Young Kim, and Kyu-Jin Cho. Jumping on water: Surface tension–dominated jumping of water striders and robotic insects. Science, 349(6247):517–521, 2015. [54] Cecilia Laschi, Matteo Cianchetti, Barbara Mazzolai, Laura Margheri, Maurizio Fol- lador, and Paolo Dario. Soft robot arm inspired by the octopus. Advanced robotics, 26(7):709–727, 2012. 128 [55] Chiwon Lee, Myungjoon Kim, Yoon Jae Kim, Nhayoung Hong, Seungwan Ryu, H Jin Kim, and Sungwan Kim. Soft robot review. International Journal of Control, Au- tomation and Systems, 15:3–15, 2017. [56] Chiwon Lee, Myungjoon Kim, Yoon Jae Kim, Nhayoung Hong, Seungwan Ryu, H Jin Kim, and Sungwan Kim. Soft robot review. International Journal of Control, Au- tomation and Systems, 15:3–15, 2017. [57] Changquing Li and Christopher D Rahn. Design of continuous backbone, cable-driven robots. J. Mech. Des., 124(2):265–271, 2002. [58] Xinquan Liang, Haris Cheong, Yi Sun, Jin Guo, Chee Kong Chui, and Chen-Hua Yeow. Design, characterization, and implementation of a two-dof fabric-based soft robotic arm. IEEE Robotics and Automation Letters, 3(3):2702–2709, 2018. [59] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep rein- forcement learning. arXiv preprint arXiv:1509.02971, 2015. [60] Kevin Lipkin, Isaac Brown, Aaron Peck, Howie Choset, Justine Rembisz, Philip Gi- anfortoni, and Allison Naaktgeboren. Differentiable and piecewise differentiable gaits for snake robots. In 2007 IEEE/RSJ international conference on intelligent robots and systems, pages 1864–1869. IEEE, 2007. [61] Xuan Liu, Renato Gasoto, Ziyi Jiang, Cagdas Onal, and Jie Fu. Learning to locomote with artificial neural-network and cpg-based control in a soft snake robot. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7758–7765. IEEE, 2020. [62] Xuan Liu, Cagdas D Onal, and Jie Fu. Reinforcement learning of cpg-regulated loco- motion controller for a soft snake robot. IEEE Transactions on Robotics, 2023. [63] Ming Luo, Mahdi Agheli, and Cagdas D Onal. Theoretical modeling and experimental analysis of a pressure-operated soft robotic snake. Soft Robotics, 1(2):136–146, 2014. [64] Ming Luo, Mahdi Agheli, and Cagdas D Onal. Theoretical modeling and experimental analysis of a pressure-operated soft robotic snake. Soft Robotics, 1(2):136–146, 2014. [65] Ming Luo, Yixiao Pan, Erik H Skorina, Weijia Tao, Fuchen Chen, Selim Ozel, and Cagdas D Onal. Slithering towards autonomy: a self-contained soft robotic snake plat- form with integrated curvature sensing. Bioinspiration & biomimetics, 10(5):055001, 2015. [66] Ming Luo, Weijia Tao, Fuchen Chen, Tri K Khuu, Selim Ozel, and Cagdas D Onal. Design improvements and dynamic characterization on fluidic elastomer actuators for 129 a soft robotic snake. Practical Robot Applications (TePRA), pages 1–6. IEEE, 2014. In 2014 IEEE International Conference on Technologies for [67] Ming Luo, Ruibo Yan, Zhenyu Wan, Yun Qin, Junius Santoso, Erik H Skorina, and Cagdas D Onal. Orisnake: Design, fabrication, and experimental analysis of a 3-d origami snake robot. IEEE Robotics and Automation Letters, 3(3):1993–1999, 2018. [68] Ajay Malshe, Kamlakar Rajurkar, Anoop Samant, Hans Nørgaard Hansen, Salil Bapat, and Wenping Jiang. Bio-inspired functional surfaces for advanced applications. CIRP Annals, 62(2):607–628, 2013. [69] Hamidreza Marvi, Gregory Meyers, Geoffrey Russell, and David L Hu. Scalybot: a snake-inspired robot with active control of friction. In Dynamic Systems and Control Conference, volume 54761, pages 443–450, 2011. [70] Fumitoshi Matsuno and Hiroki Sato. Trajectory tracking control of snake robots based In Proceedings of the 2005 IEEE International Conference on on dynamic model. Robotics and Automation, pages 3029–3034. Ieee, 2005. [71] Ebrahim Mattar. A survey of bio-inspired robotics hands implementation: New direc- tions in dexterous manipulation. Robotics and Autonomous Systems, 61(5):517–544, 2013. [72] James C McKenna, David J Anhalt, Frederick M Bronson, H Ben Brown, Michael Schwerin, Elie Shammas, and Howie Choset. Toroidal skin drive for snake robot loco- motion. In 2008 IEEE International Conference on Robotics and Automation, pages 1150–1155. IEEE, 2008. [73] Volodymyr Mnih, Andreas K others, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015. [74] Thor Morales Bieze, Alexandre Kruszewski, Bruno Carrez, and Christian Duriez. De- sign, implementation, and control of a deformable manipulator robot based on a com- pliant spine. The International Journal of Robotics Research, 39(14):1604–1619, 2020. [75] Seri Mastura Mustaza, Yahya Elsayed, Constantina Lekakou, Chakravarthini Saaj, and Jan Fras. Dynamic modeling of fiber-reinforced soft manipulator: A visco-hyperelastic material-based continuum mechanics approach. Soft robotics, 6(3):305–317, 2019. [76] Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In International Confer- ence on Machine Learning, volume 99, pages 278–287. Citeseer, 1999. [77] Cagdas D Onal and Daniela Rus. A modular approach to soft robots. In 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics 130 (BioRob), pages 1038–1045. IEEE, 2012. [78] Cagdas D Onal and Daniela Rus. Autonomous undulatory serpentine locomotion uti- lizing body dynamics of a fluidic soft robot. Bioinspiration & biomimetics, 8(2):026003, 2013. [79] Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. Count-based ex- ploration with neural density models. In International Conference on Machine Learn- ing, pages 2721–2730. PMLR, 2017. [80] Shingo Ozaki, K Hikida, and K Hashiguchi. Elastoplastic formulation for friction with orthotropic anisotropy and rotational hardening. International Journal of Solids and Structures, 49(3-4):648–657, 2012. [81] Pierpaolo Palmieri, Matteo Melchiorre, and Stefano Mauro. Design of a lightweight and deployable soft robotic arm. Robotics, 11(5):88, 2022. [82] Jeong Young Park, DF Ogletree, M Salmeron, RA Ribeiro, PC Canfield, CJ Jenks, and PA Thiel. High frictional anisotropy of periodic and aperiodic directions on a quasicrystal surface. Science, 309(5739):1354–1356, 2005. [83] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven In International Conference on Machine exploration by self-supervised prediction. Learning, pages 2778–2787. PMLR, 2017. [84] Bryan N Peele, Thomas J Wallin, Huichan Zhao, and Robert F Shepherd. 3d printing antagonistic systems of artificial muscle using projection stereolithography. Bioinspi- ration & biomimetics, 10(5):055003, 2015. [85] Rolf Pfeifer, Max Lungarella, and Fumiya Iida. Self-organization, embodiment, and biologically inspired robotics. science, 318(5853):1088–1093, 2007. [86] Panagiotis Polygerinos, Zheng Wang, Johannes TB Overvelde, Kevin C Galloway, Robert J Wood, Katia Bertoldi, and Conor J Walsh. Modeling of soft fiber-reinforced bending actuators. IEEE Transactions on Robotics, 31(3):778–789, 2015. [87] Xinda Qi, Tong Gao, and Xiaobo Tan. Bioinspired 3d-printed snakeskins enable effec- tive serpentine locomotion of a soft robotic snake. Soft Robotics, 2022. [88] Xinda Qi, Tong Gao, and Xiaobo Tan. Bioinspired 3d-printed snakeskins enable ef- fective serpentine locomotion of a soft robotic snake. Soft Robotics, 10(3):568–579, 2023. [89] Xinda Qi, Hongyang Shi, Thassyo Pinto, and Xiaobo Tan. A novel pneumatic soft IEEE snake robot using traveling-wave locomotion in constrained environments. 131 Robotics and Automation Letters, 5(2):1610–1617, 2020. [90] Xinda Qi, Hongyang Shi, Thassyo Pinto, and Xiaobo Tan. A novel pneumatic soft IEEE snake robot using traveling-wave locomotion in constrained environments. Robotics and Automation Letters, 5(2):1610–1617, 2020. [91] Ahmad Rafsanjani, Yuerou Zhang, Bangyuan Liu, Shmuel M Rubinstein, and Ka- tia Bertoldi. Kirigami skins make a simple soft actuator crawl. Science Robotics, 3(15):eaar7555, 2018. [92] Federico Renda, Michele Giorelli, Marcello Calisti, Matteo Cianchetti, and Cecilia Laschi. Dynamic model of a multibending soft robot arm driven by cables. IEEE Transactions on Robotics, 30(5):1109–1122, 2014. [93] David Rollinson and Howie Choset. Pipe network locomotion with a snake robot. Journal of Field Robotics, 33(3):322–336, 2016. [94] Wael Saab, Peter Racioppo, Anil Kumar, and Pinhas Ben-Tzvi. Design of a miniature modular inchworm robot with an anisotropic friction skin. Robotica, 37(3):521–538, 2019. [95] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [96] Donghua Shen, Qi Zhang, Cunjin Wang, Xingsong Wang, and Mengqian Tian. Design and analysis of a snake-inspired crawling robot driven by alterable angle scales. IEEE Robotics and Automation Letters, 6(2):3744–3751, 2021. [97] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489, 2016. [98] Koichi Suzumori, Satoshi Endo, Takefumi Kanda, Naomi Kato, and Hiroyoshi Suzuki. A bending pneumatic rubber actuator realizing soft-bodied manta swimming robot. In Proceedings 2007 IEEE international conference on robotics and automation, pages 4975–4980. IEEE, 2007. [99] Tung D Ta, Takuya Umedachi, and Yoshihiro Kawahara. Design of frictional 2d- anisotropy surface for wriggle locomotion of printable soft-bodied robots. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 6779– 6785. IEEE, 2018. [100] Toshio Takayama, Hirozumi Takeshima, Tomoyuki Hori, and Toru Omata. A twisted IEEE/ASME bundled tube locomotive device proposed for in-pipe mobile robot. 132 Transactions on Mechatronics, 20(6):2915–2923, 2015. [101] Toshio Takayama, Hirozumi Takeshima, Tomoyuki Hori, and Toru Omata. A twisted IEEE/ASME bundled tube locomotive device proposed for in-pipe mobile robot. Transactions on Mechatronics, 20(6):2915–2923, 2015. [102] Hirozumi Takeshima and Toshio Takayama. Geometric estimation of the deforma- tion and the design method for developing helical bundled-tube locomotive devices. IEEE/ASME Transactions on Mechatronics, 23(1):223–232, 2017. [103] Hirozumi Takeshima and Toshio Takayama. Development of a steerable in-pipe loco- motive device with six braided tubes. ROBOMECH Journal, 5(1):1–11, 2018. [104] Xiaobo Tan et al. Autonomous robotic fish as mobile sensor platforms: challenges and potential solutions. Marine Technology Society Journal, 45(4):31–40, 2011. [105] Thomas George Thuruthel, Egidio Falotico, Federico Renda, and Cecilia Laschi. Model-based reinforcement learning for closed-loop dynamic control of soft robotic manipulators. IEEE Transactions on Robotics, 35(1):124–134, 2018. [106] Catherine Tiner, Salil Bapat, Subrata Deb Nath, Sundar V Atre, and Ajay Malshe. Ex- ploring convergence of snake-skin-inspired texture designs and additive manufacturing for mechanical traction. Procedia Manufacturing, 34:640–646, 2019. [107] Halvor T Tramsen, Stanislav N Gorb, Hao Zhang, Poramate Manoonpong, Zhendong Dai, and Lars Heepe. Inversion of friction anisotropy in a bio-inspired asymmetrically structured surface. Journal of The Royal Society Interface, 15(138):20170629, 2018. [108] Halvor T Tramsen, Lars Heepe, Jettanan Homchanthanakul, Florentin Wörgötter, Stanislav N Gorb, and Poramate Manoonpong. Getting grip in changing environments: the effect of friction anisotropy inversion on robot locomotion. Applied Physics A, 127(5):389, 2021. [109] Deepak Trivedi, Amir Lotfi, and Christopher D Rahn. Geometrically exact models for soft robotic manipulators. IEEE Transactions on Robotics, 24(4):773–780, 2008. [110] James Walker, Thomas Zidek, Cory Harbel, Sanghyun Yoon, F Sterling Strickland, Srinivas Kumar, and Minchul Shin. Soft robotics: A review of recent developments of pneumatic soft actuators. In Actuators, volume 9, page 3. MDPI, 2020. [111] Zhenyu Wan, Yinan Sun, Yun Qin, Erik H Skorina, Renato Gasoto, Ming Luo, Jie Fu, and Cagdas D Onal. Design, analysis, and real-time simulation of a 3d soft robotic snake. Soft Robotics, 2022. [112] Chen Wang, Venkata Rithwick Puranam, Sarthak Misra, and Venkatasubrama- 133 nian Kalpathy Venkiteswaran. A snake-inspired multi-segmented magnetic soft robot towards medical applications. IEEE Robotics and automation letters, 7(2):5795–5802, 2022. [113] Chengjun Wang, Kyoseung Sim, Jin Chen, Hojin Kim, Zhoulyu Rao, Yuhang Li, Weiqiu Chen, Jizhou Song, Rafael Verduzco, and Cunjiang Yu. Soft ultrathin elec- tronics innervated adaptive fully soft robots. Advanced Materials, 30(13):1706695, 2018. [114] Xing Wang, Hanwen Kang, Hongyu Zhou, Wesley Au, Michael Yu Wang, and Chao Chen. Development and evaluation of a robust soft robotic gripper for apple harvesting. Computers and Electronics in Agriculture, 204:107552, 2023. [115] Yun Wang, Gang Wang, Weihan Ge, Jinxi Duan, Zixin Chen, and Li Wen. Perceived safety assessment of interactive motions in human–soft robot interaction. Biomimetics, 9(1):58, 2024. [116] Zhongkui Wang, Shinichi Hirai, and Sadao Kawamura. Challenges and opportunities in robotic food handling: A review. Frontiers in Robotics and AI, 8:433, 2022. [117] Robert J Webster III and Bryan A Jones. Design and kinematic modeling of con- stant curvature continuum robots: A review. The International Journal of Robotics Research, 29(13):1661–1683, 2010. [118] Li Wen, James C Weaver, and George V Lauder. Biomimetic shark skin: design, fab- rication and hydrodynamic function. Journal of Experimental Biology, 217(10):1656– 1666, 2014. [119] Tao Wen, Junfeng Hu, Jun Zhang, Xian Li, Sheng Kang, and Ning Zhang. Design, per- formance analysis, and experiments of a soft robot for rescue. Journal of Mechanisms and Robotics, 16(7), 2024. [120] Julian Whitman, Nico Zevallos, Matt Travers, and Howie Choset. Snake robot urban search after the 2017 mexico city earthquake. In 2018 IEEE international symposium on safety, security, and rescue robotics (SSRR), pages 1–6. IEEE, 2018. [121] Cornell Wright, Austin Buchan, Ben Brown, Jason Geist, Michael Schwerin, David Rollinson, Matthew Tesch, and Howie Choset. Design and architecture of the unified modular snake robot. In 2012 IEEE international conference on robotics and automa- tion, pages 4347–4354. IEEE, 2012. [122] Cornell Wright, Aaron Johnson, Aaron Peck, Zachary McCord, Allison Naaktgeboren, Philip Gianfortoni, Manuel Gonzalez-Rivero, Ross Hatton, and Howie Choset. Design of a modular snake robot. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 2609–2614. IEEE, 2007. 134 [123] Cornell Wright, Aaron Johnson, Aaron Peck, Zachary McCord, Allison Naaktgeboren, Philip Gianfortoni, Manuel Gonzalez-Rivero, Ross Hatton, and Howie Choset. Design of a modular snake robot. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 2609–2614. IEEE, 2007. [124] Mingxin Wu, Xingwen Zheng, Ruosi Liu, Ningzhe Hou, Waqar Hussain Afridi, Rah- dar Hussain Afridi, Xin Guo, Jianing Wu, Chen Wang, and Guangming Xie. Glowing sucker octopus (stauroteuthis syrtensis)-inspired soft robotic gripper for underwater self-adaptive grasping and sensing. Advanced Science, 9(17):2104382, 2022. [125] Weibin Wu, Shudong Yu, Paul Schreiber, Antje Dollmann, Christian Lutz, Guil- laume Gomard, Christian Greiner, and Hendrik Hölscher. Variation of the frictional anisotropy on ventral scales of snakes caused by nanoscale steps. Bioinspiration & biomimetics, 15(5):056014, 2020. [126] Yichuan Wu, Justin K Yim, Jiaming Liang, Zhichun Shao, Mingjing Qi, Junwen Zhong, Zihao Luo, Xiaojun Yan, Min Zhang, Xiaohao Wang, et al. Insect-scale fast moving and ultrarobust soft robot. Science robotics, 4(32):eaax1594, 2019. [127] Qing Xie, Tao Wang, and Shiqiang Zhu. Simplified dynamical model and experimen- tal verification of an underwater hydraulic soft robotic arm. Smart Materials and Structures, 31(7):075011, 2022. [128] Zhexin Xie, August G Domel, Ning An, Connor Green, Zheyuan Gong, Tianmiao Wang, Elias M Knubben, James C Weaver, Katia Bertoldi, and Li Wen. Octopus arm-inspired tapered soft actuators with suckers for improved grasping. Soft robotics, 7(5):639–648, 2020. [129] Zhiguang Xing, Junming Zhang, David McCoul, Yanzhen Cui, Lining Sun, and Jian- wen Zhao. A super-lightweight and soft manipulator driven by dielectric elastomers. Soft robotics, 7(4):512–520, 2020. [130] Fan Xu, Hesheng Wang, Kwok Wai Samuel Au, Weidong Chen, and Yanzi Miao. Underwater dynamic modeling for a cable-driven soft robot arm. IEEE/ASME trans- actions on Mechatronics, 23(6):2726–2738, 2018. [131] Hao Yang, Min Xu, Weihua Li, and Shiwu Zhang. Design and implementation of a soft robotic arm driven by sma coils. IEEE Transactions on Industrial Electronics, 66(8):6108–6116, 2018. [132] Osman Dogan Yirmibesoglu, John Morrow, Steph Walker, Walker Gosrich, Reece Cañizares, Hansung Kim, Uranbileg Daalkhaijav, Chloe Fleming, Callie Branyan, and Yigit Menguc. Direct 3d printing of silicone elastomer soft robots and their perfor- mance comparison with molded counterparts. In 2018 IEEE International Conference on Soft Robotics (RoboSoft), pages 295–302. IEEE, 2018. 135 [133] David Zarrouk, Moshe Mann, Nir Degani, Tal Yehuda, Nissan Jarbi, and Amotz Hess. Single actuator wave-like robot (saw): design, modeling, and experiments. Bioinspira- tion & biomimetics, 11(4):046004, 2016. [134] Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In 2020 IEEE symposium series on computational intelligence (SSCI), pages 737–744. IEEE, 2020. [135] Xiaodong Zhou and Shusheng Bi. A survey of bio-inspired compliant legged robot designs. Bioinspiration & biomimetics, 7(4):041001, 2012. [136] Lu Zongxing, Li Wanxin, and Zhang Liping. Research development of soft manipulator: A review. Advances in Mechanical Engineering, 12(8):1687814020950094, 2020. 136