ENABLING REAL-TIME COMMUNICATION FOR HUMAN AUGMENTATION SYSTEMS VIA UNOBTRUSIVE HIGH BANDWIDTH MACHINE TO HUMAN ELECTROTACTILE PERIPHERAL NERVE STIMULATION By Sina Parsnejad A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of Electrical Engineering – Doctor of Philosophy 2022 ABSTRACT ENABLING REAL-TIME COMMUNICATION FOR HUMAN AUGMENTATION SYSTEMS VIA UNOBTRUSIVE HIGH BANDWIDTH MACHINE TO HUMAN ELECTROTACTILE PERIPHERAL NERVE STIMULATION By Sina Parsnejad The advent of sensor technologies and the resulting abundance of information together with modern advanced processing capabilities makes improving human lives via human augmentation technologies ever more appealing. To establish a new effective form of human- machine-communication (M2HC) for augmentation, this dissertation explores non-invasive peripheral nerve stimulation via electrotactile waveforms. This dissertation conducts extensive convergence research between the fields of psychology, electrical engineering, neuroscience and human augmentation and established innovations to create distinct sensations that can be utilized as iconic electrotactile M2HC. Existing electrotactile stimulation models deliver a limited range of distinct sensations, making iconic communication challenging. To address this issue, we created a software/hardware infrastructure, including novel electrotactile electrode arrays and improved stimulation circuitry, that allows for rapid prototyping and testing various electrotactile innovations. We created a model for electrotactile waveform generation (MEWS) wherein a train of high-frequency electrotactile pulses is shaped into electrotactile waveforms through a multi- layer on-off-keying modulation forgoing the need for constant frequency recalibration and making painful sensations less likely to happen. Using MEWS, we conducted multiple human trials on 15 volunteering participants stimulating a total of ~6000 electrotactile sensations which led us to create 13 distinct electrotactile waveform with an accuracy of 85.4%. To increase the number of messages that can be delivered by electrotactile stimulation, a model for creating varying electrotactile waveforms (MOVES) was created based on linguistic concept of phonemes and taking a semi-heuristic approach to creating electrotactile waveforms. Using MOVES we conducted multiple human trials on 21 volunteering participants stimulating a total of ~5000 electrotactile sensations. Our human trials proved that MOVES was able to create 24 distinct sensations with an accuracy of 89% that can be used to convey messages through iconic communication and has the potential to expand further beyond the 24 messages. The number of messages delivered by MOVES pentuples the best recorded number of distinct electrotactile sensations in literature. ACKNOWLEDGMENTS Firstly, I would like to express my deepest gratitude to my advisor, Dr. Andrew Mason, for his continued guidance and support through my PhD study and dissertation work. He is very supportive of his students and is not afraid of thinking outside the box, coming up with new solutions and helping his students to the best of his abilities to achieve their vision. I will take all the lessons I’ve learned from him throughout the rest of my life, something I hope I can pass on to the next generation. Secondly, I want to thank Dr. Galit Pelled, Dr. Jan Brascamp and Dr. Subir Biswas for serving on my committee. They give me encouragement and support my dissertation work all the time. Thirdly, I want to thank my colleagues in HATLaB research group for their support. Especial thanks to Ehsan Ashoori, Sylmarie Davila-Montero, and Yousef Gtat who were instrumental to my progress through this project, as well as Derek Goderis and Anna Inohara. Lastly, I want to thank my family and my precious nieces Anil, Lara, Sevilay, and Shaylin. I also want to thank my amazing girlfriend Chauntel Wooten and my found family away from home Celine and Douglas Bridges, Hossein Kouhani, Cataoun Kargar, Zach Hoyle, Pejman Masrouri, Sharon Prince, and my beloved Morpheus for keeping me sane through my zany grad school adventures. iv TABLE OF CONTENTS LIST OF TABLES ............................................................................................................................ vii LIST OF FIGURES ......................................................................................................................... viii Chapter 1: Augmentation of human perception ........................................................................... 1 Chapter 2: Machine to human communication and electrotactile theory and review .................. 6 2.1 Augmented human perception ............................................................................................ 6 2.2 Machine to human communication (M2HC) approaches and limitations, theory and literature review ........................................................................................................................ 8 2.3 Tactile stimulation.............................................................................................................. 12 2.4 Principals of electrotactile stimulation ............................................................................... 15 2.4.1 Existing electrotactile systems ..................................................................................... 15 2.4.2 Electrophysiology of the skin........................................................................................ 17 2.4.3 Electrotactile stimulation ............................................................................................. 24 2.5 Summary and discussion .................................................................................................... 29 2.5.1 Summary ...................................................................................................................... 29 2.5.2 Discussion..................................................................................................................... 30 Chapter 3: Generating complex icons via electrotactile stimulation ........................................... 32 3.1 Complex electrotactile signals and iconic communication .............................................. 32 3.2 Model for conveying complex etacons using qualitative elements .................................... 34 3.3 Hardware and software platform for generation of electrotactile stimulation .................. 39 3.4 Eliciting distinct sensations on a single electrode using MES ............................................. 44 3.4.1 Experiment I: exploring the ability of humans to differentiate high-frequency pulses. 44 3.4.2 Experiment II: exploring the ability of humans to differentiate Packs .......................... 47 3.4.3 Experiment III: Discriminating high-frequency pulse trains .......................................... 50 3.5 Summary and discussion of work ....................................................................................... 53 3.5.1 Summary ...................................................................................................................... 53 3.5.2 Discussion and open challenges ................................................................................... 55 Chapter 4: MEWS, A model for creating a library of static electrotactile waveforms ................. 58 4.1 MEWS and Experimental Setup.......................................................................................... 59 4.2 Study set up ....................................................................................................................... 63 4.2.1 Setting MEWS’ parameter boundaries ......................................................................... 63 4.2.2 Electrotactile stimulation hardware ............................................................................. 63 4.2.3 A multi-electrode approach to creating electrotactile sensations................................ 65 4.2.3 Electrotactile stimulation software .............................................................................. 70 4.2.4 Participant recruitment and experiments’ procedure .................................................. 70 4.3 Experiments I - III: Investigating BF, TF, and TDC................................................................ 71 v 4.3.1 Experiment I: Understanding BF ................................................................................... 71 4.3.2 Experiment II: Understanding TF .................................................................................. 73 4.3.3 Experiment III: TDC ....................................................................................................... 73 4.4 Analysis techniques for MEWS experiments ...................................................................... 74 4.4.1 Experiment I-III results ................................................................................................. 77 4.4.2 Experiment I-III analysis................................................................................................ 81 4.5 Experiment IV: Creating distinct waveforms ...................................................................... 82 4.5.1 Experiment IV description ............................................................................................ 82 4.5.2 Experiment results and analysis ................................................................................... 84 4.6 Summary and discussion of work ....................................................................................... 87 4.6.1 Summary ...................................................................................................................... 87 4.6.2 Discussion and open challenges ................................................................................... 89 Chapter 5: MOVES, A Linguistically-inspired Electrotactile Icon Generation ............................... 90 5.1 Generation of distinct electrotactile sensations using etactemes ...................................... 91 5.2 Understanding Etactemes .................................................................................................. 93 5.2.1 Experiment I setup ....................................................................................................... 93 5.2.1 Experiment I results ..................................................................................................... 96 5.3 MOVES, a model for combining etactemes into variable distinct waveforms .................... 98 5.3.1 Use of architypes for simplifying the combinations of etactemes in MOVES ............. 100 5.3.2 Breaking down architypes .......................................................................................... 101 5.4 Creating a library of distinct electrotactile waveforms using MOVES .............................. 103 5.5 Summary and discussion of work ..................................................................................... 109 5.5.1 Summary .................................................................................................................... 109 5.5.2 Discussion and open challenges ................................................................................. 110 Chapter 6: Conclusions and future work ................................................................................... 112 6.1 Summary .......................................................................................................................... 112 6.2 Contributions ................................................................................................................... 113 6.3 Future work...................................................................................................................... 116 REFERENCES .............................................................................................................................. 118 vi LIST OF TABLES Table 2.1: A comparison of available tactile stimulation methods in terms of their compatibility for an M2HC system .................................................................................................................... 13 Table 3.1: list of components used for realization of electrotactile current generator .............. 40 Table 4.1: The MEWS parameters and their range of operation ................................................ 63 Table 4.2: The four corners used in BF trials defined by their corresponding TF and BDC values .................................................................................................................................................... 71 Table 4.3: The four corners used in TF trials defined by their corresponding TF and BDC values .................................................................................................................................................... 73 Table 4.4: Recorded accuracy values for NDW values 2,3 and 4 over three corners denoted by TF values .......................................................................................................................................... 81 Table 4.5: the percentages of responses by participants given an electrotactile prompt with same value, increasing TF, and increasing TDC. The majority of participants correctly associated an increase in TF and TDC with an increase in frequency and intensity respectively ....................... 81 Table 4.6: List of 23 waveforms created based on insights from Experiments I-III with their respective BF, TF, and TDC values ............................................................................................... 82 Table 4.7: Extracted figure of merit parameters from each NDW and their FoM acquired through a k-means clustering algorithm ................................................................................................... 84 Table 5.1: The metrics of success for each NDW and their corresponding additional metrics used for extracting FoM .................................................................................................................... 106 Table 5.2: The performance of the overall results captured from each round of experiment that was conducted on three participants and the standard deviation (STD) measuring the consistency of their responses ..................................................................................................................... 107 Table 5.3: The progression of breaking down our architypes from the Initial Waveform Set to the Reduced Waveform Set (through Experiment II) which were created by reducing the number of architypes and the final set of waveforms created with NDW= 24 (through Experiment III) .... 109 vii LIST OF FIGURES Figure 2.1: The anatomy of an AHP system wherein the human and the machine observe the outside world and the human. Even through the two observers see the same world, their prospective may differ. The machine sharing its unique prospective with human through an M2HC constitutes an AHP. ............................................................................................................ 7 Figure 2.2: The pathway of information from an information input element such as sight, hearing, or touch to the brain. The brain observes and stores the input in its short-term memory for processing. Once processed, the information deemed worthy will be stored in long-term memory......................................................................................................................................... 8 Figure 2.3: Different layers of human skin. The thickness of each layer is not up to scale. ........ 18 Figure 2.4: LTMR nerve axon with Myelin sheet and nodes of Ranvier. An action potential travels through Myelinated portions of the axon pathway using saltatory conduction and is amplified in nodes of Ranvier through depolarization of ionic channels. ....................................................... 21 Figure 2.5: The structure of a typical hairy skin and the placement of tactile receptors, sweat glands and hair follicles. .............................................................................................................. 22 Figure 2.6: A model for penetration of electrotactile current and potential created inside the skin. .................................................................................................................................................... 24 Figure 2.7: The electrical model for current penetration into the skin and their interaction with nodes of Ranvier of an LTMR axons. ........................................................................................... 25 Figure 2.8: The most common electrotactile circuitry in literature: (a) current mirror structure (b) improved Howland current source. ............................................................................................ 26 Figure 3.1: Electrotactile dimensions that can be used to create distinct sensations and may be used for creation of complex waveforms .................................................................................... 34 Figure 3.2: MES models for generation of complex electrotactile waveforms............................ 36 Figure 3.3: Prominent parameters of pulse-stimulation-level of MES and their respective range. .................................................................................................................................................... 37 Figure 3.4: Prominent parameters of package-level MES and their respective range. ............... 37 Figure 3.5: Prominent parameters of bundle-level MES and their respective range. ................. 38 Figure 3.6: The schematics of modified current mirror electrotactile current generator used for this chapter. Qx3 and Qx4 are cascode transistors to ensure electrotactile current is independent of skin voltage. ............................................................................................................................ 40 viii Figure 3.7: The hardware/software infrastructure used for conducting human trials. ............... 41 Figure 3.8: Electrode array used for this chapter. The electrodes are medical-grade stainless steel balls placed on a standard PCB substrate. .................................................................................. 44 Figure 3.9: Completer hardware implementation. ..................................................................... 44 Figure 3.10: Example waveforms for 3 of the 20 SRR (pulse repetition rate) frequencies used in Experiment I. Amplitude is normalized to ±1. ............................................................................. 46 Figure 3.11: Average comparative discrimination response of participants and the variation across all participants SRR frequencies spanning from 100Hz to 2.1kHz. ................................... 47 Figure 3.12: Example waveforms for 4 of the 20 PRR (bundle repetition rate) frequencies used in Experiment I. Amplitude is normalized to ±1. ............................................................................. 48 Figure 3.13: Average comparative distinction response of participant #3 as a function of BRR (bundle repetition rate). The extracted comparison distinction baselines at shown at the top. 49 Figure 3.14: Average comparative discrimination response of participants and the variation across all participants for BRR frequencies spanning from 10Hz to 100Hz in Experiment I. ....... 50 Figure 3.15: Example waveforms for different NS (number of Pulses per Bundle) used in Experiment II. .............................................................................................................................. 51 Figure 3.16: Comparative distinction reaction of participant #3 to the increasing NS (number of Pulses per Bundle) and the extracted comparison distinction baselines. ................................... 52 Figure 3.17: Baselines for NS (number of Pulses per Bundle) for all participants and the average value of their response. .............................................................................................................. 53 Figure 4.1: A normalized representation of MEWS modulation technique in which a carrier high- frequency bi-phasic pulse train is modulated by two OOK style masks named Beat frequency and Texture frequency, resulting in a finalized electrotactile waveform. .......................................... 61 Figure 4.2: Several examples of electrotactile waveforms created using MEWS. (a) depicts a case with a low BF and high TF, effectively resulting in each beat to contain 13 textures. (b) depicts a case where beat and texture are very close together with each beat containing only two texture elements. (c) and (d) depict the same BF and TF values with the only difference being that the latter has a lower TDC which results in a lower number of electrotactile pulses being present in each texture. ............................................................................................................................... 62 Figure 4.3: The electrotactile stimulation hardware for electrotactile stimulation circuitry....... 64 ix Figure 4.4: The interaction between a large electrotactile electrode and skin pores that are susceptible for breaking down. A broken-down pore has a lower resistance and would redirect the majority of electrotactile current activating HTMR pain receptors. ..................................... 65 Figure 4.5: The distributed electrotactile electrode model. The goal is to prevent a broken-down pore from conducting the majority of electrotactile current to prevent painful sensations. ...... 67 Figure 4.6: Fabricated distributed electrode array where each distributed electrode is composed of 6 individual electrodes (1.0mm G10 440C Stainless Steel Balls) in a hexagonal pattern that are surrounded by 12 ground electrodes. The stainless-steel balls were adhered and connected to the PCB board using silver-epoxy paste. ..................................................................................... 68 Figure 4.7: A simplified presentation of modified electrotactile current waveform generator for enabling forced distribution of electrotactile currents unto distributed electrodes. Each output is connected to an individual electrode through an isolation capacitance. .................................... 68 Figure 4.8: Hardware for distributed electrotactile hardware that facilitates distributed electrotactile stimulation. ........................................................................................................... 69 Figure 4.9: Average of comparison chart for all participants for (a) BF trials corner 3 and (b) TF trials corner 1. Participant responses “different” and “same” are marked with a 1 and 0, respectively. ................................................................................................................................ 75 Figure 4.10: The results of TF trials corner 1 reduced to the best performing waveforms at NDW of 7. ............................................................................................................................................. 77 Figure 4.11: progression of (a) BF steps (for corner 3) at increasing NDWs and (b) TF steps for an NDW that produces an accuracy closet to 80% over multiple corners. As indicated by the fitting curve, (a) shows that at low NDWs BF values increase in an exponential rate. The results shown in (b) indicate that all corners of TF trials also display an exponential growth. .......................... 78 Figure 4.12: The progression of RMSe, minimum element, and accuracy of best waveform sets for TF frequency, extracted with an increase in NDW. ................................................................ 79 Figure 4.13: Accuracy observed in association with an increase in NDW for all corners in (a) BF trials and (b) TF trials. In case of BF trials, corners 3 and 2 show the best and worst performance, respectively. In TF trials, overall accuracy drops with an increase in BF, indicating that lower BF values will result in higher distinction among texture frequencies. ............................................ 80 Figure 4.14: The average overall raw results from Experiment IV. ............................................. 83 Figure 4.15: The best performing waveforms (denoted by their number as described in Table 6 at NDW of 13 and their comparison to one another. .................................................................. 86 x Figure 4.16: A breakdown of the clusters that are formed for NDW of 12 and the corresponding BF and TF values that create each cluster. The X mark denotes the BF/TF pair that was selected to represent that luster. .............................................................................................................. 87 Figure 5.1: Electrotactile waveform creation model (MEWS) used in Chapter 5. This model is not limited to a 500ms timeframe and only uses texture frequency as an OOK mask. ..................... 92 Figure 5.2: The sensation classes of tapping and high-frequency vibrations with various intensity levels. (a) depicts a low intensity (2ms long) tapping sensation. (b) depicts a low intensity high frequency vibration with a frequency of 100Hz. (c) and (d) depict high frequency counterparts of tapping and vibration sensations. ............................................................................................... 92 Figure 5.3: Visualization of electrotactile waveforms used to understand tapping sensations. (a) depicts the verses used to understand the minimum viable electrotactile burst length with verse α having a variable electrotactile burst length and verse β without any burst. (b) depicts the verses used for understanding the minimum viable distance between two bursts with verse α having two electrotactile bursts with a variable distance between them while verse β only having a singular electrotactile pulse burst. ........................................................................................... 95 Figure 5.4: The verses used for test 3 wherein verse α and β start with an initial section with vibration frequencies of 200Hz and 32Hz respectively. The length of the initial section was randomly changed in the range of 20ms to 300ms to extract the minimum length under which α and β are distinguishable. ........................................................................................................... 96 Figure 5.5: The results of Tests 1-4. (a) depicts the percentage of comparisons that correctly identified between the lack of an initial burst and an initial bursts width as shown on the x Axis. (b) depicts the distinguishability of two electrotactile pulse bursts as the gap between the two bursts grows. (c) depicts the length at which an average person should be stimulated with a vibration frequency for them to be able to distinguish between vibration frequencies of 32HZ and 200Hz. (d) depicts the minimum required gap length in a vibration frequency for the gap to be noticeable. .................................................................................................................................. 97 Figure 5.6: The electrotactile waveform created by the etacteme descriptor “2xI, 3xS1” which means that the is created with two back-to-back tapping sensations, followed by a 360ms 32Hz vibration. The electrotactile pulses are a 2kHz bi-phasic pulse train with a standard amplitude of Amp. As a result, due to the large timescale present in this figure, they are seen a single golden column. ....................................................................................................................................... 99 Figure 5.7: Our process of breaking down an architype. (a) depicts the raw comparison chart for Architype 1. (b) depicts the broken-down architype 1 wherein elements were selected based on their Min and RMSe performance metrics. ............................................................................... 103 Figure 5.8: The raw comparison results for Experiment III between all 31 waveforms, performed over 7 complete comparison sets by 21 participants. ............................................................... 105 xi Figure 5.9: The connectivity map for the a) the 31 waveforms fed to Experiment III and b) the resulting waveforms for NDW= 24. ........................................................................................... 108 xii Chapter 1: Augmentation of human perception A commendable nature of human beings is their desire to surpass their limitation and stive to achieve more. Humans have always sought to enhance their understanding of the world around them through augmentation technologies. Human augmentation (HA) is a family of technologies that aims to enhance how a person views, understands, and interacts with the world around them. When thinking of human augmentation, most people believe it to be a futuristic concept wherein humans have been meticulously integrated with machines. However, the reality is different from such commonly hold beliefs. Ever since the advent of industrial revolution, humans have been integrated with machines through various means. For instance, human memory has been augmented by use of phone books, so they do not have to memorize every phone number necessary, the movement capabilities of humans have been greatly increased using vehicles, planes, electric scooters, etc., and most recently humanity’s social interactions have been forever altered with the use of social media. Humanity, for good reason, has grown to be reliant on HA tools. The reason is that HA increase quality of life by augmenting various aspects of humanity such as their cognitive and physical abilities. Furthermore, it is possible to help people with disabilities overcome their limitations. Some examples are insulin regulation systems, exo-skeletons, object recognition tools, etc. Despite all the recent advancements in HA technologies, there are bottlenecks to what human augmentation technologies can do. A more commonly known example is for the case of prosthetic organs and limbs where the prosthetics can’t be too heavy because they will put pressure on human skeleton which is resolvable using novel material. However, there are 1 fundamental bottlenecks in the field of human augmentation that are preventing great progress in the field. One prominent case is the communication path from machines to human. Traditionally augmentation agents often communicate with their corresponding human through visual and auditory cues. A prime example of this is the way the most prevalent augmentation agent up to day, the modern smartphone, can only communicate with its user through messages displayed on a screen. There is a fundamental limit of how much information can be conveyed to a person through visual cues. For instance, the average reading speed for a typical human is 3-4 words per second (~300ms per word) which is way lower than the processing capabilities of the corresponding agent (smart phone), as opposed inter-device communication speed of hundreds of megabytes per second. As a result, there is a need for high-speed machine-to-human communication (M2HC). Furthermore, too many visual cues are a burden to human attention span and usually lead to issues such as split attention span, anxiety, and unproductivity. There are multiple approaches to addressing the fundamental issue of M2HC. The field of human computer interaction (HCI) focuses on optimizing the interactions between humans and a visual M2HC using various programming and psychological tactics. There is a major frontier for facilitating brain implants for observing and communicating with the brain. However, the HCI technology is limited to the fundamentals of visual prowess of human brain and brain implant technologies are still in their infancy stages and require major surgical procedures that hinders their progress. There are alternative for effective M2HC that have the potential to be high-speed and not be over-bearing on the human attention span. Tactile sensory is an alternative that utilize the skin, the largest sensory organ in human body, for establishing an M2HC. Tactile communication is an established method of communication with humans that has been utilized 2 in numerous technologies such as prosthetic feedback, smartphone notifications, etc. There are multiple iterations of tactile communication but they can be classified as vibrotactile, electro vibration, and electrotactile. However, tactile communication has not been extensively studied and implemented as an alternative for M2HC because its ability to relay complex have not been properly studied. This dissertation addresses the limitations of tactile communication for M2HC by developing electrotactile M2HC as an alternative. Electrotactile stimulation was selected because it has a significantly higher flexibility for creation of new sensations and it has an unobtrusive nature. The dissertation will discuss the concept and fundamentals of electrotactile communication and will explore solutions to the major challenges that currently impede the use of electrotactile for M2HC. These challenges include: • A cohesive description of physiological and perceptual aspects of electrotactile stimulation from an engineering point of view is scant. • The models for creating electrotactile messages are severely limited in the number of messages they can deliver for M2HC. • The existing models for creating electrotactile messages have not been subject to rigorous double-blinded human trials • Electrotactile stimulation comes with the possibility of inducing painful sensations, especially when performing intensity modulation, limiting the widespread use and acceptance of electrotactile stimulation Even though much research effort has been put in electrotactile stimulation and how it interacts with neurons present in the skin, the information about skin tactile electrophysiology, 3 especially hairy skin is scattered across multiple disciplines and not easily accessible for electrotactile engineers. As a result, the first priority in this thesis would be to compile all an extensive review of skin topology, operation principals of electrotactile stimulations, and best strategies for adopting electrotactile circuitry and electrodes and their safety concerns Despite the fact that electrotactile stimulation is a form of neural stimulation, the range of electrotactile messages delivered as a M2HC are severely limited. This can be attributed to the fact that electrotactile stimulation is primarily being used for prosthetic feedback and haptic interactions and not as a means for messaging. The highest reported number of messages being transferred as M2HC by a single electrode are less than 5 distinct messages which is a stark contrast to the range of sensations the average human receives through tactile senses. Since we wish to use electrotactile as a messaging tool, we will create a model for creating a significantly higher number of messages with M2HC as a target application. The human trials in literature that are conducted for understanding the role of electrotactile messaging are not rigorous. They usually suffer from one or multiple of the following issues: They subject population is usually less than 10 which is only sufficient for exploratory studies. The experiments are not double blinded. The studies are subject-dependent and the data acquired are not comparable between subjects. As such, throughout this body of work, we will undergo a massive undertaking of creating a software/hardware platform for conducting human trials on large subject populations and create a method for reliably capturing and comparing data from multiple subjects in a double-blinded method. Electrotactile literature relies on modulating amplitude of electrotactile signals to create and modify intensity. Since the signal amplitudes that can be applied are subjective to each 4 person and may change with many environmental conditions, it is best if creating intensity by changing electrotactile signal levels was avoided. However, signal intensity can be an important signifier for creating tactile sensations and should not be ignored. As such, a method of electrotactile stimulating that coo-opts intensity should be created that does not rely on modulating electrotactile signal amplitude. In Chapter 2 we will delve into principals of M2HC and HA and justify the need for an electrotactile solution. We will then review relevant literature on electrophysiology of skin and operating principals of electrotactile stimulation and explain how the two concepts intertwine. In Chapter 3, we will discuss our process of internalizing electrotactile literature insights into a model for electrotactile stimulation and explore various aspects of electrotactile stimulation via multiple human trials. In Chapter 4, we will finalize our hardware for electrotactile stimulation and create an electrotactile stimulation model for producing electrotactile icons. We will create a software/hardware infrastructure to test our model and prove the functionality of our setup through rigorous human trials. In Chapter 5 we will further our model for electrotactile stimulation by utilizing varying electrotactile signals to create a large library of distinct electrotactile sensations that eclipses electrotactile messaging literature in the number of unique messages it can create that can be used for electrotactile iconic communication through conducting large human trials. 5 Chapter 2: Machine to human communication and electrotactile theory and review 2.1 Augmented human perception Based on definitions described in [1], human augmentation (HA) is an interdisciplinary field that studies methods, technologies and their application for enhancing sensing, actions and cognitive abilities of humans. This is achieved through sensation and actuation technologies driven by signal processing techniques. The goal of HA is to improve quality of human lives through augmentation of senses, cognition, and actuation. Augmentation of human senses is done through collecting and interpreting multisensory information from the world and presenting it to the user in a comprehendible method such as vision substitution technologies or vision enhancement technologies such as infra-red goggles. Augmentation of cognition can assist the user in cognitive tasks through a tight coupling of user and an information processing system such as brain rehabilitation technologies or even memory enhancement technologies that can be as simple as an address book on a smartphone. Augmentation of actuation is done through improving human action, for example prosthetics or exoskeletons. The combination of augmented sensing and actuation can be studied under the general term of augmented human perception (AHP). AHP refers to a field of HA wherein the purpose is to collect information from environment, compile and process that information, and present them to the human in manner that augments their perception of the world. 6 In an AHP system, the augmenting agent works in parallel to the human, observing what the human cannot perceive. The objects of perception can be the outside world or the human themselves. The outside world can include any physical phenomenon like the temperature of the room, the pollution level of air, the important messages that need to be reviewed, etc. The observation of self can be the mental state of the user, their physiological signals, social encounters, etc. Figure 2.1 depicts a representation of such concepts. The user communicates to the augmentation agent through control signals. Such control signals may be as simple as commands on a console or as complicated as hand gestures or a direct brain machine interface. The M2HC is any method that the augmenting agents can use to communicate with their human counterparts. The most common M2HC method is conveying messages through visual cues on a display such as a monitor, a smartphone screen, etc. However, M2HC is not only limited by vision. M2HC may leverage any of the primary senses such as visual, auditory, tactile, etc. as well as invasive methods described by BMI. In concept, AHP may sound similar to available technologies such as Augmented Reality (AR). However, AR predominantly focus on communication through Figure 2.1: The anatomy of an AHP system wherein the human and the machine observe the outside world and the human. Even through the two observers see the same world, their prospective may differ. The machine sharing its unique prospective with human through an M2HC constitutes an AHP. 7 vision whereas AHP may leverage any communication method. In computer science community, this level of augmentation is called Multimutated Reality (MR) [2]–[4]. M2HC must satisfy certain criteria as described below. The M2HC should primarily be capable of communicating with human effectively at a high data rate so as not to limit the use of AHP. On the receiving end, the human brain needs to be able to efficiently process AHP information and transfer that information to long-term memory. Furthermore, M2HC needs to be unobtrusive for daily human lives. M2HC should not deny information to the user by obtruding human senses or cognitive capabilities. For example, an effective M2HC should not solely use a smartphone screen since the screen will prevent the user from paying attention to their surroundings and will make them prone to shorter attention span [5]. 2.2 Machine to human communication (M2HC) approaches and limitations, theory and literature review All incoming information to the brain must be processed by human attention span. Attention is defined as the cognitive process of allocating brain processing resources to perform Figure 2.2: The pathway of information from an information input element such as sight, hearing, or touch to the brain. The brain observes and stores the input in its short-term memory for processing. Once processed, the information deemed worthy will be stored in long-term memory. 8 a task by selectively concentrating on one aspect of environment and ignoring other things [6]. Cognitive theories indicate that the process of attention when it comes to perceptual stimulation is as shown in Figure 2.2 [7]. Human attention can be divided between multiple tasks. However, there is a limitation to splitting of human attention due to limitations in human working memory [8]. A simple theory of human attention is introduced by Kahneman 1973 [9], in which it states that there is a single pool of attention that can be divided between multiple tasks. With the advent of information age and saturation of information using web and smartphone, human attention span is suspected to have decreased [5]. There are studies that suggest this limitation leads to a decrease in human quality of life and produce anxiety and other complications [10], [11]. Hence, the perceptual and cognitive load of an M2HC are important parameters that need to be considered. The human brain has internal options for addressing the attention span limitations and multi-task resource management called resource theory [12]. This theory indicates that while multi-tasking, the brain divides each complex task into smaller pieces that it can automate, saving attention span resources. However, as evident by literature, the brain can be further assisted with reduction of its attention span load. A literature survey conducted for this dissertation point out to three possible solutions: 1. It has been proven that it is easier to ignore non-task related perceptual cognitive load and thereby increasing attention by increasing task-related perceptual stimuli [8]. A simple example of this proposition is the conventional use of tactile vibrators in console video game controllers 9 2. It has been proposed that attention span load in a multi-tasking environment is decreased when stimulus-response channels for the multi-tasks are separated [13]. Stimulus- response channel refers to the input and output pair of our senses used for HCI. For example, the stimulus-response channel for a task of typing would be sight input and tactile output. An example for this theory would be multi-tasking of typing an academic paper while listening to music. Even-though two tasks are being simultaneously conducted, because the stimulus-response channels are different (sight, sound, and tactile) multitasking has less cognitive load on attention span 3. It is possible to reduce cognitive load by pre-processing information and down sampling information and having it presented to the proper human sensory. For option 3, the work presented in [14] presents the framework for reducing cognitive load. Suppose a machine communicates with a human by flashing 10 consecutive red or blue colored lights. The user reads this input in roughly 10s using their vision. According to the classical definition of data rate: 𝑑𝑎𝑡𝑎 𝑟𝑎𝑡𝑒 (𝑏𝑖𝑡𝑠⁄𝑠) = (𝐷𝑒𝑐𝑖𝑠𝑖𝑜𝑛𝑠 𝑚𝑎𝑑𝑒⁄𝑠) × 𝑙𝑜𝑔2 (𝑡𝑜𝑡𝑎𝑙 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑜𝑝𝑡𝑖𝑜𝑛𝑠) (2.1) where the options are red and blue and a decision is the person selecting between the options red and blue. In this case, the user’s attention had a data output rate of 1 bit/s. However, the human visual output is far greater than 1 (bits/s). A study published by University of Pennsylvania School of Medicine suggests that the data output rate of human eye can reach 10 Mb/s [15]. In fact, the human visual attention works in a two-stage process. In step 1, attention is uniformly distributed over the external visual scene and all information is processed in parallel. In step 2, 10 attention is concentrated to a specific visual scene and all data is processed in serial [6]. As such, 10Mb/s is down sampled to 1b/s. It can be concluded that vision uses a lot of attention span for this simple task and is not a proper input method for this M2HC. Due to their ease of implementation, visual and auditory communication are the predominantly used methods of M2HC. The observations above indicate that attention span can be conserved by having multiple stimulus-response channels. As a result, audio and visual cues are not preferred for this scope. There is preliminary research being conducted on real-time communication with the brain using invasive electrical stimulation implants [16], [17], optogenetical stimulation [17]–[19], and non-invasive methods such as transcranial magnetic stimulation or focused ultrasound stimulation [20]–[22]. However, invasive technologies are still under development and have yet to respond to the fundamental issue of brain accessibility and invasive surgical processes. Valuable research, especially in the area of prosthetic M2HC has been conducted in direct peripheral nerve stimulation using electrodes that interface with peripheral nerves [23]–[25]. However, the greatest drawbacks of this method is that it is still invasive and there is no fine control over the individual nerves being stimulated and induction of a sense of paresthesia [25]. Paresthesia is sense of tickling and prickling which likely arises from unnatural afferent nerve responses due to over-excitation or cross-talk between afferent pathways [25]. The senses of smell and taste are not ideal candidates for this task due to their sensor input circumstances. Among the remaining non-invasive pathways to the brain, tactile stimulation emerges as an outstanding candidate for an alternative route for M2HC. There is a massive network of tactile receptors over human body that can be leveraged for an effective M2HC with a high density of neural pathways. 11 2.3 Tactile stimulation Tactile stimulation of information is an alternative for providing M2HC and assisting the brain with managing attention. However, it should be noted that the tactile receptor density is not uniform across human body based on the human homunculi map [26]. Furthermore, the total number of tactile neural pathways to the human brain does not match the density of retina. However, the case can be made that levying a massive neural pathway for M2HC will introduce the same perceptual cognitive load issue that vision creates. As a result, it is beneficial to perform external preprocessing of data and providing M2HC using a very small set of tactile channels for the brain that does not impose a perceptual cognitive load on the brain. As a result, a local tactile communication over a limited surface area would be preferred for M2HC. Tactile communication operates by accessing the primary somatosensory context of brain through afferent peripheral nervous system. This can be done invasively using nerve cuff electrodes [23]–[25] or non-invasively by leveraging skin tactile receptors and their immediate afferent nerves. There are three predominant methods of non-invasive tactile interfacing in literature, vibrotactile, electrotactile, and electrovibration. Vibrotactile generally refers to the process of activating tactile receptors through physical deformation of skin [13], [14], [27]–[37] This process is usually done through a vibrating or pressure apparatus such as rotary or linear motors. Electrotactile stimulation operates by injecting a charge through electrotactile electrodes into the human dermis, causing the afferent unmyelinated neurons to activate, mimicking a tactile sensation [14], [38]–[74]. More details on the physiology of this process is discussed in next chapter. Electrovibration operates by inducing an electrical field to into the dermis layer, creating a capacitive interface with the ions in the epidermis layer of the skin [75]. 12 If enough electrical field is available, through capacitive force the epidermis layer of skin will create a pressure on the vibration or pressure receptors of skin, causing them to activate. The difference between vibrotactile and electrovibration is that, in the latter, the tactile input is not induced by mechanical deformation of skin. The comparison table shown below demonstrates key advantages and disadvantages of each technology: Table 2.1: A comparison of available tactile stimulation methods in terms of their compatibility for an M2HC system. Electrotactile Vibrotactile - Rotary Electrovibration Size Any size, the smaller the Typically, a cylinder 14 Any size better mm long dia. 6 mm Weight Less than 5g Typical motor 5 – 10g Less than 5g Frequency Up to 200Hz Typically, up to 160 Hz 5–1000 Hz Onset Time Negligible/Immediate 50–80ms windup time Negligible/Immediate Max stimulation duration No limit No limit Limited Power ~±100V 0.05 – 0.2 W Negligible Sensory Specificity All tactile receptors, Mechanoreceptors Mechanoreceptors predominantly vibration sensitive to vibration Punctateness Very High Medium High Audible signature None High, housing dependent None Mounting Direct skin contact No special requirements Dry skin, movement However, if observed in the context of consistent M2HC, the distinctions between these technologies become clearer. Electrovibration has the unique condition of being a temporary sensation when a dry skin is moved across an electrical field emitting device. The reader may have experienced this sensation while sweeping their fingertips across a wall with live wires on the other side. Such conditions make electrovibration not suitable for consist M2HC. 13 Electrotactile and vibrotactile usually have similar size, weight and power consumption. However, electrotactile is less obtrusive to the user because it does not rely on any mechanical apparatus. As a result, electrotactile has a lower audible signature compared to most common vibrotactile techniques and is therefore a more discreet form of communication. In terms of elicited sensations, electrotactile has a high punctateness and can activate axons connected to all six known tactile receptors where the six receptors are free nerve endings, root hair plexus, tactile discs, tactile corpuscles, lamellated corpuscles, and Ruffini corpuscles [76]. Punctateness describes the area of focus under each electrode [69], hence, higher punctateness means a more focused sensation which in return lets electrotactile target very specific regions of skin with localized sensations allowing for more consistent sensations and higher density of electrotactile electrodes. As a result, electrotactile stimulation can create a sensation of vibration and pressure with high spatial acuity. Furthermore, electrotactile has no onset delay, which elicits a faster reaction from the somatosensory cortex. As a result, electrotactile stimulation utilizes skin real state efficiently i.e., it is more spatially information-rich. Furthermore, when compared with vibrotactile, electrotactile stimulation is capable of creating a wide range of sensation without any onset delay or mechanical sound profile. Our in-house tests suggest that electrotactile stimulations as short as 5ms are instantly detectable by human participants. As a result, electrotactile stands out as a preferred candidate for a rapid and precise M2HC through tactile stimulation. 14 2.4 Principals of electrotactile stimulation 2.4.1 Existing electrotactile systems There are a multitude of success stories utilizing electrotactile in an M2HC system. Traditionally electrotactile stimulation has been used for prosthetics, rehabilitation, haptics, and most recently AR/VR experiences. Perhaps the most successful implementation of electrotactile stimulation has been in prosthetic feedback by providing a sensory substitution for sense of touch by re-creating the tactile signals felt by artificial limb and recreating them on body surface. The work provided in [34] reviews such latest efforts for both invasive and noninvasive methods with a focus on electrotactile communication. The work presented in [36] indicates that an electrotactile feedback from a prosthetic arm increases the sense of embodiment. The work presented in [45] introduces an electrotactile array for forearm and a corresponding pressure sensor array for a prosthetic hand. The researchers prove that a user wearing an electrotactile array can understand patterns being played on pressure sensor array. This work later-on gets implemented in a prosthetic system in [60], [61]. Even though electrotactile stimulation is considered a method of peripheral nerve stimulation, complete recreation of sense of touch will require producing the idiosyncratic action potential firing pattern of the majority of tactile receptors involved which is currently technically impossible to achieve [77], [78]. As such, current electrotactile literature is limited to creation of simple sensations such as vibration and intensity and is not able to completely recreate a natural-feeling sense of touch [79]. Prosthetic limbs are not the only targets for electrotactile communication. The work presented in [50], [80] showcase sensory substitution systems for the visually impaired by implementing electrotactile communication over tongue and forehead respectively. The tongue 15 electrotactile communication for assistance to blind people is called BrainPort and is an early success story of electrotactile M2HC having passed clinical trials [72] and been commercialized. Electrotactile stimulation can be used as a method for rehabilitation purposes as well. [81] uses BrainPort for vestibular rehabilitation for patients with bilateral vestibular loss who did not respond well to conventional rehabilitation methods. This study has achieved statistically significant improvements in patient's sensory organization test scores in control groups. BrainPort has also been used as a two-week balance rehabilitation program for patients with progressive cerebellar ataxia [74]. One reason that BrainPort has seen success in sensory substitution is that it fore-goes interfacing with external skin and instead opts to use the tongue. This allows BrainPort to forego a lot of limitations for electrotactile stimulation such as possibility of painful sensations due to dehydration or high-voltage circuitry to bypass skin dermis layer. However, using the tongue for sensory substitution is not a sustainable solution because of its disruptive nature to the day-to-day life. AS such BrainPort is more suitable as a temporary rehabilitation application as highlighted in [74]. Electrotactile can be used to convey information to its user. A successful implementation of an electrotactile information display has been demonstrated in [82] utilizing the spatial resolution of tactile receptors. Electrotactile has been reported in literature as a means of displaying information in interactive and virtual environments. The work presented in [83] proposes the idea of large electrotactile displays for giving directional information. Another possible use for electrotactile stimulation is communication for virtual and augmented reality (VR and AR). The work presented in [33] stimulates fingertips with a hybrid electrotactile/vibrotactile communication that can present most surface structures in Realtime to the VR/AR user. 16 Information-bearing electrotactile stimulation can also be used in other areas such prosthetics. The work presented in [61] can convey eight data points regarding the position of user’s artificial hand position with an average success rate of 90%. The idea of using electrotactile for VR/AR feedback is lucrative. However, the high-voltage nature of electrotactile stimulation means that the supporting circuitry is bulky and not yet suitable for wearable applications. Despite the success of electrotactile technology as a solution for some M2HC applications, it has not enjoyed general acceptance and commercial success. Our analysis of electrotactile literature attributed this issue to the following: • Our understanding of skin tactile receptors, especially hairy skin, is nascent. As such, cohesive description of physiological and perceptual aspects of electrotactile stimulation from an engineering point of view is scant. • Even though electrotactile stimulation is a form of neural stimulation, the range of electrotactile messages delivered as a M2HC are severely limited. • We cannot use intensity of electrotactile sensation (through amplitude modulation as discussed in literature) as a means for conveying electrotactile messages since modulating intensity can create painful sensations. 2.4.2 Electrophysiology of the skin Full understanding of operation conditions of electrotactile stimulation require a basic knowledge of skin structure and the afferent nerve types and placement. The human skin is the largest organ in human body with three primary tasks of protecting human physiology from external threats, heat regulation, and tactile input to the human brain. However, it should be noted that human skin is not the only organ that produces tactile communication. The library of 17 tactile inputs that are known as somatosensory communication include tactile communication from the skin, muscles, tendons, joints, and bones. However, in this work tactile input will be limited to only input from human skin. The human skin can be classified into three categories: mucocutaneous, hairy, and glabrous [84]. The mucocutaneous skin tissue is primarily the connecting tissue between other skin types and the internal membranes such as lips and eyelids and is out of the scope of this dissertation. The hairy skin refers to the smooth skin that contains hair glands i.e., dermal filaments. Glabrous skin refers to the smooth undulating hairless skin of the palm, soles of feet, etc. This skin type is the predominant target for electrotactile stimulation in literature due to its high concentration of receptors. However, glabrous skin is actively used in day-to-day activities of humans as the use of hands and feet are essential to daily human life and their usage for M2HC would be too obtrusive. The focus of this study will be the hairy/glabrous skin since they are the primary pathways for unobtrusive tactile input pathways. Figure 2.3: Different layers of human skin. The thickness of each layer is not up to scale. Human skin is comprised of three layers. As shown in Figure 2.3, these layers are called: epidermis, dermis and subcutis. The epidermis is the outermost layer of the skin, creating a boundary between the aquatic dermis layer and the outside world. This ~100 µm thick layer of skin which contains 50-100 cells, is the primary isolation and defensive layer of the human 18 defense system [85]. This layer of skin is everchanging, being produced in the aqueous layer of epidermis and being peeled off on the outer layer. During this constant growth mechanism, as cells are being pushed outside, they start producing keratohyalin which will eventually suffocate the cell and form a watertight defensive barrier on the outer epidermis layer [86]. The dermis layer of the skin contains hair follicles, sweat glands, sebaceous glands, apocrine glands, lymphatic vessels, blood vessels, afferent fibers and their corresponding tactile receptors [86], [87]. The dermis is comprised of two sublayers, the upper papillary, and the reticular. The upper papillary layer is a wavy undulating structure that is molded into the base of epidermis layer. Due to its wavy nature, this layer has a large surface area and is host to some of the more spatially accurate tactile receptors. The reticular layer contains a lot of elastin, fibroblast, and collagen material that help with the structural integrity of the skin. The innermost layer of the skin is the subcutis layer which is predominantly a connecting tissue between deeper tissues and the skin. This layer contributes to skin stiffness and is responsible for storage of fat. The somatosensory system of human brain oversees three main tactile functions: exteroceptive, interoceptive, proprioceptive [88]. The exteroceptive and interoceptive function are the brains perception of outside and inside the body respectively whereas the proprioceptive function oversees tracking body position and maintaining balance. The cutaneous sensory neurons that interface with exteroceptive and interoceptive function are classified as Aβ, Aδ, and C neurons based on their cell body size, axon diameter, degree of myelination and axon conduction velocity [89]. Myelination is a condition when nerve axon paths are covered with a layer of fat as shown in Figure 2.4. Myelination speeds up the movement of action potential through an axon (Na+ ions) by reducing the parasitic capacitance between the axon pathways 19 and the outside ions and enabling saltatory conduction [90]. In a normal axon path, movement of electrical signals is enabled via diffusion conduction of ions. In saltatory conduction, the ionic signal traveling through axons is periodically amplified through unmyelinated spots on axon path called nodes of Ranvier. Nodes of Ranvier are covered with voltage-gated ionic channels which get activated if an action potential is traveling through the axon and rejuvenate the diffusion current by absorbing free-roaming ions outside the axon. As a result, nodes of Ranvier act as amplifiers along the axon path, and increase axon conduction velocity. The three types of afferent neurons can be described as shown below: • C afferent: smallest body size, most abundant, unmyelinated axons, slowest conduction velocity • Aδ: Medium sized, lightly myelinated, intermediate velocity (5-30 m/s) • Aβ: Large sized, heavy myelination, rapid velocity (16-100 m/s) 20 Figure 2.4: LTMR nerve axon with Myelin sheet and nodes of Ranvier. An action potential travels through Myelinated portions of the axon pathway using saltatory conduction and is amplified in nodes of Ranvier through depolarization of ionic channels. The myelination degree of an afferent neuron determines their activation threshold with reverse correlation [89]. Each of these afferent neurons have an end organ that is called a mechanoreceptor. Mechanoreceptors are tactile sensors that can be found in most locations of human body. Based on their threshold levels, they can be categorized as low threshold/high threshold (LTMR/HTMR). Generally, LTMR are associated with sensing of light touch whereas HTMR are used for sensing toxic inputs (pain, burn, etc.). However, there are LTMR classes for non- Aβ receptors that are called Aδ-LTMR and C-LTMR that are found in hair follicles. Due to their nature, tactile communication systems for M2HC aim for activation of easily activated LTMR while avoiding the more challenging HTMR. Figure 2.5 depicts the known receptors in different layers of hairy skin. Please note that the mechanoreceptors available in hairy and glabrous skin may differ in their type, location and function. The LTMRs can be further divided into four subcategories based on their receptive field and adaptation frequency range: • Frequency range: 21 o Fast adapting (FA): will show a short burst of action potentials with deformation, but then they will cease firing despite continued application o Slow adapting (SA): will start firing action potentials (or increase their firing rate) at the onset of the stimulus and continue to fire until the stimulus ends • Receptive field categories: o Type 1 units have small receptive fields with well-defined borders o Type 2 units have wider receptive fields with poorly defined borders and only a single point of maximal sensitivity, from which there is a gradual reduction in sensitivity with distance Figure 2.5: The structure of a typical hairy skin and the placement of tactile receptors, sweat glands and hair follicles. 22 The HTMR seen in Figure 2.5 are responsible for detecting toxic inputs such as temperature and pain. The qualities of each receptor are compared in Table 1.2. Electrotactile stimulation method leverages the threshold differences between LTMR and HTMR by injecting an electrical charge through the epidermis layer of skin enough to activate LTMR and not HTMR [14], [91]. Assuming a grounded and uniform skin and a small electrode, the penetration and distribution of an electrotactile current can be shown as Figure 2.6 [92]. The voltage created by a very small electrode in a Cartesian coordinate in respect to a grounded body is calculated by [92] to be: 𝛹(𝑥, 𝑦, 𝑧) = − ∫𝑅 𝐸⃗ 𝑑𝑅⃗ (2.2) where Ψ is the voltage at coordinates x, y, and z with the electrode as the center, R is the radius originated from the electrode center, and ρ is skin resistivity (assumed to be constant). Assuming there exists an LTMR axon node of Ranvier at coordinates (xe,ye, ze), the voltage Ψ(xe,ye, ze) may be strong enough to activate voltage-gated ionic channels at this location and create an artificial action potential that will travel along the axon and into the somatosensory system of the brain. As a result, due to the electrotactile stimulation of skin, the brain may feel a tactile stimulation on where the mechanoreceptor connected to the stimulated node of Ranvier is. Hence, if done properly, electrotactile signals can mimic one of the many LTMR tactile mechanoreceptors from the prospective of brain somatosensory system. The hairy and glabrous skin have a different structure of tactile receptor arrangement and range as demonstrated in Table 1.2. The glabrous skin has a higher concentration and diversity of tactile receptors compared to hairy skin. As a result, most electrotactile literature focus on glabrous skin rather than hairy skin. However, glabrous skin is sued for day-to-day tasks and is 23 not suitable for a consistent M2HC system. An M2HC system for AHP then has to be mounted on hairy skin. Hence, to realize a fully functional AHP, the mechanics of how to best stimulate hairy skin need to be analyzed and understood better. Figure 2.6: A model for penetration of electrotactile current and potential created inside the skin. 2.4.3 Electrotactile stimulation The electrochemical current inducted into the epidermis layer of skin is not uniform [14], [87], [92]. This electrochemical current is most likely conducted through pathways such as sweat ducts, sebaceous glands, and minute epithelial breaks through the skin [93]. The interaction between electrotactile electrodes can be described with a circuit model as discussed in [14], [93]. Figure 2.7 shows a combination of models presented in shows a combination of nerve Frankenhaeuser and Huxley model for nerve conduction [92], [94], and Stratum corneum impedance model of [87], [93], [95] where two electrodes are placed on the skin and are relatively far away from one another and only have one node of Ranvier accessible. Gp is the conductance of epidermis layer (through sweat ducts and epithelial breaks), Cp refers to the 24 capacitive element of epidermis layer, Gd is the ionic conductance of dermis layer, Gm and Cm refer to the capacitance and conductance of voltage-gate ionic channels of nodes or Ranvier, Gn is the internal conductance of a LTMR mechanoreceptor axon according to Frankenhaeuser and Huxley model [94]. The skin conductance Gp is variable and may change from 10 kΩ to 100 kΩ [96]. potential buildup Nodes of Myeline Sheath Ranvier Figure 2.7: The electrical model for current penetration into the skin and their interaction with nodes of Ranvier of an LTMR axons. It should be noted that Gp is non-linear and decreases with electrotactile current [87]. Due to nonlinearity in Gp, a voltage based electrotactile charge injection is challenging because under a constant voltage, the injected current would increase due to non-linear resistance drop observed at epidermis layer. As such, the transferred charge will increase until HTMRs are activated causing a pain or burning sensation [14], [87]. It is possible to implement a voltage- based stimulator with a rigid communication mechanism [49]. However, constant current electrotactile systems are simple and are preferred by the body of electrotactile literature. 25 constant charge delivery system [71] is also not as viable as constant current delivery systems due to their added complexity and no added benefit. +supply Mp1 Mp2 Op-amp Ciso Mnf Input Signal + Skin electrode - Rf Mpf Riso Mn1 Mn2 -supply (a) (b) Figure 2.8: The most common electrotactile circuitry in literature: (a) current mirror structure (b) improved Howland current source. Throughout electrotactile literature, two major methods have been leveraged for producing electrotactile current: improved Howland current source [96], [97], and current mirrors [98], [99]. These methods are shown in Figure 2.8. The current mirror technique operates 26 by displaying the input voltage waveform over a reference resistance Rf through a negative communication loop hat is consist of Mnf and Mpf. The current generated by displaying input signal over Rf is preserved by Mnf and Mpf and is conveyed to current mirror transistors Mn1 and Mp1. This current is then mirrored and injected into the skin electrode passing through isolation resistor and capacitor Riso and Ciso. The isolation elements act as a high-pass filter that isolate the DC skin voltage from the current mirror output DC voltage to minimize the risk of electrical shock and painful sensations in case of catastrophic system failure [100]. The isolation capacitances limit the total charge transferred to the skin in case of transistor and fuse failures as described in [52] where the two transistors fail and a total voltage swing of two times the VDD happens in the current mirror output. This capacitor limits the total charge delivered to the electrode. Another popular opinion for implementation of electrotactile current generator is using Howland current source [48], [70], [97], [101] as shown in Figure 2.8. The working principal of these current sources dictate that as long as the following condition is met: 𝑅1 𝑅3 = (2.4) 𝑅2 +𝑅𝑂𝑈𝑇 𝑅4 the output current displayed over Rload will be: 𝑉𝐼𝑁+ −𝑉𝐼𝑁− 𝐼𝐿 = (2.5) 𝑅𝑂𝑈𝑇 which is independent of load resistance (in this case skin resistance). Each of these methods have advantages and disadvantages associated with them. The current mirror is simpler to implement with the only requirement that the current mirror transistors can handle the high supply voltage demands. Due to its structure, the matching between the current mirror transistors is crucial. In the case of Howland current mirror, the matching between the resistors is important but the biggest issue with this structure is the fact 27 that the op-amp needs to be able to handle high-voltage supply voltages and the fact that Howland current mirror output dynamic range is limited compared to current mirror structure. Another important factor to consider for electrotactile stimulation is electrode material and surface profile. Traditionally the most widely used electrodes are for Transcutaneous electrical nerve stimulation (TENS) or Electroencephalography (EEG) applications where they are metal electrodes that are connected to a conductive polymer surface coated with adhesive gels such as Polyoxethylene 20 Cetyl Ether [102]. However, such electrodes have limited lifespan, and are too big and too impractical for day-to-day used in an electrotactile communication system. The current flow in an electrochemical system is an electrochemical current in nature [14]. As a result, to reduce skin irritation and possible damages, the electrotactile electrode must not introduce non-native ions to the skin. As such, the electrode-tissue equilibrium state where no net ion loss happens for electrode or tissue [14], [103]. The electrode may produce ions during stimulation; however, the reaction must be reversible. Furthermore, a reaction on electrode surface may produce insulating layer on electrode surface, further reducing its performance. In general, most limitations that restrict implanted electrodes also apply for electrotactile electrodes as well. The most common electrode materials reported in literature are gold, platinum, silver, and stainless steel, carbon (polymer coated with conductive graphene ink), and conductive rubber [14], [68], [69], [103]. For electrotactile stimulation, electrodes must have uniform surface profile. Inequal distribution of current under the electrode due to shape or surface contact may lead to concentration of current in specific spots. Furthermore, Inhomogeneities on the electrode surface, which exist in all practical electrodes, may produce local potential differences that favor continued loss of metal ions (pitting or crevice corrosion). 28 2.5 Summary and discussion 2.5.1 Summary Human augmentation (HA) principals and their possible effects on human lives were discussed. Real-world examples of HA and current dependency of humanity on HA and its current limitations were demonstrated. Augmented human perception (AHP) was introduced as an easily attainable form of HA with possible major impact on the human condition. The need for an effective machine-to-human communication (M2HC) for an effective AHP was introduced. Different methods of implementing M2HC and their effect on human brain, especially attention span, were introduced. It was stated that current over-reliance on visual M2HC has resulted in attention-span complications and anxiety for the world population that can be mitigated. Methods for assisting and improving attention span such as separating stimulus channels for multi-tasking or pre-processing of data were discussed. Among possible M2HC pathways, tactile communication seems to be most appropriate method of M2HC with its ease of implementation and being a separate stimulus channel. Tactile stimulation methods vibrotactile, electrovibration, and electrotactile were introduced and compared. It was concluded that electrotactile stimulation is the best option for M2HC because electrotactile has the possibility of generating complex sensations with acute punctateness and in an unobtrusive manner (no sound profile, moving parts, etc.). Examples of electrotactile stimulation from literature were discussed to provide historical precedence. The biology of electrotactile stimulation was discussed and the differences between glabrous and hairy skin were discussed under the framework of M2HC. Finally, principals of electrotactile circuitry and electrodes were discussed with examples from the literature. However, several challenges remain to be resolved for electrotactile to create an 29 impactful presence in M2HC such as a universal model for creation of distinct electrotactile sensations at a higher order than is present in literature. 2.5.2 Discussion Exploration of literature in M2HC with a focus on electrotactile M2HC brough about several important conclusions. With the advent of information technology, humanity has become more prone to anxiety and split attention due to over-abundance of information. While access to information is a positive change, it is possible to alleviate some of the pressure on brain with novel approaches to M2HC. It was proven that electrotactile stimulation has the opportunity to be a viable alternative for M2HC in a manner that alleviates the attention span burden created on the brain caused by current M2HC methods such as visual communication. This provides a new pathway to the brain that was not present before. Through our investigation of electrotactile literature, we realized that electrotactile stimulation is at its core, a non-invasive method of peripheral nerve stimulation and presents greater access to the brain compared with traditional haptic techniques. However, several challenges need to be addressed for electrotactile to be a viable method of M2HC. For electrotactile M2HC to be viable, the electrotactile circuitry needs to be scaled down to a wearable size, possibly with wireless information communication capabilities. The electrotactile stimulation circuitry needs to be modified such that the skin resistance and the voltage variations on the skin do not compromise the charge injected into the skin. The circuitry presented on Figure 2.8 is dependent on the voltage swings of the output node, creating unreliable electrotactile current levels at high current outputs. Electrotactile stimulation hardware should capable of producing electrotactile waveforms at much higher frequencies to 30 facilitate emerging high-frequency electrotactile stimulation techniques with the capability for driving multiple electrotactile simultaneously. Finally, embedded control systems need to be implemented in a wearable/portable package size that would allow for electrotactile to be moved out of lab environment and into real world. Creating unique electrotactile stimulation patterns mean that we need to invest in software infrastructure such as a graphical or text-based user interface to create and stimulate various electrotactile waveforms. Electrotactile has not been extensively used as an M2HC with a focus on information/data transfer to the brain. Electrotactile has been traditionally used as a method of haptic or prosthetic communication. As a result, groundwork needs to be done to enable using data-bearing electrotactile information. The literature suggested some basic techniques for doing so through frequency and amplitude stimulation that range from 500ms to tens of seconds which are not adequate for electrotactile M2HC. The work presented in Chapters 3, 4, and 5 will and introduce novel methods for creating electrotactile messages that are acceptable for M2HC. 31 Chapter 3: Generating complex icons via electrotactile stimulation A key requirement for having an effective M2HC is the ability to convey complex messages to the human. Traditionally this feat has been achieved by using visual communication. Any competing M2HC technology should at least be comparable to visual communication in terms of complexity of signals they can communicate to be a viable option for AHP. Observing the intricacies of the human tactile communication, it is safe to assume that there is a myriad of ways for an electrotactile communication to generate and convey complex messages. However, these methods have not been sufficiently studied in the literature. This is mainly because electrotactile literature is in general concerned with conveying rudimentary M2HC for prosthetic devices which are focused on conveying simple spatial cues to the somatosensory cortex. This chapter will analyze and classify possible methods for generation of complex M2HC electrotactile waveforms through creation of distinct sensations. These waveforms will in return be used in later chapters to convey complex electrotactile messages. 3.1 Complex electrotactile signals and iconic communication Given the complexity of human nervous system, it is no surprise that electrotactile stimulation would elicit a complex array of responses. Due to its intrinsic ability to create distinct sensations, through vibration frequency and intensity, electrotactile can be used as a pathway for M2HC [59]. Vibrotactile literature suggest that a possible method to create M2HC the sense of touch is through iconic communication [104] [61], [105]. An icon is a brief and tangible stimulation of human senses with a meaning associated with each icon [106], given that the stimulation is a differentiable, identifiable, and learnable distinct sensation [107]. It is possible for a combination of icons can be assembled to communicate complex messages to a person. As 32 such it is essential for M2HC icon generation software and hardware to produce distinct and reliably detectable sensations. Furthermore, the distinct sensations need to be brief enough to be comparable with audio and visual icons. The average human reading speed is 300ms. As such the icons need to create a distinct sensation within a comparable time-table. In this thesis, we utilize electrotactile icons (etacons) as a means for creating M2HC. Research efforts are being made to create distinct sensations which is crucial for a creation of successful etacons, even though no literature has used etacons to the best of our knowledge. For example, the work presented in [59] produces four distinguishable electrotactile pulse frequencies at 10, 15, 35, and 100 Hz and four distinct perceived intensities, effectively creating 16 distinct sensations. The work presented in [61] was able to create 5 distinct sensations using stimulation frequency on the forearm with a success rate of ~80% in a prosthetic arm feedback. In order to understand the possibilities created by electrotactile M2HC, a classification method was devised by attributing the complexities created to the quality, temporal parameters, and spatial parameters of the electrotactile M2HC as shown in Figure 3.1. This model will be referred to as electrotactile dimensions. Spatial stimulation is attributed to sensations that are activated on multiple locations on the skin, sequentially or simultaneously. They often create a sense of movement or a widespread signal. Temporal stimulation is when multiple sensations are produced back-to-back, creating a signal that may evolve with time. Quality of sensation is factor that is created when the sensation a singular electrotactile stimulation evokes is subject to change. For example, perceiving a difference due to changes in stimulation frequency, amplitude, perceived pressure, etc. can be classified as changes in quality of stimulation. 33 Figure 3.1: Electrotactile dimensions that can be used to create distinct sensations and may be used for creation of complex waveforms Even though spatial and temporal parameters are important, they are relatively easier to evoke. However, changes to the quality a stimulation is a more complicated topic and has the possibility to make major impacts for M2HC complex messaging. As a result, this thesis will focus on trying to convey complex etacons on a single electrode. Once supremacy with a singular electrode has been achieved, the electrotactile etacons can be expanded to include spatial and temporal parameters as variables for creation of complex M2HC. 3.2 Model for conveying complex etacons using qualitative elements Literature indicates that electrotactile stimulations must be modulated to prevent fatigue in tactile receptors [14]. However, this modulation has been proven to influence the perceived frequency of [59], [61]. Based on our in-house experiments and literature, the design space for creation of qualitatively distinct signals is consist of frequency and amplitude of electrotactile stimulation [59], [61], [108]. As a result, users can differentiate electrotactile inputs based on 34 their perceived frequency and amplitude. However, amplitude is not a reliable stimulation variable [109]. For an effective stimulation, the electrotactile current amplitude must be between a minimum and maximum threshold called the “sensation threshold” and “discomfort threshold” [59]. As a result, all amplitude variations must be within this range. However, the threshold levels may vary based on the stimulation duration, site, amplitude, and frequency [58]. As such, stimulation amplitude is not a reliable source of generating distinct electrotactile signals. The perceived texture is usually studied in the mid-range frequencies (10-100Hz) wherein 3-4 unique textures have been reliably distinguished in the literature [57], [108]. This is because electrotactile stimulation operates by activating afferent nerve axons from tactile receptors. As a result, the electrotactile emulates the response of a tactile receptors. As such, it can be assumed the closer the electrotactile mimics the natural response of a tactile receptor, the more natural and comfortable they will feel. The natural response of different electrotactile receptors is shown on Table 2.2. On the oher hand, not much attention has been expended in electrotactile literature to grasp the effects of electrotactile stmulation frequency beyond a value that may change linearly [57], [108]. As a result, we created a model for generating multi-frequency electrotactile stimulation (MES) for creation of complex electrotactile stimulations. The goal of MES is to rely solely on multi-frequency modulation of basic electrotactile pulses for generation of distinct sensations. The MES model as it stands has ten controllable parameters (explained below) out of which eight are independent, meaning they can be individually controlled. The overall goal for introducing MES method is to try and draw, mix, and match “distinction elements” from each of these parameters and use these distinction elements to create complex etacons that can be conveyed 35 to users as individual M2HC. MES is composed of three frequency layers modulating one another. As shown in Figure 3.2 these three layers are: pulse signals, packs and bundles. The reason for such specific classification was the fact that the average brain perceives the sub ~10Hz signals (bundles) consciously while frequencies above 10Hz (packs) usually get translated as texture. Furthermore, Packs and pulse signals do not have a one-on-one correlation i.e., pulse signals can be used to alter packs. Figure 3.2 depict and example electrotactile waveform created with these three elements. Bundles Packs Pulse signals Figure 3.2: MES models for generation of complex electrotactile waveforms. Pulse signals are the fundamental element of our electrotactile stimulation wherein an electrical current is injected into the skin in the shape of a pulse. Majority of pulse signals in literature are typically in mono-phasic or bi-phasic square wave shapes, even though they can take any shape possible such as pulsons, triangular, etc. [24]. Figure 3.3 depicts control parameters for pulse signals and their typical values based on [52]. NS denotes the number of 36 back-to-back pulses, SRR is pulse signal repetition rate, SPW is a single pulse width, and SW is pulse signal width. Square wave pulses are the recommended pulse signal shapes because they are charge neutral and thus, do not oxidize the electrodes or create charge build-up in skin. Figure 3.3: Prominent parameters of pulse-stimulation-level of MES and their respective range. Packs shape and modulate pulse signals in the 10-100Hz frequency range. The parameters controlling pack-level modulation are depicted in Figure 3.4 where PRR is pack repetition rate, NP is the number of packs and PDC is the pack duty cycle. PDC effectively controls the NS parameters in pulse signal layer. The value of PDC can be calculated using the following formula: 𝑃𝑅𝑅 𝑃𝐷𝐶 = NS (3.1) 𝑆𝑅𝑅 and the value of PDC cannot go above 1 or 100%. Figure 3.4: Prominent parameters of package-level MES and their respective range. 37 Bundles provide high-level signal shaping for electrotactile stimulation. Similar to packs, three parameters control bundling. BRR is Bundle repetition rate which ranges from 1Hz to 20 Hz, number of bundles NB and Bundle duty cycle. What set bundling apart is that we have ensured on a hardware and software level that each bundle may be independent from other bundles in terms of its BDC, pack and pulse signal elements. This will facilitate changing patterns during a single stimulation. The total number of bundles can be as low as 1 (making the MES effectively a two-level modulation) or 10s of bundles. However, it is preferred to keep the total stimulation time less than 500ms. Similar to PDC, BDC can be calculated using the following formula: 𝐵𝑅𝑅 𝐵𝐷𝐶 = NP (3.2) 𝑃𝑅𝑅 Figure 3.5: Prominent parameters of bundle-level MES and their respective range. To evaluate the ability of MES model to elicit multiple distinct sensations on a single electrode, an electrotactile stimulation hardware platform was designed and implemented. This hardware was used to test how different parameters such as BRR, PRR, etc., affect the sensation quality of electrotactile stimulation. 38 3.3 Hardware and software platform for generation of electrotactile stimulation For the electrotactile stimulation core, the current mirror approach was opted due to its flexibility for future expansions, expandable electrotactile output current, and simple nature. The expandability is important because it allows for creation of multiple output electrotactile currents by just adding a new current mirror output. However, modifications were made to improve the performance of the current mirror. As depicted in Figure 3.6 we opted to use a cascode current mirror as opposed the classic current mirror found in literature to ensure variations in output voltage do not affect the accuracy of current mirror. In a current mirror structure, the matching between the mirror transistors is important to ensure correct operation. However, since discrete transistor are used for a current mirror structure, the matching between Qp1 and Qp2, and Qn1 and Qn2 would not be predictable. A list of essential components used for realizing the current mirror is shown on Table 3.1. To ensure matching between the two transistors. The transistors used in this project are high-voltage discrete BJT transistors. As a result, they are not matched with one another which is vital to ensure functionality of current mirror structure. As a result, Ru1, Ru2, Rd1, and Rd2 were used to calibrate and match the current mirror transistors. The output of the presented current mirror without the presences of an electrotactile electrode in contact with a skin would be a high-resistance node. As such, Ru2 and Rd2 are also used for ensuring the idle operating voltage of output node will stay around ground voltage for when the electrotactile probe is not in contact with skin. The connection of an electrotactile probe will reduce the resistance of this node so floating output voltage will not be of concern. 39 Figure 3.6: The schematics of modified current mirror electrotactile current generator used for this chapter. Qx3 and Qx4 are cascode transistors to ensure electrotactile current is independent of skin voltage. Table 3.1: list of components used for realization of electrotactile current generator. Qp ZTX558 High voltage PNP transistor Qn APT27HZTR High voltage NPN transistor OP-amp 1 AD8028 Rail-to-rail low distortion amplifier OP-amp 2 OPA604AU low distortion amplifier PWM DAC ATmega32U4 8-Bit embedded USB microcontroller 8 channel MUX HV2201 High-voltage analog multiplexer The pulse signals for this setup are created by pulse width modulation (PWM) digital to analog converter (DAC) available through ATmega32U4 microcontroller unit Timer 4. The PWM pulses are at a frequency of 10MHz and their goal is to create analog pulse signals at frequency of 100kHz and lower. As such, a Sallen-key filter using Op-amp1 was used to perform filtering and removing DC baseline signals. Since we wish to preserve the option to display electrotactile signals over multiple electrodes, an analog multiplexer unit (MUX) was implemented for conveying electrotactile signals to 8 electrodes. The analog MUX was controlled by the 40 ATmega32U4 microcontroller using SPI. For generation of the demanding supply voltages two R05-100B DC-DC converters were used to convert a 5V supply into ±100V. The overall block- diagram depicted in Figure 3.7. Figure 3.7: The hardware/software infrastructure used for conducting human trials. The operation is controlled by a PC software that is in constant contact through a USB communication with the microcontroller firmware. We wrote the PC interface to be composed of three Python scripts as shown by the “PC software” in Figure 3.7. The waveform generation takes user instructions about the shape of electrotactile waveforms through a .csv file. The waveform generator then translates the user-friendly instruction set into a microcontroller firmware compatible script and stores the instructions in a separate .csv file referred to as seed file. Furthermore, the waveform generation block scrambles the order of electrotactile waveforms in the seed file through a randomization function with a specific random number generator seed number (hence the name seed file). The seed file key is stored in the seed file name so that it can be used later to unscramble the test participant’s responses to the 41 electrotactile waveforms. The user interface takes the seed files and transfers them as instructions to the microcontroller. We wrote the user interface to provide the user with a set of tools to control the flow of an electrotactile stimulation experiment. These tools are stream, repeat, and quit wherein stream send the next set of electrotactile waveforms as instructions to the microcontroller to be played, repeat resends the previous instructions to the microcontroller, and quit stops the experiment. The user interface also has a special function that sends a premade electrotactile waveforms to the microcontroller at increasing intensities to allow for us to find the sensation and discomfort thresholds of a participant. Once a participant has been stimulated by an electrotactile waveforms, their response is recorded and processed through another set of PC software that we wrote for this research. The participant primarily interacts with the PC graphical user interface (GUI), which is a Python script written to capture participant responses (e.g., “Same”, “Different”) through a keyboard input and records them together with meta data such as date and participant number into a .csv file. The participant responses are then read by the information analysis block which is composed of multiple modules. As part of information analysis, a Python script reads the participant response csv file, and de-scramble the responses based on the random number generator seed number. The de-scrambled responses are then read and analyzed by an application specific MATLAB script. Multiple studies have been conducted in this thesis and each usually have their own specific MATLAB analysis file. We prepared the ATmega32U4 firmware to run as a state machine that controls its operations and allows for it to communicate via USB. The state machine receives and stores instructions for a single waveform and then as soon as it receives the stream command from the 42 user interface, it executes the waveform on PWM DAC. The PWM DAC operation is run by Timer 4 of the microcontroller and its timings are controlled by a parallel running Timer 1. The two timers together perform an On-Off-Keying (OOK) operation based on received waveform shape information. There are a slew of options that the microcontroller state machine provides for fine- tuning the shape, frequency, and amplitude of the waveforms. We created the complete suite of codes used for the PC and microcontroller firmware in this project and have made them publicly accessible via [110]. The electrodes on skin surface play an important role in the quality of sensation received. An imperfect connection between the electrode and the skin could result in a painful experience. [14]. The skin surface area is not uniform in terms of its conductivity. current flow through a small low resistance region (sweat ducts, sebaceous glands, and minute epithelial breaks, 1-6 per mm2 skin area[111] will result in pore breakdown and painful sensations [14]. As a result, a curved electrode surface was opted for a flat one to avoid any defective connections. An array containing 23 electrodes, 8 anode/cathode and 15 ground, are fabricated on a standard PCB to realize the electrotactile stimulation as shown on Figure 3.8. Only one of the cathode/anode electrodes are used for this work. The copper pads are connected to medical-grade stainless steel balls (316 l surgical balls) using conductive silver epoxy paste and then covered with biocompatible epoxy layer. The electrodes are place on the left brachioradial muscle due to their smooth surface lacking any major nerve bundles that may affect the sensation. The overall system and the fabricated electrodes can be viewed in Figure 3.9. 43 Figure 3.8: Electrode array used for this chapter. The electrodes are medical-grade stainless steel balls placed on a standard PCB substrate. Current Mirror/op-amp Electrode array Multiplexer/ Filter ATmega32U4 Figure 3.9: Completer hardware implementation. 3.4 Eliciting distinct sensations on a single electrode using MES Three experiments were designed and executed to understand the effects of high- frequency electrotactile stimulation for verifying MES model and producing a wide range of uniquely perceived electrotactile waveforms. A total stimulation window of 0.5s has been selected to provide a messaging speed similar to the typical reading speed of humans (1 word per 0.3 s). 3.4.1 Experiment I: exploring the ability of humans to differentiate high-frequency pulses This Experiment was conducted to evaluate the reaction of human subjects to an electrotactile stimulation with SRR above 100Hz. The goal is to explore the assertion made by 44 literature [52], [59], [61] that this frequency range does not yield a difference in terms of perceived sensation. Three participants (two males, one female) within the age-group of 18-30 were selected for this Experiment. In this setup, biphasic stimulation pulses with pulse-width of 120µs and a total length of 480µs were selected as base pulses. Hence the maximum SRR achievable with this setup is ~2100Hz. For this Experiment, SRR was swept from 100Hz to 2.1kHz and each iteration was presented as a waveform for 0.5s. Figure 3.10 depicts three example waveforms with different SRR values. SRR was increased linearly with 100Hz steps, constituting a total of 20 steps. Each SRR waveform step was presented to the user as a comparison in which the target waveform was compared to a reference waveform with SRR of 100Hz. The reference and target waveforms were stimulated on the participant with a 1s delay between them. The sequencing of waveform comparisons was random from the pool of 100Hz to 2.1kHz to prevent bias. Immediately after each comparison was presented, participants were asked to rate the distinction between the two comparison waveforms as same or different (0 or 1, respectively). 45 Figure 3.10: Example waveforms for 3 of the 20 SRR (pulse repetition rate) frequencies used in Experiment I. Amplitude is normalized to ±1. The average comparative discrimination response of participants is presented in Figure 3.11, where a 1 indicates that all participants unanimously marked the SRR value as different from the reference and a 0 means they unanimously could not tell them apart. These results indicate that, even though some high frequency waveforms demonstrate some distinction in sensation, the uncertainty in these responses suggest that high-frequency stimulation alone cannot consistently produce noticeable differences. This conclusion corroborates the general assumption in literature that high-frequency pulses fail to provide distinct vibration sensations. However, it also demonstrates that pulse frequencies up to 2100Hz can be reliably detected, which motivated Experiment II. 46 Figure 3.11: Average comparative discrimination response of participants and the variation across all participants SRR frequencies spanning from 100Hz to 2.1kHz. 3.4.2 Experiment II: exploring the ability of humans to differentiate Packs This Experiment was conducted to evaluate the reaction of human subjects to an electrotactile stimulation with SRR of 1kHz modulated by a varying PRR in the range of 10Hz to 100Hz, and keeping NB = 1. The goal of Experiment II was to confirm the assertion made by literature [40], [48], [50] that frequency range of 10Hz to 100Hz yields multiple distinction levels and that these levels are not impeded by the presence of high-frequency SRR pulses. Confirmation of these assertions would indicate that pulse level signal modulation could be adopted to create more diversity in electrotactile sensations. Three participants (two males, one female) within the age-group of 18-30 were selected for this Experiment. SRR was kept at 1kHz and NS was set to 5. PRR was varied from 10Hz to 100Hz with an initial step size of 2Hz over a 0.5s long messaging window. Figure 3.12 depicts four example waveforms with increasing PRR values. PRR step values were presented to participants where PRR was randomly varied between baseline and baseline plus 40Hz (2*20Hz). Each PRR target waveform step was presented to the user as a comparison in which the target waveform was 47 compared with a reference baseline PRR waveform. The reference and target waveforms were stimulated on the participant with a 1s delay between them. Immediately after each comparison was presented, participants were asked to rate the distinction between the two waveforms as same or different (0 or 1, respectively). Once a PRR value was found that produced consistent distinction, it was recorded and the test was reset. This process was repeated three times, at the end of which the average response to varying PRR was calculated. The PRR where participants produced consistent distinction was noted and set as the new baseline PRR for testing higher PRR values. An example results for average PRR distinctions performed on participant #3 is shown on Figure 3.13. Once the second baseline was found, the PRR step was increased from 2Hz to 5Hz to cover a wider frequency range. PRR= 10Hz Normalized Amplitude PRR= 20Hz PRR= 50Hz PRR= 100Hz Time Figure 3.12: Example waveforms for 4 of the 20 PRR (bundle repetition rate) frequencies used in Experiment I. Amplitude is normalized to ±1. 48 Baselines: BRR= 10Hz BRR= 24Hz BRR= 34 Successful distinction at Average decisions BRR = 22Hz Successful Successful distinction at distinction at BRR = 32Hz BRR = 64Hz BRR frequency (Hz) Figure 3.13: Average comparative distinction response of participant #3 as a function of BRR (bundle repetition rate). The extracted comparison distinction baselines at shown at the top. The distinction baselines, representing ranges of PRR that are clearly distinguished as unique sensations, are depicted in Figure 3.14 for each of three participants along with the average distinction baseline across participants. From the starting baseline PRR of 10Hz, the participants identified at least three distinction ranges with average distinction thresholds at PRR= 18Hz, 32Hz, and 67Hz. These results are similar to distinction levels reported in literature for low and mid-frequency base pulses [40], [48], [50]. As such, it can be inferred that utilizing high-frequency SRR does not interfere with lower-frequency bundles distinction levels. This provides the opportunity of utilizing high-frequency pulses for producing more distinct waveforms without compromising the performance in other frequency bands. Based on this result, a third Experiment was designed to evaluate the response of individuals to modulation of high-frequency Pulses and their effects on the perceived sensation in a fixed Bundle frequency. 49 Individual responses BRR = 10Hz BRR = 18Hz BRR = 32Hz BRR = 67Hz BRR = 10Hz BRR = 24Hz BRR = 34Hz BRR = 69Hz p4 BRR = 10Hz BRR = 16Hz BRR = 32Hz BRR = 72Hz p1 p2 p3 BRR = 10Hz BRR = 18Hz BRR = 30Hz BRR = 60Hz 10 20 40 80 BRR frequency (Hz) Figure 3.14: Average comparative discrimination response of participants and the variation across all participants for BRR frequencies spanning from 10Hz to 100Hz in Experiment I. 3.4.3 Experiment III: Discriminating high-frequency pulse trains This Experiment explored the perception of new and distinct sensations by varying the number of high-frequency Pulse signals within each low frequency Bundle, defined by NS in Figure 3.3, while modulating the signal at a fixed BRR. It should be noted that varying NS is effectively the same as varying the duty cycle of the pack or PDC. If proven true, this Experiment will prove that the full frequency range higher than 100Hz can also be used as a distinction element even though these frequencies cannot create distinction elements on their own. Four participants (three males, one female) within the age-group of 18-30 were selected for this Experiment. SRR was set to 400Hz and PRR was set to 6Hz. This permits the number of Pulses signals per pack (NS) to range from 1 to 66. This variation is controlled by PDC to make sure as opposed to Experiment I wherein pulse signals were distributed over stimulation window uniformly. For this Experiment, NS was tested in steps of two pulses, with a total stimulation window of 0.5s. Figure 3.15 depicts four modulation waveforms for different NS and PDC values. 50 Similar to Experiment I, each value of NS was presented as a comparison of waveforms, separated by a 1s delay, against a baseline waveform where the initial baseline used NS=1. Immediately after each comparison, each participant was asked to rate the distinction between the two waveforms as same or different (0 or 1, respectively). To prevent bias, the target NS value was randomized from the baseline value to the baseline plus 20. As depicted in Figure 3.16, which plots the results for participant #3, once an NS value was found that produced consistent distinction, the comparison baseline was set to the distinction NS plus one. The Experiment was then continued by randomly varying NS between the new baseline and the new baseline plus 20 and repeating this cycle until the limit of NS = 66 was reached. Figure 3.15: Example waveforms for different NS (number of Pulses per Bundle) used in Experiment II. 51 Figure 3.16: Comparative distinction reaction of participant #3 to the increasing NS (number of Pulses per Bundle) and the extracted comparison distinction baselines. Figure 3.16 depicts the experimental results for one participant (participant #3) who achieved distinction baselines at NS=8 and NS=23. Figure 3.17 depicts the distinction baselines for each of four participants, where the top graph shows the average of distinction baselines of all participants. The participants identified at least three distinction points for NS < 66 with average distinction thresholds at NS= 7 and 33 or PDC= 10% and 50%. These results demonstrate that high frequency electrotactile pulses can be employed as distinction elements by modulating them at lower frequency and varying the modulation duty cycle. 52 Figure 3.17: Baselines for NS (number of Pulses per Bundle) for all participants and the average value of their response. 3.5 Summary and discussion of work 3.5.1 Summary This chapter introduced four milestones toward the overall goal of proposing a fully functional electrotactile M2HC for AHP. Multi-frequency electrotactile stimulation (MES) method for generating complex electrotactile etacons was introduced. MES is inspired by the natural action-potential generation behavior of tactile receptors and relies on a three-level frequency modulation to produce distinct electrotactile signals. The MES model has eight controllable parameters. The overall goal for introducing this method is to try and draw, mix, and match “distinction elements” from each of these variable and use these distinction elements to create complex etacons. The frequency modulation ranges are bundles (1-20Hz), packs (10-200Hz), and pulse signals (100Hz - 2.1kHZ). An embedded system was created to generate electrotactile signals using an enhanced current mirror approach paired with a microcontroller-based waveform generator composed of a PWM DAC and a bandpass Sallen-Key filter. Medical-grade 316L stainless steel-based electrotactile electrodes were created by affixing stainless steel balls 53 on the surface of a standard PCB using silver-epoxy and covering the whole board with a protective polymer layer. Multiple safety features were implemented to ensure the safety of exciting electrotactile sensations. Three experiments were designed and executed to understand the effects of high- frequency electrotactile stimulation for producing a wide range of uniquely perceived electrotactile stimulations. Experiment I was conducted to evaluate the reaction of human subjects to an electrotactile stimulation with pulse signal frequency (SRR) above 100Hz with the goal of confirming the assumption that standalone high-frequency pulse signals do not yield a difference in terms of perceived sensation. The results depicted in Figure 3.12 confirm this assumption by demonstrating that even though some high frequency waveforms demonstrate some distinction in sensation, this distinction is inconsistent as a standalone distinction element. Experiment II evaluated the reaction of human subjects to an electrotactile stimulation with a fixed SRR of 1kHz modulated by a varying PRR in the range of 10Hz to 100Hz with NB = 1. The goal of Experiment II was to confirm the assumptions that frequency range of 10Hz to 100Hz yields multiple distinction levels [40], [48], [50] and that these levels are not impeded by the presence of high-frequency SRR pulses. Results indicated that starting at the baseline PRR of 10Hz, the participants identified at least three distinction ranges with average distinction thresholds at PRR= 18Hz, 32Hz, and 67Hz. PRR can be used as a distinction element and that utilizing high-frequency SRR does not interfere with lower-frequency bundle distinction levels. This provides the opportunity of utilizing high-frequency pulses for producing more distinct waveforms without compromising the performance in other frequency bands, which is explored 54 in Experiment III. It should be noted that the results acquired for the relationship between pulse signals and packs and their independency can be extended to pulse signals and bundles. Experiment III explored the perception of new and distinct sensations by varying the number of high-frequency Pulse signals within each low frequency Bundle, defined by NS and PDC, while modulating the signal at a fixed PRR. SRR was set to 400Hz, PRR was set to 6Hz, and NP was set to 1. NS was increased in the range of NS=1-66. This increase was dictated by an increase in PDC which is different than the distributed increase presented in Experiment I. It should be noted that varying NS is effectively the same as varying the duty cycle of the pack or PDC. The results indicated that varying NS over a constant BRR created at least three distinction levels at NS= 1, 7 and 33 or PDC= 2%, 10% and 50%. As such it can be inferred PDC can be used as a parameter to draw distinction upon without it affecting the other well-established distinction elements PRR and BRR. 3.5.2 Discussion and open challenges This chapter has identified two parameters of distinction for electrotactile stimulation, namely the pack duty cycle (PDC) and frequency (PRR). Experiments covered in this chapter have shown that both PRR and PDC have multiple distinction levels. Based on reports in literature, the bundle frequency (BRR) can be assumed to provide multiple distinction levels as well. If, by a conservative estimate, each of these three variables had three levels of distinction, 33= 27 distinct etacons could be delivered to the user using a single electrode. This hypothesis obviously needs to be tested. However, if proven true, this would be the highest number of distinct etacons delivered using a single electrode based on available literature. Preliminary test results seem to align with this hypothesis wherein the combination of different frequencies create a distinct 55 sensation of vibration. Note that these three parameters are only three out of eight controllable parameters introduced by the MES model. Hence, there is potential for delivery of much more complex signals. This potential only goes higher once spatial and temporal elements are added into the mix. However, it is unclear that there is a correlation between the maximum transferable sensations and etacons. For example, the human brain can perceive 2 million colors based on conservative estimates only using the three RGB cones in human eye retina. However, it is not feasible that 2million distinct etacons can be transferred to a human using only color in an unobtrusive and non-attention-consuming manner. The same concepts may apply to tactile M2HC as well. Hence, the question is, how many possible distinct etacons can be reliably relayed to humans without them becoming attention draining and obtrusive? Furthermore, there are a lot of questions about the ability of humans to learn and respond to tactile information. There is historical precedent that tactile input can be used as a method of reading for the visually impaired through the Braille system. However, it is not clear how fast and effective is it for a person with no prior exposure to the Braille system learn and retain tactile information. Despite the tremendous potential of the MES model, there are issues that need to be resolved moving forward. Even though the MES models was built on a simple concept of mixing and matching different electrotactile elements, the reality is that the number of possible combinations created is too high to be completely explored. Furthermore, in-house experiments have concluded that most of these parameters overlap with one another and do not create tangible specificity. As a result, it is safe to assume that not all of the variables chosen for MES model have a significant impact. Thus, the MES model needs to be simplified and optimized to be able to create differentiable signals efficiently. This assumption is in line with some of the 56 latest publications in the field of electrotactile stimulation wherein distinction is only attributed to frequency of electrotactile pulses and their amplitude [59], [61] rather than the MES with a full list of electrotactile parameters. The work in this thesis seeks to be independent from the stimulation amplitude as a parameter of distinction, so a new replacement for amplitude should be introduced. Furthermore, the techniques used in this chapter to determine the sensitivity of participants to a parameter from MES model did not provide consistent consensus about a global threshold that work for all participants. The progression of thresholds for each participant was evaluated anecdotally, resulting in thresholds that worked for each person but would not provide a picture of the general perception of thresholds for MES parameters. As a result, the work conducted in this chapter served as a great foundation for creating a global model for creating distinct electrotactile waveforms that can serve as etacons. The next chapters discus our further investigations for creating an ultimate model for creation of distinct electrotactile sensations. 57 Chapter 4: MEWS, A model for creating a library of static electrotactile waveforms Literature and Chapter 3 points to limitations for the realization of electrotactile M2HC using intensity and vibration frequency as parameters for creation of distinct sensations. The perceived intensity of electrotactile stimulation is usually invoked by changing electrotactile signal amplitude, pulse width, or pulse concentration which depend on skin condition and electrotactile stimulation signal frequency [112], [113]. The electrotactile waveform analog amplitude must be higher than a sensation threshold and lower than a discomfort threshold [79]. However, the sensation threshold and the perceived intensity levels are subject to change according to skin hydration, location, and type [57], [67] and will change over time with constant stimulation [114], [115]. As a result, curating a consistent level of perceived electrotactile intensity requires constant re-calibration [116]. Furthermore, perceived intensity increases with electrotactile stimulation frequency due to an increase in axon firing rates in peripheral nerves [117]. As a result, there is reason to doubt the effectiveness of perceived intensity as a means for creating distinct sensations. Currently, the range of reported distinct sensations using the sensation of vibration frequency is limited to 4-5 distinct sensations at electrotactile stimulation frequencies between 4Hz and 100Hz [59][61]. Furthermore, the results of nerve cuff stimulations and brain activity mapping in response to vibration frequency shown in [118], [119] suggest that the most consistent distinct artificial peripheral nerve stimulation sensations are to be found at stimulation frequencies below 50Hz further limiting the potential range of viable frequencies for creation of distinct sensations. To increase the usability of electrotactile stimulation as an M2HC, novel methods that allow to increase the number of distinct sensations are needed. 58 The work presented in this chapter triples the number of distinct sensations delivered to the brain using vibration frequency on a single electrode, alleviating the perceptual shortcomings of electrotactile stimulation for M2HC. This feat is performed by introducing a multi-dimensional electrotactile waveform model that provides a framework for creating a multitude of diverse electrotactile waveforms using an on-off-keying (OOK) style modulation that overlays two frequencies called beat and texture over a biphasic electrotactile pulse train. Furthermore, MEWS increases the number of evoked sensations by creating perceived intensity without changing electrotactile signal amplitude. The results from double blinded human trials, with 6440 test-points performed on 20 participants each, demonstrate that MEWS is capable of creating 18 reasonably distinct waveforms with at least 13 waveforms with an accuracy of 85% over a 500ms long stimulation window on a single electrode. 4.1 MEWS and Experimental Setup To enable electrotactile M2HC, a library of electrotactile waveforms that can act similar to an alphabet is necessary, where individual distinct waveforms within the library of electrotactile waveforms can be combined together to create complex etacons. The multi- dimensional electrotactile waveform (MEWS) model is designed to create such a library of electrotactile waveforms. To be comparable to the average English word reading speed, which is 300ms [120], the total length of the electrotactile waveform in MEWS was set to 500ms. The original concept of MEWS models was created by combining electrotactile communication and theories for electronics communication wherein a relatively high-frequency train of bi-phasic electrotactile pulses serve as carrier signal in an on-off keying (OOK) style modulation. The bi- phasic electrotactile pulses are the primary mean by which axons are activated. The carrier 59 frequency was selected to be 2kHz which is faster or equal to the refractory period of axons in the skin [121] and experiments conducted in Chapter 3. Modulation patterns can be used to carve unique waveforms from the carrier electrotactile pulse train and thus, create unique stimulation patterns that may create distinct sensation. Since there are no limits to how the modulation pattern is selected, it can be a combination of multiple frequencies. MEWS uses two modulating frequencies one in lower (~10Hz) and one in higher (~100Hz) frequencies working together to create distinct sensations. MEWS, as shown in Figure 4.1, modulates a carrier pulse train of bi-phasic electrotactile pulses at 2kHz with two sets of OOK style masks named texture and beat. The resulting electrotactile waveform consists of bursts of electrotactile pulses as shown in Figure 4.2. Texture frequency (TF) and beat frequency (BF) form the backbone of MEWS operations wherein TF controls a sensation of vibration and BF segments the waveform into mensural cycles similar to musical beats. Figure 4.1: A normalized representation of MEWS modulation technique in which a carrier high- frequency bi-phasic pulse train is modulated by two OOK style masks named Beat frequency and Texture frequency, resulting in a finalized electrotactile waveform. 60 TF and BF values are discreet due to the fact that electrotactile biphasic pulses are at a fixed frequency of 2kHz and TF and BF are selected so that no biphasic electrotactile pulse is clipped. The formulas that govern beat and texture frequency parameters are as follows: TF = 2𝑘𝐻𝑧⁄𝐾 × TDC⁄100 (1) TF = 𝐵𝐹⁄ 𝐵𝐷𝐶 × I (2) ( ) 100 where K and I are independent variables, BDC and TDC are the duty cycle (in percentages) of beat and texture masks, respectively, and BF, TF, K, and I only take integer values. Figure 4.2 depicts several examples of unique waveforms created with BF, TF, and TDC. Preliminary in-house experiments showed that BDC is inconsequential in terms of creating distinct sensations, as such is kept at 50%. 61 Figure 4.2: Several examples of electrotactile waveforms created using MEWS. (a) depicts a case with a low BF and high TF, effectively resulting in each beat to contain 13 textures. (b) depicts a case where beat and texture are very close together with each beat containing only two texture elements. (c) and (d) depict the same BF and TF values with the only difference being that the latter has a lower TDC which results in a lower number of electrotactile pulses being present in each texture. To fully define the optimum parameters of the MEWS model, it is important to determine what combination of beat and texture mask values result in distinct sensations. In other words, it is not clear what are the just-noticeable-difference (JND) values of BF, TF, and TDC and if these JNDs are independent of each other. For example, to know the JND for TF would mean that given a base TF, how much should a comparison TF be increased to create a noticeable difference. Extracting JNDs for beat and texture masks has significance because a quantized map of JNDs can be used as building blocks for crafting distinct electrotactile waveforms. To understand the JNDs for MEWS parameters, a series of four experiments were designed wherein Experiments I-III 62 studied JNDs for BF, TF, and Experiment IV tested a library of waveforms created from the insights of Experiment I-III. 4.2 Study set up 4.2.1 Setting MEWS’ parameter boundaries To make MEWS less abstract, it was limited with boundaries for the beat and texture masks. The parameters governing MEWS and their limitations are shown in Table I. The maximum TF was set to 200Hz which according to (1), results in each texture mask containing at least 10 biphasic electrotactile pulses (at TDC= 100%). The perceived intensity of an electrotactile stimulation is related to concentration of electrotactile pulses and as a result MEWS can hypothetically control perceived intensity of a stimulation using TDC. Figure 4.2(c) and 4.2(d) depict two cases where a texture frequency of 200Hz is set to TDC values of 50% and 20%, respectively. The maximum BF was set to 20Hz, an order of magnitude lower than maximum TF. Table 4.1: The MEWS parameters and their range of operation. Symbol Est Range Beat frequency BF 2Hz-20Hz Beat duty cycle BDC 0%-100% Texture frequency TF (BF/BDC)-200Hz Texture duty cycle TDC 0%-100% 4.2.2 Electrotactile stimulation hardware A hardware platform was designed and implemented to test MEWS. Inspired by [52], a ±80V voltage to current converter, shown in Figure 4.3 (a), was used to turn waveforms created by a PWM DAC with a frequency of 10MHz into electrotactile waveforms. As a result, a high pass and low pass filter with cutoff frequencies at 1Hz and 40kHz, respectively, were utilized to 63 condition the signal before being handed out to OP2. The capacitors Cf and Cin were film capacitors to ensure input equal response to positive and negative voltage directions (due to bi- phasic nature of input voltage, ceramic, tantalum and chemical capacitors did not provide equal biphasic charge characteristics). Op2 displays the resulting bi-phasic waveform on a 1kΩ resistor, Rf, creating a biphasic current that is replicated and reflected into an electrotactile electrode by a cascode current mirror. High voltage commercial PNP (ZTX558) and NPN (APT27HZTR) transistors were used for realizing the current mirror. However, since the transistors are not optimized as matching pairs, Ru and Rd were used to manually match the transistors. The electrotactile electrode was isolated from the current mirror through a 0.1µF capacitance as part of safety procedures to isolate the DC values between current mirror and the electrode [52]. A supply voltage of ±80V was used to ensure that the current mirrors can deliver the desired current without saturating the high impedance load (typically 10-200kΩ [14]) of the skin surface. Figure 4.3: The electrotactile stimulation hardware for electrotactile stimulation circuitry. 64 4.2.3 A multi-electrode approach to creating electrotactile sensations Electrotactile signals in form of electrochemical currents are conducted through the epidermis layer of the skin through sweat ducts and epithelial breaks. Because the density of sweat glands in the skin is 400-800 pores/mm2 [111], the current pathway through skin is spatially quantized and not uniform [93]. Under presence of electrotactile stimulation, especially if the skin is dehydrated, some of these pores may break down. A broken-down pore has a lower resistance and may conduct the majority of the charge that was meant for all of the pores. Figure 4.4 depicts this process under the presence of a large electrotactile electrode conducting a current of IIN. Because of majority of charge travelling through a singular pore, a region of over- stimulation will be created that will activate HTMR receptors and cause painful sensations [14]. Electrotactile current Iin Electrotactile Iin redirected to electrode breakdown Epidermis Breakdown of layer sweat ducts or Dermis epithelial layer breaks sweat ducts Over-stimulated and epithelial region breaks Figure 4.4: The interaction between a large electrotactile electrode and skin pores that are susceptible for breaking down. A broken-down pore has a lower resistance and would redirect the majority of electrotactile current activating HTMR pain receptors. Pore breakdown has traditionally been addressed by conditioning skin through hydration and conductive gels[14], [60]. However, skin conditioning would be a chore for users in daily usage of an MHF device, and the possibility of uncomfortable sensations with inadequate 65 conditioning would greatly diminish usability. There are multiple options to tackle the issue of hydration dependency and pore breakdowns. Options such as conductive polymers with non- linear resistance where the resistance increases with current. Another option was to artificially generate sweat in skin by injecting moisture or stimulating the skin to generate sweat. However, these options were scratched since forcing the skin to generate sweat will lead to discomfort and the other options were too complex. In this dissertation we predominantly try to avoid modulating the electrotactile signal amplitude to keep the signal levels at a comfortable level without breaking down the skin pores. We also decided to try a novel idea of forceful distribution of current into skin. This is achieved by decimating the input current IIN into smaller portions and forcing them into smaller electrodes called sub-electrodes (SE) in close proximity. Consequently, in the case that a breakdown happens, the current going through the broken-down pore will be much smaller than IIN while keeping the overall charge delivered to the skin region and potential created in epidermis layer similar to that of a singular electrode. Figure 4.5 depicts this concept with a hypothetical three- way distribution of IIN. 66 Electrotactile current Iin/3 Iin/3 Iin/3 Distributed electrotactile electrodes Over-stimulation is subdued Figure 4.5: The distributed electrotactile electrode model. The goal is to prevent a broken- down pore from conducting the majority of electrotactile current to prevent painful sensations. To implement this strategy, an array of small stainless-steel sub-electrodes (1.0mm G10 440C Stainless Steel Balls) were mounted on a PCB board as shown in Figure 4.3 and called a “distributed array”. The mounting of stainless-steel balls on a flat copper-based PCB was challenging due to complexities in mounting the electrodes on a PCB and impossible adhesion between stainless steel and copper through conventional soldering techniques. As a result, VIAs with the same diameter as the stainless-steel balls were created on the PCB array wherein the stainless-steel balls were placed. The balls were connected to the PCB copper trace on the backside of the array with a silver epoxy paste as shown in Figure 4.6. In the current distributed array, each electrode is composed of six individual sub-electrodes and surrounded by twelve ground electrodes. The distributed array was driven by a modified current mirror electrotactile driver to facilitate distributed electrotactile current between the sub- electrodes. The modified current driver is depicted in Figure 4.7. Each of the six transistor outputs are connected to an individual ball electrode in the center of distributed electrode. The distributed electrode driver was designed and implemented as shown in Figure 4.8. 67 Distributed electrode front Back Front Silver-epoxy-filled vias to Ground electrode ring front hold each stainless steel electrode on the Front side 2/3/2020 Figure 4.6: Fabricated distributed electrode array where each distributed electrode is composed of 6 individual electrodes (1.0mm G10 440C Stainless Steel Balls) in a hexagonal pattern that are surrounded by 12 ground electrodes. The stainless-steel balls were adhered and connected to the PCB board using silver-epoxy paste. +100V 500Ω Ru2 500Ω Ru1 Qp1 Qp2 Qp3 Qp4se1 Qp4se2 Qp4se6 Op-amp 2 0.1 µF 0.1 µF 0.1 µF Sub-Electrode se1 Sub-Electrode se2 PWM Sub-Electrode se6 … DAC + Qn5 and filter Rf - Qp5 1 kΩ 10 MΩ 10 MΩ 10 MΩ Qn3 Qn4se1 Qn4se2 Qn4se6 Qn1 Qn2 500Ω Rd2 500Ω Rd1 -100V Figure 4.7: A simplified presentation of modified electrotactile current waveform generator for enabling forced distribution of electrotactile currents unto distributed electrodes. Each output is connected to an individual electrode through an isolation capacitance. 68 Isolation capacitance and resistances (0.1µF and 10MΩ) Ru2 and RD2 Qn4seX and Qp4SEX transistors Figure 4.8: Hardware for distributed electrotactile hardware that facilitates distributed electrotactile stimulation. Through various human trials conducted, we were unable to prove or disprove with solid proof that this new electrode infrastructure helped prevent pore breakdown. The only case of success we had was an anecdotal case wherein the participant felt more comfortable with the circuit from Figure 4.7 and 4.8 activated rather than having all electrode fed by a single transistor. We were not able to replicate this condition. As such, this electrode structure requires further testing to prove or disprove its functionality. However, multiple participants expressed that the sensations created by Figure 4.6 were more comfortable compared to our electrodes from Figure 3.9. As such, throughout the rest of this thesis, we used the electrodes from Figure 4.7 as our primary electrotactile stimulation electrodes. Throughout the experiments covered in Chapters 4 and 5 we have done more than ~1000 stimulations for which we have recorded results using these electrodes. Hundreds of additional electrotactile stimulations were conducted that were not recorded. Throughout all these stimulations using the distributed electrode array, we have recorded zero complaints from participants regarding painful sensations. 69 4.2.3 Electrotactile stimulation software As shown on Figure 3.8, the PWM DAC signals are produced by an ATmega32U4 microcontroller controlled live through a USB host run by a Python script. In the case of MEWS, our waveform generation Python script reads a .csv file that lists all the waveforms we wish to test in terms of their MEWS parameters such as TF, BF, TDC, waveform analog amplitude, length, timing, etc. and translates them into a list of waveforms that can be transmitted and read by the ATmega32U4 microcontroller embedded state machine as requested by the user through user interface. The microcontroller code, Python script, and a set of example MEWS waveforms are available in [110]. 4.2.4 Participant recruitment and experiments’ procedure This study was approved by the Michigan State University Biomedical and Health Inst. Review Board (STUDY00004403). Twenty-two healthy adults from ages 18 to 55 were recruited voluntarily to participate in this Experiment. There was no scientific basis for exclusion of participants based on their gender, ethnicity, or skin color. All study trials were conducted by placing the distributed electrotactile array electrodes shown in Figure 4.6 on the participant’s non-dominant hand over the brachioradialis muscle. This placement ensured that the electrotactile electrodes are not in contact with any major nerve bundles. Prior to the placement of electrodes, the target skin region was sprayed two times with tap water using a standard water spray to ensure adequate hydration was present. Upon placement of distributed electrotactile array electrodes, a calibration step was conducted to identify the participant’s sensation and discomfort thresholds [61] based on a protocol under which we exposed participants to an increasing signal amplitude. This calibration data was used to evaluate the most comfortable yet 70 clear signal amplitude for the rest of the testing session. As part of the study set up, a complimentary Python code is being run on a screen in front of the participant to collect their experiments’ responses through a keyboard. The exploration of all MEWS parameters was facilitated by presenting the participant with a comparison of two electrotactile waveforms (each waveform was 500ms long) separated from one another with a 1s delay. Each comparison was unique and the waveforms forming the comparison were randomly selected from a library of waveforms. The experiment personnel and the participant were not aware of the order of waveforms being presented, and the responses of the participant are not visible until the end of experiment. As a result, all experiments conducted in this Chapter were double blinded. After presenting a comparison, the participants were asked to rate the comparison as “same”, “different”, or “unknown” using a keyboard input. The option “unknown” is used for cases where the participant was distracted, or due to external reasons could not decide. The “unknown” responses were discarded from the results, leaving only “same” and “different” responses. 4.3 Experiments I - III: Investigating BF, TF, and TDC 4.3.1 Experiment I: Understanding BF For the case of BF, a library of candidate waveforms was created with BF values spanning 1Hz to 20Hz (as noted in Table I) in a 500ms time window resulting in 11 total BF values. The 11 BF values can also be justified by the number of individual beats present in a 500ms window. The Table 4.2: The four corners used in BF trials defined by their corresponding TF and BDC values BDC= 30% BDC= 60% TF= 40Hz Corner 1 Corner 2 TF= 80Hz Corner 3 Corner 4 71 participant is stimulated with a combination of all comparisons extracted from the library of candidate waveforms. In order to observe the interaction of other parameters such as TF and BDC on the participant’s ability to distinguish BFs, four corners were tested, each with a different TF and BDC value. Each of the four variations is called a corner. The four corners used for the BF experiment are shown in Table 4.2 based on their TF and BDC values. For each comparison, the Python script picks a pair of BFs from a csv file of all possible BF values at random and presents them (with a specific TF and BDC determined by the corner) to the user and prompts for a decision on the distinction between the two waveforms by marking them as “same” or “different”. Once a comparison is presented to the participant, that comparison will not be presented again for the rest of the trial. Due to the imbalance between the number of “same” and “different” comparisons, the number of “same” comparisons were increased to be ~40% of the comparisons to create a sense of balance. As a result, in a trial for Experiment I, the participant would have to sift through 88 total comparisons, out of which 55 are “different” and 33 are “same”. For Experiment I, every participant was tested with the 4 corners from Table 4.2, and each corner was repeated for 3 trials. As a result, 3*4*88= 1056 comparisons were performed by each participant; all of which were performed in three sittings (with ample break time after every 88 comparisons per trial). It should be noted that since the length of all waveforms are 500ms, BF value of 1Hz was treated as “no beats”, i.e., when BF= 1, the only MEWS parameters that matter were TF and TDC. On the same note, BF= 2Hz was never used since in the context of a 500ms waveform, BF values of 1 and 2 were interchangeable. 72 4.3.2 Experiment II: Understanding TF For the case of TF, a library of candidate waveforms was created with TF spanning the values 2*BF to 200Hz, as dictated by (2), with an exponential distribution. Four corners were tested for TF Experiment, composed of BF values of 1, 4, 8, and 16Hz that give a range of low to high values. Experiment II was primarily focused on the interaction between BF and TF, so BDC and TDC were kept at constant values of 50% and 10%. The TF values used to create the library of candidates for TF Experiment is shown in Table III. In a similar manner to BF Experiment, comparisons composed of pairs of waveforms created based on TF values extracted from table III were presented to the participants and the number of “same” comparisons were artificially increased by 40%, resulting in a total of 320 comparisons performed by each participant. Each corner was performed in 1 trial and was not repeated unlike Experiment I. 4.3.3 Experiment III: TDC For the case of TDC, a library of candidate waveforms was created with three corners each with a different TF value of 20Hz, 40Hz, and 80Hz representing low, mid, and high TF values. TDC is not dependent on the beat, and as such BF and BDC were kept at 1Hz and 50%. The value of TDC spanned TF/2kHz*100 in compliance with (1) and 70% in an exponential fit curve. A maximum TDC of 70% was selected as opposed to 100% because a TDC value of 100% eliminates Table 4.3: The four corners used in TF trials defined by their corresponding TF and BDC values 73 texture. To ensure that the sensations created by an increase in TDC indeed felt as intensity, a sub-experiment was conducted wherein the participants were presented with two waveforms, with the second waveform randomly increasing in TDC or TF, if not same. The participants were then asked to comparatively identify the waveforms qualitatively and select any of the 4 options: intensity, frequency, length, and other to describe the difference. 4.4 Analysis techniques for MEWS experiments Six participants (2 female and 4 male) were recruited to perform BF, TF, and TDC Experiments. The participants had varying levels of experience with electrotactile stimulation. Each participant performed a total of 1440 comparisons for Experiments I-III. Each participant’s “same” response is recorded as 0 and a “different” response is recorded as 1 resulting in a numerical representation of each waveform’s performance when being compared with itself and other waveforms. Figure 4.9 shows examples of the average response of all participants in form of a comparison chart. Experiment I: BF corner 3 is depicted in comparison chart shown in Figure 4.9(a), where the BF frequencies tested are shown in blue blocks and the comparison results depict the relationship between two BFs. All of the responses on the main diagonal are comparisons of a waveform with itself and should ideally be a 0 (red) whereas all the rest of responses should be ideally a 1 (green) since they indicate difference between two waveforms. The sequence of how waveforms were presented in a comparison were randomized. For example, when comparing BF 1Hz and 3Hz, some participants receive the 1Hz stimulation followed by the 3Hz stimulation while others may receive the 3Hz stimulation followed by 1Hz stimulation, effectively eliminating any effect the order of stimulations may have on the average participant’s judgment. As a result, the comparison charts in Figure 4.9 are symmetrical around 74 the main diagonal even though only one comparison pair was performed. The responses of participants to corner 1 of TF trial is depicted in Figure 4.9(b) wherein the blue boxes represent TF values. Figure 4.9: Average of comparison chart for all participants for (a) BF trials corner 3 and (b) TF trials corner 1. Participant responses “different” and “same” are marked with a 1 and 0, respectively. The purpose of the comparison charts are to identify JNDs and the dependency of JNDs on experiment corners. As such, a subset of distinguishable waveforms with performance metrics above a certain threshold (to be determined later) were extracted from average comparison charts obtained for each experiment. Three performance metrics were used in this study: root mean square error (RMSe), minimum comparison score (Min), and accuracy. Given that the raw results from each comparison chart are converted into “correct” and “incorrect” responses (by simply replacing the diagonal elements with 1-diagonal), RMSe can be used as a measure of the performance level of a given subset of waveforms. Min is the lowest correctness score in a given 75 comparison chart. A low Min indicates that some waveforms in a selected subset are indistinguishable from one another regardless of RMSe score. Accuracy of a given comparison chart is calculated through confusion matrix analysis and simply states the chances of a person giving a correct answer in a dataset [100]. RMSe is the primary performance metric since it highlights outlier waveforms better, followed by Min and accuracy. In order to extract the best waveforms from a comparison chart for a given number of distinguishable waveforms (NDW), a clustering algorithm was used that divides the comparison chart into a number of NDW clusters and then select the best performing waveform within each cluster based on RMSe, Min, and accuracy. The raw comparison charts shown in Figure 4.9 are a non-Euclidean data structure that describe the relationship between waveforms i.e., ignoring the self-comparisons on the diagonal, the raw comparison charts can be viewed as a distance matrix. Hence, a Spectral clustering [122] algorithm, which is better suited for analysis of non-Euclidean data, was chosen to extract NDW clusters from the raw comparison chart. Numerous NDW by NDW matrix were formed by extracting every combination of waveforms from each cluster, and a final comparison matrix was selected that demonstrated the best RMSe, Min, and accuracy performance in order. For example, Figure 4.10 depicts the best performing NDW= 7 waveforms extracted from TF corner 1 from in Figure 4.9(b) and have an RMSe of 0.24, a Min of 0.33 and an accuracy of 85%. 76 Figure 4.10: The results of TF trials corner 1 reduced to the best performing waveforms at NDW of 7. 4.4.1 Experiment I-III results The purpose of BF Experiments was to validate if the relationship between different BFs is exponential as stated by Weber’s law, especially since BF values for this Experiment was on a linear scale as shown in Figure 4.9(a), and if the ability to differentiate BF is affected by other parameters such as TF and BDC. The results for BF Experiment were extracted and clustered (using Spectral clustering) into various NDWs. Since 11 values for BF were tested, NDW can take any value between 2 and 11. Figure 4.11(a) depicts the progression of BFs for NDWs of 3, 4, 6, and 11 on an exponential vertical scale for data from corner 3. The fit curve shown in Figure 4.11(a) is as expected on a linear scale whereas the best fit curve for NDW of 4 perfectly fits an exponential curve. As a result, even though the resolution for this Experiment is limited to a maximum NDW of 11, which is on a linear scale, at a lower NDW the selected BF values follows an exponential progression. 77 The results of all corners of TF Experiment were processed similarly to BF by performing spectral clustering and extracting the best set of waveforms for a given NDW based on performance metrics. The progression of TF frequencies for an NDW that produces an accuracy closest to 80% is shown in Figure 4.11(b) wherein the figure has an exponential vertical scale as well. As such, we can conclude that both BF and TF obey an exponential frequency progression, as predicted by Webber’s law. Furthermore, Figure 4.11(b) shows that the number of available NDWs for TF changes in reverse correlation to BF. For example, BF= 1 produces 7 NDWs for TF at accuracy of 80% whereas BF= 8 only produces 3 NDWs at accuracy of 80%. As a result, there are limitations to the flexibility created by combining TF and BF as BF gets higher. The progression with NDW of the three-performance metrics in TF trials corner 1 (BF= 1 i.e., no beat) is shown in Figure 4.12. As expected, the performance metrics are getting worse (RMSe increasing while accuracy and Min decreasing) as NDW increases. It should be noted that the maximum NDW for corner 1 is 15 which is the limit of Experiment II. However, because the performance metrics a 32 y = 0.3873e0.9968x R² = 0.9989 b 256 BF value at each waveforms (Hz) 16 y = 1.9182x - 1.3273 TF frequency (Hz) 8 R² = 0.9978 64 y = 1.6698e0.646x R² = 0.9835 4 NDW= 3 NDW= 4 16 BF= 1, NDW= 7 2 NDW= 6 BF= 4, NDW= 4 NDW= 11 BF= 8, NDW= 3 1 BF= 16, NDW= 2 4 1 2 3 4 5 6 7 8 9 10 11 1 2 3 4 5 6 7 0.5 Progression of BF steps Progression of TF steps for each corner and their NDW Figure 4.11: progression of (a) BF steps (for corner 3) at increasing NDWs and (b) TF steps for an NDW that produces an accuracy closet to 80% over multiple corners. As indicated by the fitting curve, (a) shows that at low NDWs BF values increase in an exponential rate. The results shown in (b) indicate that all corners of TF trials also display an exponential growth. 78 reached an unacceptable Min level at NDW= 8, we can assume that the selected resolution of maximum NDW of 15 was sufficient. Figure 4.12: The progression of RMSe, minimum element, and accuracy of best waveform sets for TF frequency, extracted with an increase in NDW. In both BF and TF trials a comparison between different corners was performed to analyze any underlying interdependency of MEWS model parameters. Figure 4.13(a) depicts the accuracy value of each of 4 corners of BF trials as NDW increases. Figure 4.13(b) shows the accuracy of comparisons the 4 corners of TF trials with an increase in NDW. The results depicted in Figure 4.13(a) shows that the differences between BF Experiment corners become more pronounced with an increase in NDW, with corner 3 and 2 showing best and worst performances respectively. The differences between corners in TF trials are more prominent as shown in Figure 4.13(b) with higher limitations on accuracy being exerted with an increase in the underlying BF. Experiments on TDC showed similar results to that of BF and TF trials in terms of exponential growth of steps for distinction of TDC but on a limited scale. Table 4.4 shows the accuracy values for NDWs of 2, 3, and 4 which indicate that TDC cannot create a high number of 79 distinct sensations especially compared with BF and TF. Furthermore, the performance of TDC for creation of intensity decreases as TF increases. Table 4.5 shows the qualitative responses of participants to TDC sub-experiment, indicating that the participants correctly associated an increase in TDC with an increase in the intensity of sensations perceived. 100 a 95 90 Accuracy % 85 80 corner1: TF= 40Hz, BDC= 30% 75 corner2: TF= 40Hz, BDC= 60% corner3: TF= 80Hz, BDC= 30% 70 corner4: TF= 80Hz, BDC= 60% 2 3 4 5 6 7 Number of distinguishable waveforms (NDW) b 100 90 80 Accuracy % 70 60 Corner1: BF= 1 Corner2: BF= 4 50 Corner3: BF= 8 Corner4: BF= 16 40 2 3 4 5 6 7 8 Number of distinguishable waveforms (NDW) Figure 4.13: Accuracy observed in association with an increase in NDW for all corners in (a) BF trials and (b) TF trials. In case of BF trials, corners 3 and 2 show the best and worst performance, respectively. In TF trials, overall accuracy drops with an increase in BF, indicating that lower BF values will result in higher distinction among texture frequencies. 80 Table 4.4: Recorded accuracy values for NDW values 2,3 and 4 over three corners denoted by TF values NDW TF= 20Hz TF= 40Hz TF= 80Hz 2 93 94 88 3 84 85 66 4 78 68 56 Table 4.5: the percentages of responses by participants given an electrotactile prompt with same value, increasing TF, and increasing TDC. The majority of participants correctly associated an increase in TF and TDC with an increase in frequency and intensity respectively 4.4.2 Experiment I-III analysis Experiment I explored aspects of MEWS model i.e., BF, TF, and TDC. Experiment I-III showed that BF, TF, and TDC can be effectively split into JNDs within their allowed range and that they are interdependent. However, the exact nature of the interdependency between MEWS parameters are not known. The growth of each parameter within their range was found to be exponential as expected. However, Figure 4.12 and 4.13 and TDC Experiments showed that higher density of pulses should be avoided to make the waveforms created easily distinguishable. The possible reasoning for this phenomenon may be attributed to the fact that higher pulse concentration will result in a higher baseline of intensity which may mask the more subtle changes in sensation of vibration frequency. Considering the limitations above, JNDs for each MEWS parameter can be selected based on the application in mind and its required performance metrics. 81 4.5 Experiment IV: Creating distinct waveforms 4.5.1 Experiment IV description Now that a framework for BF, TF, and TDC has been established by defining a growth rate and range of values that create distinct sensations, the next step is to create distinct sensations. The approach taken in this Chapter was to create a library of candidate waveforms with a combination of BF and TF JNDs and test all waveforms against one another to identify the best performing group of waveforms. As such, BF and TF signals were selected observing their JND growth rate extracted from Experiment I-III to construct a library of 23 candidate waveforms shown in Table 4.6. The library of candidate waveforms was selected primarily from BF and TF values. TDC was utilized to equate the intensity of the waveforms. The reason for keeping TDC mostly constant is that this study wishes to focus on evoked distinctions from vibration frequency and not intensity. The only cases where TDC does not have a constant value of 10% is in waveforms 1 and 2 that possess a condensed train of pulses creating high intensity. In these Table 4.6: List of 23 waveforms created based on insights from Experiments I-III with their respective BF, TF, and TDC values Candidate Candidate BF TF TDC BF TF TDC Waveform Waveform a 1 4 4 m 3 96 10 b 1 7 6 n 3 192 10 c 1 12 10 o 6 12 10 d 1 20 10 p 6 30 10 e 1 36 10 q 6 72 10 f 1 64 10 r 6 174 10 g 1 112 10 s 12 36 10 h 1 200 10 t 12 84 10 i 3 6 10 u 12 204 10 j 3 12 10 v 20 40 10 k 3 24 10 w 20 160 10 l 3 48 10 82 cases, TDC was used to reduce the perceived intensity of waveforms 1 and 2 in line with other waveforms. Fifteen participants (8 female and 7 male) with ages ranging 18 to 60 years old, and skin tones light to very dark were recruited. The waveforms were presented in comparison pairs similar to Experiment I-III and the participants were asked to classify the waveforms as “same”, “different”, or “unknown”. The total number of comparisons performed by each participant was 322 composed of 30% “same” and 70% “different” cases. The Experiment was done in 8 stimulation sets of 40 comparisons over two days with ample time for participants to rest in between each stimulation set. Experiment IV was performed over a span of two months. The raw results of the Experiment are shown on Figure 4.14. Figure 4.14: The average overall raw results from Experiment IV. 83 4.5.2 Experiment results and analysis An average comparison chart for all of the 23 waveforms was formed by taking the average response of all participants, an analysis protocol, similar to that of experiments I-III, wherein the best performing waveforms for a given number of distinguishable waveforms or NDW were extracted based on their performance metrics. RMSe and Average Lowest were utilized as performance metrics. Average Lowest is the average of lowest 12% of all comparisons in a given set of waveforms. Average Lowest was used instead of the previously defined Min because Average Lowest scales with the total number of waveforms involved and, as such, is a better metric of performance for large datasets. The extracted NDWs and their associated RMSe and Average Lowest are shown in Table 4.7. Table 4.7: Extracted figure of merit parameters from each NDW and their FoM acquired through a k-means clustering algorithm 84 The RMSe and Average Lowest values express a spectrum of performances without defining a specific threshold NDW value with an “acceptable” performance. As such, a figure of merit (FoM) needed to be constructed to reflect the best NDW value. The FoM includes elements from RMSe, Average Lowest, and distinguishability rank. Distinguishability Rank highlights how many comparisons in a selected set of waveforms have an average score below 0.5. Average comparison scores below 0.5 mean that a person has a lower than 50% chance of correctly identifying two waveforms in a given set of waveforms. Table 4.7 shows the distinguishability ranks for NDW spanning NDW= 4 to NDW= 21. To create a FoM from RMSe, Average Lowest, and distinguishability rank, a k-means clustering algorithm with 3 clusters was performed. The resulting clusters act as a FoM where each of the clusters can be interpreted as the difficulty level of a given NDW as shown on Table 4.7. As a result, NDW of 13 was identified as the cut-off point between easily distinguishable NDWS and more challenging ones. The waveforms that create NDW= 13 are shown in Figure 4.14. This set demonstrates an accuracy of 85.4% and a Distinguishability Rank of 8. Literature such as [59] and [61] demonstrate four to five distinct electrotactile vibration frequencies. Observing the BF and TF values selected for NDW= 13, five waveforms (a, c, e, g, and h) have no beats (BF= 1Hz) and create a similar sensation of vibration frequency as observed in literature. However, the rest of waveforms that possess BFs are unique contributions of this work. As such, our work in Chapter 4 increased the number possible sensations created with vibration frequency by 260% for NDW=13 and 360% for NDW= 18 while demonstrating an accuracy of 85.4% for NDW=13. A cross-examination between Figure 4.15 and Table 4.6 reveals that the number of distinguishable waveforms that can be extracted using BF declines with higher 85 BF values. Figure 4.16 demonstrates this fact clearly by displaying the clusters for NDW of 12 on a plot with TF and BF values. Figure 4.16 maps all the BF and TF values that we tested and clusters them into groups based on their similarity for achieving an NDW of 12. Figure 4.16 also exemplifies the fact that there is minimal cross-over between clusters formed by BF values of 1 (no BF), 3, and 6. This can be attributed to the fact that a lack of BF creates a sensation of vibration whereas BF values of 3 and 6 create a distinct sense of tapping. However, as BF increases the distinction between the sensation of tapping and vibration becomes more diminished. Therefore, care must be given to selecting the BF and TF values that will return the best performance metrics, and there are limitations on how many distinct sensations can be created with MEWS. a c e g h i k m n o q r v a 0.05 0.89 1 1 1 0.25 0.88 0.94 0.83 0.69 0.82 0.75 1 c 0.89 0.12 0.94 0.94 0.94 0.94 0.94 1 0.94 0.67 0.83 0.78 0.72 e 1 0.94 0.06 0.44 0.78 1 0.94 1 1 0.94 0.94 1 0.56 g 1 0.94 0.44 0.02 0.24 1 1 1 0.94 1 0.89 1 0.94 h 1 0.94 0.78 0.24 0.07 1 0.88 1 0.83 1 0.88 1 0.94 i 0.25 0.94 1 1 1 0.06 0.76 0.72 0.88 0.78 0.89 0.82 1 k 0.88 0.94 0.94 1 0.88 0.76 0.12 0.67 0.67 0.94 0.72 0.75 0.94 m 0.94 1 1 1 1 0.72 0.67 0.05 0.47 1 0.67 0.88 0.94 n 0.83 0.94 1 0.94 0.83 0.88 0.67 0.47 0.05 1 1 0.83 1 o 0.69 0.67 0.94 1 1 0.78 0.94 1 1 0.11 0.5 0.59 0.83 q 0.82 0.83 0.94 0.89 0.88 0.89 0.72 0.67 1 0.5 0.25 0.56 0.94 r 0.75 0.78 1 1 1 0.82 0.75 0.88 0.83 0.59 0.56 0.18 0.89 v 1 0.72 0.56 0.94 0.94 1 0.94 0.94 1 0.83 0.94 0.89 0.08 Figure 4.15: The best performing waveforms (denoted by their number as described in Table 6 at NDW of 13 and their comparison to one another. 86 Figure 4.16: A breakdown of the clusters that are formed for NDW of 12 and the corresponding BF and TF values that create each cluster. The X mark denotes the BF/TF pair that was selected to represent that cluster. 4.6 Summary and discussion of work 4.6.1 Summary Our work presented in Chapter 4 introduced MEWS, a model for creation of distinct sensation using on-off-keying modulation of two frequencies, beat and texture, imposed on a 2kHz carrier frequency of biphasic electrotactile pulses. Through multiple double blinded experiments, Chapter 4 showed that: intensity and a wide range of vibration frequency can be incited, using only vibration frequency, a set of 12 distinct sensations can be created with an accuracy of 85%. The importance of creating distinct sensation is that they can be used to convey information to the user in a human augmentation system by association of a meaning to distinct sensations. In this Chapter we performed two experiments to study the MEWS model. Experiment I was conducted to understand and quantize the parameters that govern the MEWS 87 model. Experiment II was conducted to create a list of distinguishable waveforms as proof of MEWS capabilities. For all experiments conducted, a comparison of two waveforms was presented to a participant at random and the response of the participant to the comparison was recorded. The comparison results were treated as a proximity map of the experiment waveforms and spectral clustering was used as the main analysis method. Experiment I showed that it is possible to reliably detect 4-6 levels of beat frequency in a range of 1-20 Hz. The exact value of beat frequency steps was determined in conjunction with other parameters of MEWS model. The same approach was taken to analyze texture frequency and texture duty cycle elements. Texture frequency ranged from 4Hz to 200Hz and created distinction levels with a high correlation to beat frequency. Texture duty cycle range from 0% to 70% and created 2-4 distinction levels. It was shown that the sensation created through TDC corresponded with intensity of the electrotactile waveform. Experiment I suggested that there is interdependency between MEWS parameters for creating distinct sensation and they cannot be treated independently. Based on literature, it was known that higher overall density of electrotactile pulses such as high beat and texture frequencies or high texture duty cycle will create a higher sense of intensity. However, Experiment I suggested that high intensity will interfere with the ability of participants to sense vibration frequencies. Furthermore, Experiment I showed that the range of intensities that can be incited with MEWS model at a carrier frequency of 2kHz are limited. The goal of Experiment II was to create a list of waveforms with high distinction using vibration frequency as the predominant factor of distinction. As such, a human trial was conducted on 15 participants wherein 23 waveforms created based on insights from Experiment I were tested. Similar to 88 Experiment I, all combinations of the 23 waveforms were presented to the participant in pairs and the participants were asked to differentiate the waveforms. Spectral clustering algorithm and a sorting technique based on specific performance metrics were performed to categorize the data into distinguishable waveforms ranging from 4 to 20. The determine what is the highest number of waveforms that can be easily differentiated, the waveforms were clustered into 3 categories based on their performance metric. The results demonstrated that MEWS is capable of creating at least 13 waveforms with an accuracy of 85% over a 500ms long stimulation window on a single distributed electrotactile array electrode. The distinct waveforms usually manifest as variations or combinations of the sensations tapping and vibrations depending on the BF/TF combination used. 4.6.2 Discussion and open challenges The MEWS model presented in this chapter presented opportunities for creating a library of diverse waveforms that is capable of creating 10-13 easily distinguishable waveforms that can be used as distinct etacons. Even though this number is a major improvement from the precedents from literature, 13 distinguishable etacons is not enough to create a “language”. Furthermore, the MEWS model lacks expandability. Due to our use of static Beat and Texture frequencies, the maximum number of waveforms that could be created with MEWS was already created and tested and cannot be expanded upon. As such, it may be prudent to utilize a model that can create waveforms that change over time. This new model can benefit from the findings of MEWS as well as create new waveforms that are not bound to a specific Beat and Texture frequency. Chapter 5 entails our endeavors for creating such a model. 89 Chapter 5: MOVES, A Linguistically-inspired Electrotactile Icon Generation Even though electrotactile stimulation is considered a method of peripheral nerve stimulation, complete recreation of sense of touch will require producing the idiosyncratic action potential firing pattern of the majority of tactile receptors involved which is currently technically impossible to achieve [77], [78]. As such, we focused our efforts in Chapter 4 on utilizing the accessible sensations of vibration and intensity. In the case of the sensation of vibration, the perceived frequency was correlated with electrotactile stimulation frequency which is defined by the frequency under which electrotactile charges are being injected into the skin [79]. Literature suggests the range of reported distinct sensations using the sensation of vibration frequency to be limited to 4-5 distinct sensations at electrotactile stimulation frequencies between 4Hz and 100Hz [59][61]. Our efforts in Chapter 4 expand our knowledge about this limitation with further detail as shown on Figure 4.12, wherein the figure shows a breakdown of performance of electrotactile frequencies ranging from 4Hz to 200Hz, depicting that the frequency range can be split to a maximum of 7 frequency steps. Furthermore, the results of nerve cuff stimulations and brain activity mapping in response to vibration frequency shown in [118], [119] suggest that the most consistent distinct artificial peripheral nerve stimulation sensations are to be found at stimulation frequencies below 50Hz. As a result, the potential range of viable frequencies for creation of distinct sensations is limited. Furthermore, we showed that using intensity as a means for creating distinct electrotactile sensations is limited since creating intensity with electrotactile signal amplitude risks creating painful sensations and the range of intensities created using frequency modulation is limited as we demonstrated in Chapter 4. 90 As such, in Chapter 5 we will seek novel alternative approaches to creating distinct electrotactile sensations by investigating the concept of electrotactile phonemes (etactemes). 5.1 Generation of distinct electrotactile sensations using etactemes In chapter 4, we developed MEWS, an OOK modulation technique for creating electrotactile waveforms from a train of high-frequency electrotactile pules. A key insight that we took from our work is the fact that if we forego using beats and instead opt to using a simplified version of MEWS as shown in Figure 5.1, we are capable of creating two classes of vibration sensations. The first class of sensations is that of a high-frequency vibration which can be created if the OOK texture frequency (TF) is higher than ~20Hz. The frequency of sensed vibration frequency is positively correlated with the TF. The example waveforms shown in Figure 5.2(b) and (d) are examples of high frequency vibration. The second class of sensations is often described as tapping which is usually associated with TF frequencies lower than 12Hz i.e., a continuous burst of electrotactile pulses as shown in Figure 5.2 (a) and (c). To simplify our terminology, high-frequency vibration sensation will henceforth be referred to as vibration sensation. As discussed in Chapter 4, the intensity of vibration and tapping sensations is associated with the concentration of electrotactile pulses which is achieved by increasing the texture mask duty cycle (TDC) or increasing TF. In Figure 5.2, we depict the visual contrast between high-intensity and low intensity sensations of tapping and vibration. 91 Figure 5.1: Electrotactile waveform creation model (MEWS) used in Chapter 5. This model is not limited to a 500ms timeframe and only uses texture frequency as an OOK mask. Figure 5.2: The sensation classes of tapping and high-frequency vibrations with various intensity levels. (a) depicts a low intensity (2ms long) tapping sensation. (b) depicts a low intensity high frequency vibration with a frequency of 100Hz. (c) and (d) depict high frequency counterparts of tapping and vibration sensations. 92 We hypothesize that vibration, tapping, and relative intensity can be used as etactemes (A.K.A. electrotactile phonemes) that can be progressively combined together to create truly diverse electrotactile sensations which can be turned into etacons (A.K.A electrotactile icons). We will prove the correctness of this hypothesis by creating a large set of demonstrably distinct electrotactile waveforms that supersede the number of etacons created by MEWS. However, there is no precedent for methods for best combining the etactemes to create a diverse library of etacons. The vibrotactile iconic communication literature suggest that there are no a priori methods for best combining vibrotactile phonemes [123], [124]. As such, we designed Experiment I to best understand etactemes to create a semi-heuristic method of combining them into functional etactemes. 5.2 Understanding Etactemes Experiment I was designed to specifically explore the sensations of tapping and vibration in mind. In the case of tapping sensation, we wished to understand: 1) what is the minimum length of an electrotactile pulse burst that is reliably detectable as a tapping sensation? 2) What is the minimum temporal spacing between two pulse bursts that allows a person to reliably distinguish them as two separate tapping sensations? As for the sensation of vibration, we wished to find out: 1) what is the minimum time window under which a person is able to reliably distinguish two different vibration frequencies? 2) What is the minimum detectable gap in an otherwise constant vibration sensation that is reliably detectable? 5.2.1 Experiment I setup All experiments presented in this thesis were approved by the Michigan State University Biomedical and Health Inst. Review Board (STUDY00004403). The hardware and software 93 infrastructure used in Chapter 5 are similar to that of Chapter 4. All of the experiments were performed using a single distributed electrotactile electrode array placed on the fore-arm (over the biracial muscle) of the non-dominant arm. Each Etacteme to be tested was generated by the MEWS model in Figure 5.1. Chaining etactemes back-to-back together was performed by having MEWS produce short bursts of waveforms back-to-back. Experiment I was composed of four tests, each seeking to study a certain aspect of tapping and vibration sensations to provide better understanding of these sensations. Since Experiment I was a perceptual task, 5 participants were recruited for each test (3 male and 2 females with ages ranging from 18 to 55). Each test is composed of a series of comparisons that participants makes between two electrotactile stimulations called verses. The two verses being compared contain key differences that we measure using participant responses. The first two tests performed seek to understand the behavior of tapping sensation. The verses used for tests one and two are shown in Figure 5.3(a) and (b), respectively. As shown in Figure 5.3(a), verse α contains an initial electrotactile burst (that creates a tapping sensation) with a variable length wherein the length varies from 0ms (No tapping sensation) to 10ms, while verse β does not contain an initial taping sensation. As a result, the comparison results between the two verses clarifies if the initial tapping sensation is perceptible. It should be noted that due to the nature of how MEWS produces electrotactile pulses at the frequency of 2kHz, the width of an electrotactile burst is quantized and can only be modified in increments of 1/2kHz= 0.5ms. Test 2, as shown in Figure 5.3(b), determines the minimum distance between two electrotactile pulse bursts by having verse α contain two initial electrotactile pulse bursts with a variable temporal distance in-between. As a result, the point at which the average participant can 94 differentiate between two verses would be the gap needed to differentiate the two tapping sensations. Each stimulation contains a follow-up vibration frequency to simulate an environment wherein such tapping sensations will be used in tandem with other sensations. Figure 5.3: Visualization of electrotactile waveforms used to understand tapping sensations. (a) depicts the verses used to understand the minimum viable electrotactile burst length with verse α having a variable electrotactile burst length and verse β without any burst. (b) depicts the verses used for understanding the minimum viable distance between two bursts with verse α having two electrotactile bursts with a variable distance between them while verse β only having a singular electrotactile pulse burst. Test 3 was conducted to understand the minimum time window under which a person is able to reliably distinguish two different vibration frequencies as depicted in Figure 5.4. Verses α and β start with vibration frequencies of 200Hz and 32Hz. However, in each comparison, the length of both 32Hz and 200hz vibration take a value within the range of 20ms to 300ms. The 95 point at which participants can reliably differentiate the two verses would be the length at which vibration sensations of 32Hz and 200Hz would be differentiable. Test 4 was conducted to determine the minimum detectable gap in an otherwise constant vibration sensation that is reliably detectable. In this test, Verse α was a constant vibration frequency while verse β was identical except for a gap of variable length inserted at 200ms mark. Figure 5.4: The verses used for test 3 wherein verse α and β start with an initial section with vibration frequencies of 200Hz and 32Hz respectively. The length of the initial section was randomly changed in the range of 20ms to 300ms to extract the minimum length under which α and β are distinguishable. In all tests, the participants performed the comparison for each verse twice, one with a follow-up vibration frequency of 32Hz and another with vibration frequency of 200Hz. Our stimulation protocol dictates that the order under which the verses are presented to a participant and also the progression of the variable (initial electrotactile burst width in this case) are presented at random and neither the participant nor the test administrator have any knowledge of the verses being presented, satisfying the criterion for a double-blinded experiment. 5.2.1 Experiment I results The results for all tests conducted under Experiment I are shown on Figure 5.5. The keyword “distinguishability” is used to represent the percentage of the population that correctly 96 performed the comparison between the given verses. The results from Test 1, shown on Figure 5.5(a), demonstrate that a minimum electrotactile pulse burst width of 2ms (four electrotactile pulses) are required for at least 80% of test subjects to reliably feel a tapping sensation. Figure 5.5(b) shows that a gap of at least 120ms between two pulse bursts sensations will ensure that the two tapping sensations are perceptible. The results of Test 3, shown on Figure 5.5(c) show that the two vibration frequencies of 32Hz and 200Hz are reliably distinguishable when they are stimulated for at least 350ms. Results of Test 4 shown on Fig. 5.5(c) show that the minimum noticeable gap in a vibration sensation is 120m. Figure 5.5: The results of Tests 1-4. (a) depicts the percentage of comparisons that correctly identified between the lack of an initial burst and an initial bursts width as shown on the x Axis. (b) depicts the distinguishability of two electrotactile pulse bursts as the gap between the two bursts grows. (c) depicts the length at which an average person should be stimulated with a vibration frequency for them to be able to distinguish between vibration frequencies of 32HZ and 200Hz. (d) depicts the minimum required gap length in a vibration frequency for the gap to be noticeable. 97 5.3 MOVES, a model for combining etactemes into variable distinct waveforms The data depicted on Figure 5.5 provided us with key insights for combining various tapping and vibration frequencies as etactemes as well as a gap in stimulation (henceforth known as a break). As such we created a model for varying electrotactile sensations using etactemes (MOVES). The two guiding principles of MOVES are: 1) Etactemes can be progressively combined together, given that they have enough time to establish themselves, by having each sufficiently long etacteme followed by another etacteme or a break. 2) Every electrotactile waveform can be split into units of 120ms because stimulating tapping sensation or a break in vibration sensation need a 120ms headroom to be noticeable and the two vibration frequencies 32Hz and 200Hz need at least 3*120ms headroom. It should be noted that it is debatable if a break in stimulation is itself also considered an etacteme or not, however since we cannot start or end a waveform with a break, we decided to give break its own designation. Based on results from Figure 5.5, we created seven classes of etactemes that can be utilized. The classes and their corresponding symbols are: • I: a single electrotactile pulse (usually 5ms long) followed by a 115ms of no stimulation meant to create a tapping etacteme • I*: Similar to I, however the electrotactile pulse is preceded by the 115ms gap • B: a 120ms break in stimulation • S1: A vibration sensation with texture frequency of 32Hz • S2: A vibration sensation with texture frequency of 80Hz • S3: A vibration sensation with texture frequency of 200Hz 98 • I2: a special case character wherein two electrotactile pulses are placed over a 360ms long slot with 175ms gap in-between It should be noted that S1 and S3 need to be at least 360ms long to be differentiable and S2 is only used in special cases where S1 and S3 are not present. An electrotactile waveform composed of etactemes can be described through the corresponding symbols of etactemes used. For example, “2xI, 3xS1” describes a waveform that is created with two tapping sensations, followed by a 360ms 32Hz vibration frequency as shown on Figure 5.6. Figure 5.6: The electrotactile waveform created by the etacteme descriptor “2xI, 3xS1” which means that the is created with two back-to-back tapping sensations, followed by a 360ms 32Hz vibration. The electrotactile pulses are a 2kHz bi-phasic pulse train with a standard amplitude of Amp. As a result, due to the large timescale present in this figure, they are seen a single golden column. Throughout this work, the length of every electrotactile waveform candidate created with MOVES is limited to 600ms. This was done to approximately make the length of etacons compatible with the average person’s reading speed of one word per 300ms. As such, if we are 99 to limit the minimum length of each etacteme to 120ms, a 600ms waveform created by MOVES will contain a maximum of 5 slots for etactemes or breaks. 5.3.1 Use of architypes for simplifying the combinations of etactemes in MOVES A way to prove the hypothesis of progressively combining etactemes creates distinct waveforms through MOVES is to create a library of all possible combinations of etacteme classes over 5 slots and doing a one-on-one comparison between each unique combination. However, the total number of unique combinations of etactemes is ~6^5= 7776. Realistically, it will be impossible to perform one-on-one comparisons between 7776 etactemes. Furthermore, there are redundant or imperceptible combinations that need to be eradicated from our library of possible etacons. Hence, we made a heuristic approach of creating etacons through the use of architypes. An architype is a verbal definition of a possible sensation that can be created by MOVES, i.e., an architype is a unique classifier for certain arrangement of etactemes. Through internal testing, we heuristically created 8 architypes that are described as: • A1: A sensation terminated by a 360ms vibration frequency • A2: A sensation terminated by a 360ms tapping sensation • A3: A sensation starting with a 360ms vibration frequency • A4: A sensation starting with a 360ms tapping sensation • A5: A sensation that begins or ends with a singular tapping sensation • A6: A sensation that begins or ends with a 240ms long vibration sensation • A7: A repeating pattern of sensation centered around a 120ms break in the middle • A8: Rising or falling vibration frequency and intensity 100 Each architype can contain a variety of members. For example, architype 1 contains 8 members that are as follows: • A1.1: I, B, 3x S1 • A1.2: I, B, 3x S3 • A1.3: 2x I; ,3x S1 (as seen in Figure 5.6) • A1.4: 2x I, 3x S3 • A1.5: 2xS3, 3xS1 • A1.6: 2xS1, 3xS3 • A1.7: S2, B, 3xS1 • A1.8: S2, B, 3xS3 5.3.2 Breaking down architypes The members of each architype are susceptible to being too similar to one another. As such, we conducted Experiment II to break down break down each architype to their most essential members by extracting the most distinguishable members of each architype. In order to break down each architype into its most distinct elements 5 participants (4 male and 1 female) were recruited. Experiment II was conducted by having the participants perform a comparison between all the members of each architype. The comparison was performed by randomly selecting two members of each architype as verse α and β, and then presenting them to the participant through an electrotactile stimulation. Our procedure dictated that α be presented first followed by a 1.2s delay followed by β followed by another 1.2s delay followed by α again. The time delays were utilized to ensure the participants can clearly identify α and β as separate entities. The αβα stimulation pattern was chosen so that the participant can still perform a 101 comparison just in case they missed or forgot the first α. The etacons served as α and β could be the “same” waveform or a “different” waveform. We artificially set the number of “same” cases to be 30% of the overall number of comparisons to ensure the responses of participants are not biased. The waveforms were selected at random and the test administrator and the participants did not know their order, creating a double-blinded study. Similar to Chapter 4, each participant’s “same” response is recorded as 0 and a “different” response is recorded as 1, resulting in a numerical representation of each waveform’s performance when being compared with itself and other waveforms. The raw results of participant comparisons for architype 1 members are depicted on Fig. 5.7 as a comparison chart. Our approach to breaking down an architype was by selecting a subset of distinguishable members that satisfying certain performance metrics. We used the spectral clustering technique used in Chapter 4 to extract best elements of a comparison chart with a given set of performance metrics. Two performance metrics were used in for breaking down architypes: root mean square error (RMSe) and minimum comparison score (Min). Given that the raw results from each comparison chart are converted into “correct” and “incorrect” responses (by simply replacing the diagonal elements with 1-diagonal) RMSe can be used as a measure of the performance level of a given subset of members. Min is the lowest graded comparison between any pair of waveforms in a given comparison chart. For example, in Figure 5.7(a) the Min value is 0.4. A low Min indicates that some waveforms in a selected subset are indistinguishable from one another regardless of RMSe score. Each architype was broken down by selecting the set of members that hade a Min score higher than 0.75 and the lowest achievable RMSe. Figure 5.7(b) shows the broken down and simplified version of architype 1. The same 102 process was performed on the rest of the architypes, with the finalized set of members extracted from all architypes presented in [110]. Figure 5.7: Our process of breaking down an architype. (a) depicts the raw comparison chart for Architype 1. (b) depicts the broken-down architype 1 wherein elements were selected based on their Min and RMSe performance metrics. 5.4 Creating a library of distinct electrotactile waveforms using MOVES Having broken down the architypes to their most essential members, we are one step closer to proving that out method of combining etactemes would create a larger set of electrotactile waveforms that can be used as etactemes compared to literature. After breaking down all architypes, we were left with a library of 31 waveforms. However, we expect that not all of the 31 waveforms will come across as distinct when compared with one-another. As such, we conducted Experiment III, which is a final round of eliminations using comparisons to remove non-distinct waveforms. However, performing such a comparison on 31 waveforms will lead to 103 31 × 30⁄2 = 465 unique comparisons. Furthermore, if we are to make sure that 30% total comparisons are ‘same’ cases, the number of comparisons goes up to 465⁄0.7 ≈ 655. Our previous experiences with human subjects in Chapters 3 and 5 showed that anecdotally, due to making comparisons being a tedious task, participants tend to lose focus/interest after ~50 comparisons without break or ~200 comparisons in a single session with breaks. As a result, preforming 665 comparisons posed significant difficulties in recruiting participants and retaining their attention, negatively impacting the legitimacy of our dataset. To alleviate this issue, we decided to only do comparison between architypes since we already have within-architype results from previous experiments performed in Section IV. Thus, the total number of comparisons to be made was reduced to 560, which is still too much. As a result, we split each round of comparisons between three test participants, giving each person a complimentary set of 187 comparisons. The participants complete their designated set of comparisons in two sittings with each sitting containing 47 comparisons, followed by a 5min rest period and another 47 comparisons. To realize this Experiment, we recruited 21 participants (13 male and 8 female) from ages ranging from 18 to 55 years old who possess a diverse range of skin tones and hairiness. The 21 recruited participants completed seven complete rounds of comparisons. Similar to previous Experiments, the comparisons presented to each participant were randomized and double blinded. The mean response of all participants is presented in Figure 5.8. 104 Figure 5.8: The raw comparison results for Experiment III between all 31 waveforms, performed over 7 complete comparison sets by 21 participants. The comparisons presented on Figure 5.8 Indicate that some pairs may be too similar to one another. For example, A4.1 and A3.6 are unique in their comparisons but are very similar to one another. As a result, a decision needs to be made to eliminate one of these waveforms. To facilitate this decision-making process, we created an algorithm to extract the best performing waveforms for a given number of distinguishable waveforms (NDW) were extracted based on metrics of success RMSe and Min similar to Experiment II. For example, for an NDW of 12, we extracted the 12 waveforms that achieve the highest Min and the lowest RMSe. Our algorithm looks for the best subset of comparison chart shown in Figure 5.8 that primarily has the highest Min score possible and secondarily to lowest RMSe. The metrics of success corresponding to the best subset of data presented in Figure 5.8 for each NDW are shown on Table 5.1. 105 Table 5.1: The metrics of success for each NDW and their corresponding additional metrics used for extracting FoM The NDWs and their corresponding metrics of success give us a good estimate of waveform performances, but its yet not clear how many of the waveforms are considered to be distinct enough to be considered as etacons. As a result, we extracted four additional performance metrics as shown on Table 5.1. The additional performance metrics include: 1) The 106 average of the lowest 10% of all comparisons performed (Min 10%) which is a better representation of performance of a NDW compared to just using Min because Min 10% scales with NDW. 2) Distinguishability (Dist) which highlights how many comparisons in a selected set of waveforms have an average score below 0.5 i.e., are we more likely misidentify the comparison on average? 3) Distinguishability percentage calculates percentage of total comparisons below 0.5 made for a given NDW. 4) Accuracy (Acc) of a given comparison chart for an NDW is calculated through confusion matrix analysis and simply states the chances of a person giving correct answers for a given NDW [100]. The performance of the overall results captured from each round of experiment (each round entailing the complimentary responses of three participants) are shown on Table 5.2 where it is shown that the Accuracy and RMSe scores of the rounds are similar and present low standard deviation (STD). Table 5.2: The performance of the overall results captured from each round of experiment that was conducted on three participants and the standard deviation (STD) measuring the consistency of their responses To determine the best NDW for an acceptable level of distinguishability between waveforms, we created a figure of merit (FoM) by labeling an NDW easy, medium, or hard representing the difficulty level of a given NDW. The FoM is created by running a k-means clustering algorithm with 3 clusters on all the metrics shown on Table 5.1. The results on Table 107 5.1 suggest that NDW of 24 is a great cut-off point. A list of all waveforms used in this study and the waveforms that we used for NDW are publicly accessible on [110]. This process is visualized through the connectivity graph shown on Figure 5.9 wherein a visual comparison is presented between the connectivity of full set of 31 waveforms and the extracted NDW= 24 waveforms. The figure shows that the extracted waveforms have a more unified distribution which means they will be more distinguishable from one-another. Figure 5.9: The connectivity map for the a) the 31 waveforms fed to Experiment III and b) the resulting waveforms for NDW= 24. 108 The process under which we created a large set of architypes and then reduced them to 24 distinct waveforms is depicted on Table 5.3 wherein the number of members of each architype and their internal performance are shown. The table depicts that the performance of each architype increases by simplifying them by taking out indistinguishable member first within each architype and then by comparing them with each other. Table 5.3: The progression of breaking down our architypes from the Initial Waveform Set to the Reduced Waveform Set (through Experiment II) which were created by reducing the number of architypes and the final set of waveforms created with NDW= 24 (through Experiment III) Initial Waveform Set Reduced Waveform Set NDW= 24 # Acc Min # Acc Min # Acc Min A1 8 79% 0.4 4 93% 0.8 3 98% 1 A2 16 84% 0 8 96% 0.7 7 99% 1 A3 10 72% 0 4 92% 0.8 3 89% 0.8 A4 12 69% 0 5 90% 0.8 5 92% 0.9 A5 7 69% 0.5 3 85% 0.8 2 88% 0.83 A6 3 72% 0.4 1 85% 1 1 85% N/A A7 8 67% 0.2 4 88% 0.7 2 79% 0.7 A8 3 66% 0.1 2 87% 0.8 1 87% N/A Total #: 67 31 24 5.5 Summary and discussion of work 5.5.1 Summary In Chapter 5 we investigate and created MOVES our model of progressively combining electrotactile phonemes (etactemes). MOVES progressively combines electrotactile sensations of vibration and tapping to create unique set of electrotactile waveforms. Through a heuristic approach, we came up with the concept of architypes to create a large set of electrotactile 109 waveforms through a heuristic approach using MOVES. Using our archetypal approach, we create 31 waveforms that have the potential to be distinct. We conducted a large human trial on 21 participants and we extracted 24 electrotactile waveforms that are easily distinguishable from one another. Since these waveforms are distinct form one another with an accuracy of 89%, they are eligible to be used as electrotactile icons (etacons). 5.5.2 Discussion and open challenges Given that we created 24 etacons using MOVES on a single distributed array electrode, with a stimulation widow of 600ms. Our results are a significant improvement over the literature values of 5 distinct waveforms and our work at Chapter 4 that acquired 12 distinct waveforms. As a result, we were successful in proving our hypothesis that combining etactemes will lead to creation of new and unique etacons. Furthermore, compared with precedents electrotactile literature, our approach to creating etacons has room for expansion through discovery and utilization of architypes. In this study, we only explored eight architypes, but we assume that there are more architypes to be explored. For example, except for architype 8, we did not extensively explore the effects of relative intensity. However, we are not certain about the total number of architypes that realistically can make a difference. Given that electrotactile stimulation is a method of creating action-potentials in the peripheral nervous system, the findings of this research can act as guidelines for future human augmentation efforts either through electrotactile stimulation, or invasive peripheral nervous system stimulation such as nerve cuffs and even neural probes on the brain. This is critical because a reliable method of creating distinct stimulation patterns on human nervous system is 110 essential for creating a pathway of information from machines to the human brain, especially if we do not have specialized access to each individual neuron. However, more research needs to be conducted in this field. We are not yet sure if MOVES will perform as good as shown in this Chapter if the participant was cognitively or physically engaged with another task or multi-tasking. Furthermore, we do not know how well a person can learn to associate an etacon with a piece of information as a part of M2HC and how good a person on the average is at retaining their association of etacons and information. 111 Chapter 6: Conclusions and future work 6.1 Summary In this dissertation we presented a novel approach for establishing electrotactile machine-to-human communication. As part of our efforts to understand electrotactile stimulation and ensure we are conducting impactful contributions to science, we conducted an extensive convergence research linking electrotactile stimulation, psychology, human physiology, neuroscience, and human augmentation technologies into a unified resource geared especially towards electrical engineers. We studied challenges that machine-to-human communication faces from an attention and obtrusiveness point of view and concluded that using electrotactile stimulation as a means of M2HC is beneficial for preserving attention in multi- tasking environments. As such, we pursued iconic electrotactile stimulation as a means for machine-to-human communication. Iconic communications require creation of very distinct electrotactile sensations. As a result, we experimented with various aspects of electrotactile stimulation through creating a software/hardware electrotactile rapid prototyping infrastructure and conducted human trials to better understand electrotactile stimulation. Our investigations into creating distinct electrotactile sensations lead to creation of MEWS, our first comprehensive model for creation of distinct electrotactile sensations. To prove MEWS capabilities, we conducted multiple human trials on 15 volunteering participants stimulating a total of ~6000 electrotactile comparisons. Based on our experience with MEWS, we next developed MOVES which utilizes dynamic electrotactile stimulation as opposed to the classical static stimulation techniques used in MEWS and electrotactile literature. Utilizing the linguistic concept of phonemes, MOVES defines etactemes (electrotactile phonemes) and is able to generate a 112 (potentially expandable) library of 24 distinct electrotactile waveforms, which we proved by conducting a human trial with ~5000 electrotactile stimulations. 6.2 Contributions The work presented in this dissertation bridges the gap between microelectronics, neural stimulation, and human augmentation worlds by contributing novel technologies for use of electrotactile stimulation for machine-to-human communication. These contributions include: • We created the first ever model to use a train of high-frequency electrotactile pulses and shape them into electrotactile waveforms through a multi-layer on-off-keying modulation, which allows control of sensation intensity without recalibration of amplitude and permits creation of multi-frequency highly distinguishable electrotactile waveforms At low frequency, the perceived intensity of electrotactile waveforms is a function of waveform frequency, because the charge transfer is in reverse correlation with electrotactile frequency, risking uncomfortable or feint sensations as charge transferred exits the bounds of sensation and pain thresholds. As a solution, we created MEWS, a unique method of creating electrotactile waveforms by modulating a high-frequency (2kHz) train of bi-phasic electrotactile pulses and shaping them through a multi-layer on-off-keying modulation (OOK). As such, changing the OOK mask frequencies will have the intended effect of changing electrotactile waveform frequency without significantly changing the electrotactile charge transferred to the skin. We conducted extensive human trials with over 6000 unique stimulations over 20 participants to prove MEWS is capable of creating a wide range of 13 distinct sensations at an accuracy of 85%. Furthermore, we were successful in creating 3 levels of relative intensity by controlling the concentration of electrotactile pulses being transferred to the skin without 113 actually changing electrotactile waveform amplitude. Our ability to freely change electrotactile waveform shape without having to constantly recalibrate to stay in goldilocks region between sensation and pain thresholds opens up a window of opportunity to experiment further with electrotactile sensations as a means for machine-to-human communication. • Designed a first ever model for creating varying electrotactile waveform generation (MOVES) based on linguistic concept of phonemes that pentuples the best recorded number of distinct electrotactile sensations in literature Electrotactile stimulation of distinct sensations for facilitating iconic communication is an emerging field of interest in human augmentation technologies. As such, a major contribution of this thesis is creations of MOVES, a novel model for creating electrotactile waveforms that vary over time. Utilizing MOVES, we were able to create 24 waveforms (which is five times higher than best works reported in literature) that had an accuracy of 89% and were identified as highly distinguishable electrotactile icons or etacons. This feat was achieved by combining the concept of electrotactile phonemes (etactemes) and electrotactile stimulations. The model was validated through a rigorous double-blinded human trials with more than 5000 electrotactile stimulations conducted on 21 participants. Furthermore, the 24 waveforms we created in this dissertation are not the true limit of MOVES, since our model can be expanded through discovery of more electrotactile stimulation architypes. • Created and made publicly available for other researchers a hardware/software infrastructure that allowed us to conduct more than 10000 unique electrotactile stimulations as part of our two yearlong human trial efforts 114 Prior to this work, there was no publicly available protocol for creating electrotactile stimulation and a pipeline for rapidly creating, testing, and analyzing electrotactile waveforms. As part of our investigations of electrotactile stimulations, we created a software/hardware ecosystem to address such needs. Among the recorded human trials, we demonstrably stimulated 40 participants with more than 10000 instances of electrotactile stimulation that are recorded with hundreds more un-recorded stimulation. We created a software infrastructure with over ~10000 lines of code (in C, MATLAB, and Python languages) and a robust electrotactile stimulation hardware that allow for rapid and safe prototyping of electrotactile waveforms and recording and analyzing participant responses on-the-go. Our complete software/hardware package is made available online for future researchers to access, experiment with, and expand upon. • We advanced the understanding of electrotactile stimulation through convergence of knowledge from psychology, engineering, neuroscience and human augmentation, resulting in a roadmap for this thesis work and related future work. Even though electrotactile stimulation has been around since the 60s, there has been a disconnect between science of electrophysiology and the engineering of electrotactile stimulation systems. For example, the physiology of hairy skin and specially how it interacts with electrotactile stimulation is rather un-known. Another example is that no work in literature has thoroughly investigated and classified the various ways electrotactile stimulation is being utilized for human augmentation. The work presented in this dissertation maps all the fields that pertain to electrotactile stimulation and human augmentation and offers a unique repository of 115 information that can help future electrical engineers to readily navigate various aspects of electrotactile stimulation for human augmentation. 6.3 Future work The work presented in this thesis established the foundations for large-scale machine-to- human communication through electrotactile stimulation of peripheral nervous system. However, there is much that needs to be done to see electrotactile as a means for machine-to- human communication to come to full fruition. The following are suggestions for future work: • Expand our library of etacons further beyond 24 members Compared with precedents electrotactile literature, our approach to creating etacons has room for expansion through discovery and utilization of architypes. In this study, we only explored eight architypes, but we assume that there are more architypes to be explored. For example, except for architype 8, we did not extensively explore the effects of relative intensity. However, we are not certain about the total number of architypes that realistically can make a difference. • Expand beyond electrotactile stimulation and into invasive peripheral nerve stimulation Given that electrotactile stimulation is a method of creating action-potentials in the peripheral nervous system, the findings of this research can act as guidelines for future human augmentation efforts either through electrotactile stimulation, or invasive peripheral nervous system stimulation such as nerve cuffs and even neural probes on the brain. This is critical because a reliable method of creating distinct stimulation patterns on human nervous system is essential for creating a pathway of information from machines to the human brain, especially if we do not have specialized access to each individual neuron. Furthermore, directly accessing 116 neurons may alleviate some risks associated with electrotactile stimulation such as skin condition sensitivity. • More experimentation needs to be performed with subjects under cognitive load and to evaluate a person’s ability to learn and memorize etacons We are not yet sure if our etacons perform as good as shown in this dissertation if the participant was cognitively or physically engaged with other tasks or multi-tasking. Furthermore, we do not know how well a person can learn to associate an etacon with a piece of information as a part of M2HC and how good a person on the average is at retaining their association of etacons and information. 117 REFERENCES 118 REFERENCES [1] R. Raisamo, I. Rakkolainen, P. Majaranta, K. Salminen, J. Rantala, and A. Farooq, “Human augmentation: Past, present and future,” International Journal of Human Computer Studies, vol. 131, pp. 131--143, 2019, doi: 10.1016/j.ijhcs.2019.05.008. [2] S. Uchiyama, K. Takemoto, K. Satoh, H. Yamamoto, and H. Tamura, “MR Platform: A basic body on which mixed reality applications are built,” Proceedings - International Symposium on Mixed and Augmented Reality, ISMAR 2002, pp. 248–256, 2002, doi: 10.1109/ISMAR.2002.1115095. [3] M. Turk, “Multimodal interaction: A review,” Pattern Recognition Letters, vol. 36, pp. 189- -195, 2014, doi: 10.1016/j.patrec.2013.07.003. [4] K. Aditya, P. Chacko, D. Kumari, D. Kumari, and S. Bilgaiyan, “Recent Trends in HCI: A survey on Data Glove, LEAP Motion and Microsoft Kinect,” 2018 IEEE International Conference on System, Computation, Automation and Networking, ICSCA 2018, pp. 1–5, 2018, doi: 10.1109/ICSCAN.2018.8541163. [5] L. Kibona and G. Mgaya, “Smartphones’ Effects on Academic Performance of Higher Learning Students,” Journal of Multidisciplinary Engineering Science and Technology, vol. 2, no. 4, pp. 3159–40, 2015, [Online]. Available: www.jmest.org [6] J. R. Anderson, Cognitive Psychology and Its Implications. Macmillan, 2005. [7] M. D.J., E. C., and K. E., “Cognitive load selectively influences the interruptive effect of pain on attention,” Pain, vol. 158, no. 10, pp. 2035–2041, 2017, doi: 10.1097/j.pain.0000000000001011 LK . [8] E. Lavie, Nilli and Hirst, Aleksandra and De Fockert, Jan W and Viding, “Load theory of selective attention and cognitive control,” Journal of Experimental Psychology: General, vol. 133, no. 3, p. 339, 2004. [9] D. Kahneman, Attention and effort. Citeseer, 1973. [10] D. Bawden and L. Robinson, “The dark side of information: Overload, anxiety and other paradoxes and pathologies,” Journal of Information Science, vol. 35, no. 2, pp. 180–191, 2009, doi: 10.1177/0165551508095781. [11] S. J. Bishop, “Trait anxiety and impoverished prefrontal control of attention,” Nature Neuroscience, vol. 12, no. 1, pp. 92–98, 2009, doi: 10.1038/nn.2242. [12] K. Sternberg, Robert J and Sternberg, Cognitive psychology. Nelson Education, 2016. 119 [13] S. “Claire” Lee and T. Starner, “BuzzWear: Alert Perception in Wearable Tactile Displays on the Wrist,” in Proceedings of the SIGCHI conference on Human factors in computing systems, 2010, pp. 433–442. doi: 10.1145/1753326.1753392. [14] K. A. Kaczmarek, J. G. Webster, P. Bach-y-Rita, and W. J. Tompkins, “Electrotactile and Vibrotactile Displays for Sensory Substitution Systems,” IEEE Transactions on Biomedical Engineering, vol. 38, no. 1, pp. 1–16, 1991, doi: 10.1109/10.68204. [15] P. Koch, Kristin and McLean, Judith and Segev, Ronen and Freed, Michael A and Berry II, Michael J and Balasubramanian, Vijay and Sterling, “How much the eye tells the brain,” Current Biology, vol. 16, no. 14, pp. 1428--1434, 2006. [16] M. Beygi, S. Mutlu, and B. Güçlü, “A microfabricated strain gauge array on polymer substrate for tactile neuroprostheses in rats,” Journal of Micromechanics and Microengineering, vol. 26, no. 8, 2016, doi: 10.1088/0960-1317/26/8/084006. [17] S. Ozturk et al., “Real-Time performance of a tactile neuroprosthesis on awake behaving rats,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 27, no. 5, pp. 1053–1062, 2019, doi: 10.1109/TNSRE.2019.2910320. [18] J. T. Ritt and S. Ching, “Neurocontrol: Methods, models and technologies for manipulating dynamics in the brain,” Proceedings of the American Control Conference, vol. 2015-July, pp. 3765–3780, 2015, doi: 10.1109/ACC.2015.7171915. [19] M. H. M. Kouhani, R. Luo, F. Madi, A. J. Weber, and W. Li, “A wireless, smartphone controlled, battery powered, head mounted light delivery system for optogenetic stimulation,” Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, vol. 2018-July, pp. 3366–3369, 2018, doi: 10.1109/EMBC.2018.8512936. [20] P. Wang, J. Zhang, J. Yu, S. Colin, and W. Feng, “Brain modulatory effects by low-intensity transcranial ultrasound stimulation (TUS): a systematic review on both animal and human studies,” Front Neurosci, vol. 13, p. 696, 2019. [21] B. Guse, P. Falkai, and T. Wobrock, “Cognitive effects of high-frequency repetitive transcranial magnetic stimulation a systematic review,” J Neural Transm, vol. 117, no. 1, pp. 105–122, 2010. [22] M. Zhong, C. Cywiak, A. C. Metto, X. Liu, C. Qian, and G. Pelled, “Multi-session delivery of synchronous rTMS and sensory stimulation induces long-term plasticity,” Brain Stimulation, vol. 14, no. 4, pp. 884–894, Jul. 2021, doi: 10.1016/j.brs.2021.05.005. [23] U. Ghafoor, S. Kim, and K. S. Hong, “Selectivity and longevity of peripheral-nerve and machine interfaces: A review,” Frontiers in Neurorobotics, vol. 11, no. OCT, pp. 1–21, 2017, doi: 10.3389/fnbot.2017.00059. 120 [24] C. Pasluosta, P. Kiele, and T. Stieglitz, “Paradigms for restoration of somatosensory feedback via stimulation of the peripheral nervous system,” Clinical Neurophysiology, vol. 129, no. 4, pp. 851–862, 2018, doi: 10.1016/j.clinph.2017.12.027. [25] H. P. Saal and S. J. Bensmaia, “Biomimetic approaches to bionic touch through a peripheral nerve interface,” Neuropsychologia, vol. 79, pp. 344–353, 2015, doi: 10.1016/j.neuropsychologia.2015.06.010. [26] L. V. Metman, J. S. Bellevich, S. M. Jones, M. D. Barber, and L. J. Streletz, “Topographic mapping of human motor cortex with transcranial magnetic stimulation: Homunculus revisited,” Brain Topography, vol. 6, no. 1, pp. 13–19, Sep. 1993, doi: 10.1007/BF01234122. [27] D. Tam, K. E. MacLean, J. McGrenere, and K. J. Kuchenbecker, “The design and field observation of a haptic notification system for timing awareness during oral presentations,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ’13, 2013, p. 1689. doi: 10.1145/2470654.2466223. [28] Y.-C. Liao, Y.-L. Chen, J.-Y. Lo, R.-H. Liang, L. Chan, and B.-Y. Chen, “EdgeVib,” in Proceedings of the 29th Annual Symposium on User Interface Software and Technology - UIST ’16, 2016, pp. 595–601. doi: 10.1145/2984511.2984522. [29] E. Lux, V. Dorner, M. T. Knierim, M. T. P. Adam, S. Helming, and C. Weinhardt, “Live biofeedback as a user interface design element: A review of the literature,” Communications of the Association for Information Systems, vol. 43, no. 1, pp. 257–296, 2018, doi: 10.17705/1CAIS.04318. [30] D. A. Mahns, N. M. Perkins, V. Sahai, L. Robinson, and M. J. Rowe, “Vibrotactile Frequency Discrimination in Human Hairy Skin,” Journal of Neurophysiology, vol. 95, no. 3, pp. 1442– 1450, 2005, doi: 10.1152/jn.00483.2005. [31] R. S. Johansson and J. R. Flanagan, “Coding and use of tactile signals from the fingertips in object manipulation tasks,” Nature Reviews Neuroscience, vol. 10, no. 5, pp. 345–359, 2009, doi: 10.1038/nrn2621. [32] K. J. Choi, Seungmoon and Kuchenbecker, “Vibrotactile display: Perception, technology, and applications,” Proceedings of the IEEE, vol. 101, no. 9, pp. 2093–2104, 2013. [33] V. Yem and H. Kajimoto, “Wearable tactile device using mechanical and electrical stimulation for fingertip interaction with virtual world,” in Proceedings - IEEE Virtual Reality, 2017, pp. 99–104. doi: 10.1109/VR.2017.7892236. [34] P. Svensson, U. Wijk, A. Björkman, and C. Antfolk, “A review of invasive and non-invasive sensory feedback in upper limb prostheses,” Expert Review of Medical Devices, vol. 14, no. 6, pp. 439–447, 2017, doi: 10.1080/17434440.2017.1332989. 121 [35] S. Panchanathan, S. Chakraborty, and T. McDaniel, “Social Interaction Assistant: A Person- Centered Approach to Enrich Social Interactions for Individuals with Visual Impairments,” IEEE Journal on Selected Topics in Signal Processing, vol. 10, no. 5, pp. 942–951, 2016, doi: 10.1109/JSTSP.2016.2543681. [36] K. Li, Y. Fang, Y. Zhou, and H. Liu, “Non-Invasive Stimulation-Based Tactile Sensation for Upper-Extremity Prosthesis: A Review,” IEEE Sensors Journal, vol. 17, no. 9, pp. 2625–2635, 2017, doi: 10.1109/JSEN.2017.2674965. [37] I. M. Koo, K. Jung, J. C. Koo, J. do Nam, Y. K. Lee, and H. R. Choi, “Development of soft- actuator-based wearable tactile display,” IEEE Transactions on Robotics, vol. 24, no. 3, pp. 549–558, 2008, doi: 10.1109/TRO.2008.921561. [38] M. Solomonow, J. Lyman, and A. Freedy, “Electrotactile two-point discrimination as a function of frequency, body site, laterality, and stimulation codes,” Annals of Biomedical Engineering, vol. 5, no. 1, pp. 47–60, 1977, doi: 10.1007/BF02409338. [39] K. Sato and S. Tachi, “Design of electrotactile stimulation to represent distribution of force vectors,” in 2010 IEEE Haptics Symposium, HAPTICS 2010, 2010, pp. 121–128. doi: 10.1109/HAPTIC.2010.5444666. [40] M. A. Eid and H. al Osman, “Affective Haptics: Current Research and Future Directions,” IEEE Access, vol. 4, no. c, pp. 26–40, 2016, doi: 10.1109/ACCESS.2015.2497316. [41] H. Kajimoto, “Design of cylindrical whole-hand haptic interface using electrocutaneous display,” in International Conference on Human Haptic Sensing and Touch Enabled Computer Applications, 2012, pp. 67–72. doi: 10.1007/978-3-642-31404-9_12. [42] P. L. Marcus and A. J. Fuglevand, “Perception of electrical and mechanical stimulation of the skin: implications for electrotactile feedback,” Journal of Neural Engineering, vol. 6, no. 6, 2009, doi: 10.1088/1741-2560/6/6/066008. [43] H. Kajimoto, H. Ando, and K. U. Kyung, “Haptic interaction: Perception, devices and applications,” Lecture Notes in Electrical Engineering, vol. 277, pp. 91–96, 2015, doi: 10.1007/978-4-431-55690-9. [44] A. M. Echenique and J. P. Graffigna, “Electrical stimulation of mechanoreceptors,” Journal of Physics: Conference Series, vol. 332, no. 1, p. 012044, 2011, doi: 10.1088/1742- 6596/332/1/012044. [45] M. Franceschi, L. Seminara, S. Dosen, M. Strbac, M. Valle, and D. Farina, “A System for Electrotactile Feedback Using Electronic Skin and Flexible Matrix Electrodes: Experimental Evaluation,” IEEE Transactions on Haptics, vol. 10, no. 2, pp. 162–172, 2017, doi: 10.1109/TOH.2016.2618377. 122 [46] V. G. Chouvardas, A. N. Miliou, and M. K. Hatalis, “Tactile displays: Overview and recent advances,” Displays, vol. 29, no. 3, pp. 185–194, 2008, doi: 10.1016/j.displa.2007.07.003. [47] M. Germani, M. Mengoni, and M. Peruzzini, “Electro-tactile device for material texture simulation,” International Journal of Advanced Manufacturing Technology, vol. 68, no. 9– 12, pp. 2185–2203, 2013, doi: 10.1007/s00170-013-4832-1. [48] C. J. Poletto and C. L. van Doren, “A high voltage, constant current stimulator for electrocutaneous stimulation through small electrodes,” IEEE Transactions on Biomedical Engineering, vol. 46, no. 8, pp. 929–936, 1999, doi: 10.1109/10.775402. [49] Y. Shen, J. Gregory, and N. Xi, “Stimulation current control for load-aware electrotactile haptic rendering: Modeling and simulation,” Robotics and Autonomous Systems, vol. 62, no. 1, pp. 81–89, 2014, doi: 10.1016/j.robot.2012.07.010. [50] H. Kajimoto, “Forehead Electro-tactile Display for Vision Substitution,” Proc. EuroHaptics, p. 11, 2006, [Online]. Available: http://lsc.univ- evry.fr/~eurohaptics/upload/cd/papers/f62.pdf%5Cnpapers2://publication/uuid/DA67D 1A2-09FE-47CF-BC2D-F12B7393E338 [51] S. Fukushima and H. Kajimoto, “Palm Touch Panel: Providing Touch Sensation Through the Device,” in Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces - ITS ’11, 2011, p. 79. doi: 10.1145/2076354.2076370. [52] K. A. Kaczmarek, K. M. Kramer, J. G. Webster, and R. G. Radwin, “A 16-Channel 8-Parameter Waveform Electrotactile Stimulation System,” IEEE Transactions on Biomedical Engineering, vol. 38, no. 10. pp. 933–943, 1991. doi: 10.1109/10.88439. [53] A. Higashiyama and G. B. Rollman, “Perceived Locus and Intensity of Electrocutaneous Stimulation,” IEEE Transactions on Biomedical Engineering, vol. 38, no. 7, pp. 679–686, 1991, doi: 10.1109/10.83569. [54] D. S. Pamungkas and W. Caesarendra, “Overview Electrotactile Feedback for Enhancing Human Computer Interface,” Journal of Physics: Conference Series, vol. 1007, no. 1, 2018, doi: 10.1088/1742-6596/1007/1/012001. [55] D. Pamungkas, “Enhancing human computer interaction with electrotactile feedback,” 2016. [Online]. Available: https://ro.uow.edu.au/theses/4811 [56] M. D’Alonzo, L. F. Engels, M. Controzzi, and C. Cipriani, “Electro-cutaneous stimulation on the palm elicits referred sensations on intact but not on amputated digits,” Journal of Neural Engineering, vol. 15, no. 1, 2018, doi: 10.1088/1741-2552/aa81e2. [57] K. A. Kaczmarek, J. G. Webster, and R. G. Radwin, “Maximal Dynamic Range Electrotactile Stimulation Waveforms,” IEEE Transactions on Biomedical Engineering, vol. 39, no. 7, pp. 701–715, 1992, doi: 10.1109/10.142645. 123 [58] K. A. Kaczmarek, “Electrotactile adaptation on the abdomen: Preliminary results,” IEEE Transactions on Rehabilitation Engineering, vol. 8, no. 4, pp. 499–505, 2000, doi: 10.1109/86.895953. [59] K. A. Kaczmarek, M. E. Tyler, U. O. Okpara, and S. J. Haase, “Interaction of Perceived Frequency and Intensity in Fingertip Electrotactile Stimulation: Dissimilarity Ratings and Multidimensional Scaling,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 25, no. 11, pp. 2067–2074, 2017, doi: 10.1109/TNSRE.2017.2702628. [60] M. Perovic et al., “Electrical stimulation of the forearm: A method for transmitting sensory signals from the artificial hand to the brain,” Journal of Automatic Control, vol. 21, no. 1, pp. 13–18, 2013, doi: 10.2298/JAC1301013P. [61] V. Kojić et al., “Integrated and flexible multichannel interface for electrotactile stimulation,” Journal of Neural Engineering, vol. 13, no. 4, p. 046014, 2016, doi: 10.1088/1741-2560/13/4/046014. [62] N. Kitamura, J. Chim, and N. Miki, “Electrotactile display using microfabricated micro- needle array,” Journal of Micromechanics and Microengineering, vol. 25, no. 2, 2015, doi: 10.1088/0960-1317/25/2/025016. [63] H. P. Saal, B. P. Delhaye, B. C. Rayhaun, and S. J. Bensmaia, “Simulating tactile signals from the whole hand with millisecond precision,” Proceedings of the National Academy of Sciences, vol. 114, no. 28, pp. E5693–E5702, 2017, doi: 10.1073/pnas.1704856114. [64] M. E. Altinsoy and S. Merchel, “Electrotactile feedback for handheld devices with touch screen and simulation of roughness,” IEEE Transactions on Haptics, vol. 5, no. 1, pp. 6–13, 2012, doi: 10.1109/TOH.2011.56. [65] R. Andrade et al., “Echo-house: Exploring a virtual environment by using echolocation,” in Proceedings of the 30th Australian Conference on Computer-Human Interaction, 2018, pp. 278--289. [66] D. J. Djozic, D. Bojanic, G. Krajoski, N. Popov, and V. Ilic, “Psychophysical characteristics of electrotactile stimulation: The impact of changes in stimulation pulse width and frequency on human perception,” 2015. doi: 10.1109/BIBE.2015.7367711. [67] H. Chen, L. Shuai, W. Zhu, and M. Miao, “An investigation of stimuli-current thresholds on the non-steady contact condition,” Industrial Robot: the international journal of robotics research and application, p. IR-11-2018-0230, 2019, doi: 10.1108/IR-11-2018-0230. [68] B. Stephens-Fripp, V. Sencadas, R. Mutlu, and G. Alici, “Reusable Flexible Concentric Electrodes Coated With a Conductive Graphene Ink for Electrotactile Stimulation,” Frontiers in Bioengineering and Biotechnology, vol. 6, no. December, pp. 1–9, 2018, doi: 10.3389/fbioe.2018.00179. 124 [69] J. B. van Erp and B. P. Self., “Tactile Displays for Orientation , Navigation and Communication in Air , Sea and Land Environments,” 2008. [70] J. Cornman, A. Akhtar, and T. Bretl, “A portable, arbitrary waveform, multichannel constant current electrotactile stimulator,” in International IEEE/EMBS Conference on Neural Engineering, NER, 2017, pp. 300–303. doi: 10.1109/NER.2017.8008350. [71] H. Kajimoto, “Electrotactile display with real-time impedance feedback using pulse width modulation,” IEEE Transactions on Haptics, vol. 5, no. 2, pp. 184--188, 2011. [72] A. Nau, M. Bach, and C. Fisher, “Clinical Tests of Ultra-Low Vision Used to Evaluate Rudimentary Visual Perceptions Enabled by the BrainPort Vision Device,” Translational Vision Science & Technology, vol. 2, no. 3, p. 1, 2013, doi: 10.1167/tvst.2.3.1. [73] C. K. Mortensen and M. A. Garenfeld, “Evaluation of Electrotactile Feedback Schemes in a Closed-Loop Myoelectric Prosthesis Master,” 2019. [74] O. Čakrt et al., “Balance rehabilitation therapy by tongue electrotactile biofeedback in patients with degenerative cerebellar disease,” NeuroRehabilitation, vol. 31, no. 4, pp. 429–434, 2012, doi: 10.3233/NRE-2012-00813. [75] K. A. Kaczmarek, K. Nammi, A. K. Agarwal, M. E. Tyler, S. J. Haase, and D. J. Beebe, “Polarity effect in electrovibration for tactile display,” IEEE Transactions on Biomedical Engineering, vol. 53, no. 10, pp. 2047–2054, 2006, doi: 10.1109/TBME.2006.881804. [76] H. Olausson, J. Wessberg, I. Morrison, F. McGlone, and Å. Vallbo, “The neurophysiology of unmyelinated tactile afferents,” Neuroscience and Biobehavioral Reviews, vol. 34, no. 2, pp. 185–191, 2010, doi: 10.1016/j.neubiorev.2008.09.011. [77] S. J. Bensmaia, D. J. Tyler, and S. Micera, “Restoration of sensory information via bionic hands,” Nature Biomedical Engineering, pp. 1--13, 2020, doi: 10.1038/s41551-020-00630- 8. [78] E. v. Okorokova, Q. He, and S. J. Bensmaia, “Biomimetic encoding model for restoring touch in bionic hands through a nerve interface,” Journal of Neural Engineering, vol. 15, no. 6, 2018, doi: 10.1088/1741-2552/aae398. [79] S. Parsnejad, J. Brascamp, G. Pelled, and A. Mason, “A review of electrotactile stimulation for machine-to-human communication,” IEEE Reviews in Biomedical Engineering [in review], 2022. [80] E. Sampaio, S. Maris, and P. Bach-y-Rita, “Brain plasticity: ‘Visual’ acuity of blind persons via the tongue,” Brain Research, vol. 908, no. 2, pp. 204–207, 2001, doi: 10.1016/S0006- 8993(01)02667-1. 125 [81] C. G. C. Barros, R. S. M. Bittar, and Y. Danilov, “Effects of electrotactile vestibular substitution on rehabilitation of patients with bilateral vestibular loss,” Neuroscience Letters, vol. 476, no. 3, pp. 123–126, 2010, doi: 10.1016/j.neulet.2010.04.012. [82] N. Asamura, T. Shinohara, Y. Tojo, N. Koshida, and H. Shinoda, “Necessary spatial resolution for realistic tactile feeling display,” Proceedings - IEEE International Conference on Robotics and Automation, vol. 2, no. February 2001, pp. 1851–1856, 2005, doi: 10.1109/robot.2001.932878. [83] H. Kajimoto, “Enlarged electro-tactile display with repeated structure,” in World Haptics Conference, 2011, pp. 575--579. [84] M. R. Schneider, R. Schmidt-Ullrich, and R. Paus, “The Hair Follicle as a Dynamic Miniorgan,” Current Biology, vol. 19, no. 3, pp. R132–R142, 2009, doi: 10.1016/j.cub.2008.12.005. [85] P. K. Farris, “SKIN ANATOMY AND PHYSIOLOGY,” nuskin. https://www.nuskin.com/en_ZA/corporate/company/science/skin_care_science/skin_an atomy_andphysiology.html (accessed Jun. 30, 2020). [86] A. I. Weber et al., “Spatial and temporal codes mediate the tactile perception of natural textures,” Proceedings of the National Academy of Sciences, vol. 110, no. 42, pp. 17107– 17112, 2013, doi: 10.1073/pnas.1305509110. [87] J. C. Forst et al., “Surface electrical stimulation to evoke referred sensation,” Journal of Rehabilitation Research and Development, vol. 52, no. 4, pp. 397–406, 2015, doi: 10.1682/jrrd.2014.05.0128. [88] C. Antfolk, M. D. Alonzo, B. Rosen, G. Lundborg, F. Sebelius, and C. Cipriani, “Sensory feedback in upper limb prosthetics,” Expert Rev Med Devices, vol. 10, no. 1, pp. 45--54, 2013. [89] V. E. Abraira and D. D. Ginty, “The sensory neurons of touch.,” Neuron, vol. 79, no. 4, pp. 618–39, 2013, doi: 10.1016/j.neuron.2013.07.051. [90] B. Y. A. F. Huxley and A. D. R. Stampfli, “Evidence for Saltatory Conduction in,” J. Physiol., vol. 108, no. 1946, pp. 315–339, 1948. [91] A. G. Konshina, P. v Dubovskii, and R. G. Efremov, “Structure and dynamics of cardiotoxins.,” 2012. doi: 10.2174/138920312803582960. [92] H. Kajimoto, N. Kawakami, T. Maeda, and S. Tachi, “Electro-Tactile Display with Tactile Primary Color Approach,” Graduate School of Information and Technology, The University of Tokyo, 2004. 126 [93] F. Lu et al., “Review of stratum corneum impedance measurement in non-invasive penetration application,” Biosensors (Basel), vol. 8, no. 2, 2018, doi: 10.3390/bios8020031. [94] D. R. McNeal, “Analysis of a Model for Excitation of Myelinated Nerve,” IEEE Transactions on Biomedical Engineering, vol. BME-23, no. 4, pp. 329–337, 1976, doi: 10.1109/TBME.1976.324593. [95] K. R. Foster and H. C. Lukaski, “Whole-body impedance--what does it measure?,” Am J Clin Nutr, vol. 64, no. 3, pp. 388S-396S, 1996, [Online]. Available: http://www.ncbi.nlm.nih.gov/pubmed/8780354 [96] A. Akhtar, B. Boyce, and T. Bretl, “The relationship between energy, phase charge, impedance, and perceived sensation in electrotactile stimulation,” in IEEE Haptics Symposium, HAPTICS, 2014, pp. 69–74. doi: 10.1109/HAPTICS.2014.6775435. [97] J. Caldwell, “A High-Voltage Bidirectional Current Source,” 2013. [98] S. Parsnejad, Y. Gtat, R. Aridi, J. W. Brascamp, and A. J. Mason, “Inciting high fidelity tactile sensations using a single electrotactile electrode pair,” in International IEEE/EMBS Conference on Neural Engineering, NER, May 2019, vol. 2019-March, pp. 778–781. doi: 10.1109/NER.2019.8717131. [99] S. Parsnejad, Y. Gtat, T.-Y. Lin, X. Liu, P. B. Lillehoj, and A. J. Mason, “Self-Ranging Thumb- sized Multichannel Electrochemical Instrument for Global Wearable Point-of-Care Sensing,” 2018. [100] S. Parsnejad, Y. Gtat, R. Aridi, J. W. Brascamp, and A. J. Mason, “Inciting high fidelity tactile sensations using a single electrotactile electrode pair,” in 9th International IEEE/EMBS Conference on Neural Engineering, NER, 2019, vol. 2019-March, pp. 778–781. doi: 10.1109/NER.2019.8717131. [101] S. Cheng and D. Zhang, “A wearable armband ‘iFeel’ for electrotactile stimulation,” in 2017 10th International Conference on Human System Interactions (HSI). IEEE, 2017, pp. 120– 124. doi: 10.1109/hsi.2017.8005012. [102] Covidien, “Kendall TM ECG Electrodes Product Data Sheet.” 2008. [103] J. T. Mortimer, “Motor Prostheses,” in Comprehensive Physiology, 2011. doi: 10.1002/cphy.cp010205. [104] M. M. Blattner, D. A. Sumikawa, and R. M. Greenberg, “Earcons and icons: Their structure and common design principles,” Human–Computer Interaction, vol. 4, no. 1, pp. 11–44, 1989. 127 [105] D. Stanke, T. Duente, and M. Rohs, “TactileWear: A Comparison of Electrotactile and Vibrotactile Feedback on the Wrist and Ring Finger,” 2020. doi: 10.1145/3419249.3420107. [106] D. Ternes and K. E. Maclean, “Designing Large Sets of Haptic Icons with Rhythm,” 2008. [107] M. Enriquez, K. MacLean, and C. Chita, “Haptic phonemes: basic building blocks of haptic communication,” in Proceedings of the 8th international conference on Multimodal interfaces, 2006, pp. 302–309. [108] K. A. Kaczmarek and S. J. Haase, “Pattern identification as a function of stimulation current on a fingertip-scanned electrotactile display,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 11, no. 3, pp. 269–275, 2003, doi: 10.1109/TNSRE.2003.816874. [109] K. A. Kaczmarek, J. G. Webster, P. Bach-y-Rita, and W. J. Tompkins, “Electrotactile and Vibrotactile Displays for Sensory Substitution Systems,” IEEE Transactions on Biomedical Engineering, vol. 38, no. 1, pp. 1–16, 1991, doi: 10.1109/10.68204. [110] Sina Parsnejad, “Sina Parsnejad Electrotactile PC interface code repository,” 2022. https://github.com/sina-parsnejad/Electrotactile-Stimulation (accessed Jun. 20, 2022). [111] P. A. Low, “Sweating,” in Primer on the Autonomic Nervous System, 2012. doi: 10.1016/B978-0-12-386525-0.00051-2. [112] J. Gregory, S. Tang, Y. Luo, and Y. Shen, “Bio-impedance identification of fingertip skin for enhancement of electro-tactile-based preference,” International Journal of Intelligent Robotics and Applications, vol. 1, no. 3, pp. 327–341, 2017, doi: 10.1007/s41315-016- 0010-6. [113] J. L. Dideriksen, I. U. Mercader, and S. Dosen, “Closed-loop Control using Electrotactile Feedback Encoded in Frequency and Pulse Width,” IEEE Transactions on Haptics, vol. 13, no. 4, pp. 818–824, 2020, doi: 10.1109/TOH.2020.2985962. [114] I. M., M. J., K. T., K. M., and Š. M., “Optimization of Semiautomated Calibration Algorithm of Multichannel Electrotactile Feedback for Myoelectric Hand Prosthesis,” Applied Bionics and Biomechanics, vol. 2019, 2019, doi: 10.1155/2019/9298758 LK. [115] S. Vizcay, P. Kourtesis, F. Argelaguet, C. Pacchierotti, and M. Marchal, “Electrotactile Feedback in Virtual Reality For Precise and Accurate Contact Rendering,” 2021, [Online]. Available: http://arxiv.org/abs/2102.00259 [116] A. Akhtar, J. Sombeck, B. Boyce, and T. Bretl, “Controlling sensation intensity for electrotactile stimulation in human-machine interfaces,” Science Robotics, vol. 3, no. 17, p. eaap9770, 2018, doi: 10.1126/scirobotics.aap9770. 128 [117] M. A. Muniak, S. Ray, S. S. Hsiao, J. F. Dammann, and S. J. Bensmaia, “The neural coding of stimulus intensity: Linking the population response of mechanoreceptive afferents with psychophysical behavior,” Journal of Neuroscience, vol. 27, no. 43, pp. 11687–11699, 2007, doi: 10.1523/JNEUROSCI.1486-07.2007. [118] E. L. Graczyk, B. P. Christie, Q. He, D. J. Tyler, and S. J. Bensmaia, “Frequency Shapes the Quality of Tactile Percepts Evoked Through Electrical Stimulation of the Nerves,” bioRxiv, pp. 2020--08, 2021, [Online]. Available: https://doi.org/10.1101/2020.08.24.263822 [119] J. D. Lieber and S. J. Bensmaia, “High-dimensional representation of texture in somatosensory cortex of primates,” Proc Natl Acad Sci U S A, vol. 116, no. 8, pp. 3268– 3277, 2019, doi: 10.1073/pnas.1818501116. [120] R. W. Bailey, Human performance engineering designing high quality professional user interfaces for computer products, applications and systems. Prentice-Hall, Inc., 1996. [121] J. P. Reilly, “Electrical Principles of Nerve and Muscle Function,” in Applied Bioelectricity: From Electrical Stimulation to Electropathology, New York, NY: Springer New York, 1998, pp. 73–104. doi: 10.1007/978-1-4612-1664-3_3. [122] U. von Luxburg, “A tutorial on spectral clustering,” Statistics and Computing, vol. 17, no. 4, pp. 395–416, 2007, doi: 10.1007/s11222-007-9033-z. [123] L. A. Jones, “Tactile communication systems: optimizing the display of information,” in Progress in Brain Research, vol. 192, Elsevier B.V., 2011, pp. 113–128. doi: 10.1016/B978- 0-444-53355-5.00008-7. [124] K. E. MacLean, “Foundations of transparency in tactile information design,” IEEE Transactions on Haptics, vol. 1, no. 2, pp. 84–95, 2008, doi: 10.1109/TOH.2008.20. 129