. it. ll 4.3%? .32.. .9 ‘ll' : F... by . .3, . nix _ 954.55 Hyman”. ‘ J3 , 4.5 p... x . an .M.» w... ‘ .3 H up... WN- w. ‘1 swfimx 0. gm. ‘- '0. E: s an.”- .1. _. {I is .. L «I‘llri Ac... 2.: 5.1. (v .’ILIZ.’Ir r .u 3-1 1"! 0‘ 21:; :9 a .0:- 1‘ a flux. THESIS IIIIIIIIIIIIIIIIIIIIIIIIIIIII II IIIIIIIIIIIIIIIIII 3 1293 0177183 I LIBRARY Michigan State University This is to certify that the dissertation entitled Analog CMOS Implementation of Artificial Neural Networks for Temporal Signal Learning presented by Hwa-Joon Oh has been accepted towards fulfillment of the requirements for Ph . D . degree in anineering Major professor MSUi: an Affirmative Action/Equal Opportunity Institution 0-12771 PLACE IN RETURN BOX to remove this checkout from your record. To AVOID FINES return on or before date due. MAY BE RECALLED with earlier due date if requested. DATE DUE DATE DUE DATE DUE use creme/mm“ Analog CMOS Implementation of Artificial Neural Networks for Temporal Signal Learning By Hwa-Joon 0h A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Department of Electrical Engineering 1996 ABSTRACT Analog CMOS Implementation of Artificial Neural Networks for Temporal Signal Learning By Hwa-Joon 0h A recurrent neural network with a recurrent learning rule is implemented using CMOS technology. We employ several building blocks for the implementation in- cluding a wide-range transconductance amplifier, a modified Gilbert multiplier, and a vector multiplier. A sigmoid function generator is designed using the wide-range trans-conductance amplifier. The output of the wide-range transconductance amplifier is current. To convert the current output to voltage, we use active resistors. The modified Gilbert multiplier and the vector multiplier are implemented using current bus and active resistors. Their four-quadrant and dot-product multiplications are verified through the PSPICE circuit simulations. We have developed a modified recurrent back-propagation learning rule for tempo— ral learning. Its forward instantaneous update scheme is suitable for analog hardware implementations. We have designed 4-neuron and 6-neuron recurrent neural network prototypes. We have implemented the neural network using standard CMOS circuits and verified their performance using extensive PSPICE circuit simulations. We have trained the two prototype neural networks to learn different state trajectories and the PSPICE circuit simulation shows that the recurrent neural network learn the temporal signals for reproduction and classification successfully. Finally, a two—dimensional scalable array configuration is designed for a large-scale implementation of fully connected recurrent neural network with learning. With the 2-D array configuration, the layout offers a simple and scalable VLSI architecture. Copyright by Hwa—Joon Oh 1996 To my parents, wife J eong-Hyeon, and son Eugene ACKNOWLEDGEMENTS I would like to express my sincere gratitude to my advisor Dr. Fathi M. Salam for his able guidance and constructive criticism throughout the course of my research. I would like to thank Dr. Hassan Khalil, Dr. Timothy Grotjohn, Dr. Gregory M. Wierzba, and Dr. Frank Hoppensteadt for being on my graduate thesis committee. Their critical comments and useful discussions aided in improving the quality of my thesis. My colleagues at the Circuits, Systems and Artificial Neural Networks Labora- tory, Department of Electrical Engineering, MSU, provided extensive help during the course of this research. I would like to sincerely thank Ammar Gharbi and Kay Hyounseok for their help, moral support and useful discussions. I also record my appreciation of the help rendered by the staff and the faculty of the Department of Electrical Engineering, MSU. I would like to express my sincere gratitude to my wife, Jeong-Hyeon and son, Eugene, for love, affection and patience. I would like to express my sincere thanks and appreciation to my parents and parents-in-law for their support and sacrifice. I also gratefully acknowledge the partial support during the research from ONR, the Michigan Research Excellence Fund (REF), and Innovating Computing Technol- ogy, Inc. (IC Tech). vi TABLE OF CONTENTS LIST OF TABLES LIST OF FIGURES 1 Introduction 2 Neural Network Models 2.1 The model of a Neuron .......................... 2.2 McCulloch and Pitts neural network model ............... 2.3 Feedforward neural network model ................... 2.3.1 The standard back-propagation algorithm ........... 2.3.2 The modified update law ..................... 2.3.3 Learning methods ......................... 2.4 Feedback neural network model ..................... 3 Analog CMOS Circuit Blocks 3.1 CMOS transistor ............................. 3.1.1 MOS Models ........................... 3.2 Transconductance amplifier ....................... 3.2.1 Active resistors/ Loads ...................... 3.2.2 Simple transconductance amplifier ................ 3.2.3 Wide range transconductance amplifier ............. 3.2.4 Sigmoid function generator .................... 3.3 The Modified Gilbert Multiplier ..................... 3.4 The Vector Multiplier ........................... 3.5 Implementation of the modified update law ............... 4 Learning Temporal Signals 4.1 Time-dependent recurrent back-propagation .............. 4.2 Classification of temporal trajectories .................. 4.3 Recurrent Back-propagation ....................... vii xi 11 13 16 20 22 23 27 27 28 32 33 38 44 47 50 53 55 58 59 61 63 4.4 Real-time recurrent learning ....................... 4.5 Hardware limitation of the real-time recurrent learning rule ...... 5 Implementation of the Modified Recurrent Back-propagation 5.1 The modified recurrent back-propagation rule ............. 5.2 Stability of the modified recurrent back-propagation .......... 5.3 Implementation of the modified recurrent back—propagation ...... 5.4 Simulation results of 4 neuron recurrent neural network ........ 5.4.1 A circular trajectory generation ................. 5.4.2 Trajectory recognition ...................... 5.5 Simulation results of 6 neuron recurrent neural network ........ 5.6 Hardware implementation considerations ................ 5.6.1 Hardware requirements of the modified recurrent back- propagation learning rule ..................... 5.6.2 Offset voltage adjustment .................... 5.6.3 The learning rate ......................... 5.6.4 Weight refresh ........................... 5.6.5 Temperature effects ........................ 5.6.6 Future work ............................ 6 2-Dimensional Scalable Array Configuration 6.1 Subcell design ............................... 6.2 Implementation of Floor Plan ...................... 7 Conclusion A SPICE parameters A.l The SPICE parameters: M0813 2.0 pm ORBIT ANALOG process A.2 The SPICE parameters: M0313 0.5 pm HP process .......... B PSPICE Input Files 3.1 Simple transconductance amplifier .................... 3.2 Wide range transconductance amplifier ................. B.3 Sigmoid function generator 1 ....................... B.4 Sigmoid function generator 2 ....................... B.5 Modified Gilbert Multiplier ....................... B.6 3-D Vector Multiplier ........................... C PSPICE simulation results with different temperature viii 66 69 72 72 75 82 86 86 101 106 114 114 115 115 116 116 117 119 119 121 129 132 132 133 134 134 135 136 138 140 142 144 C.1 Trajectory generation with 50°C ..................... 144 C.2 Trajectory generation with 0°C ..................... 145 C.3 Trajectory generation with 125°C and -50°C .............. 146 BIBLIOGRAPHY 147 ix 3.1 3.2 3.3 3.4 3.5 3.6 4.1 5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 6.1 LIST OF TABLES Resistance with different W / L ratios for Active resistors ....... 35 Resistance with different W/ L ratios for the cascaded active resistor . 37 The transistor sizes of the simple transconductance amplifier ..... 43 The transistor sizes of the wide range transconductance amplifier . . 45 The transistor sizes of the sigmoid function generator ......... 48 The transistor sizes of the modified Gilbert multiplier ......... 52 Hardware requirements of the real-time recurrent learning rule . . . . 70 The MATLAB simulation results .................... 78 The simulation results of the circular trajectory experiment ...... 89 The averaged weights in the learning phase from the PSPICE transient analysis: example 1 ............................ 92 The averaged weights in the learning phase from the PSPICE transient analysis: example 2 ............................ 92 The averaged weights in the learning phase from the PSPICE transient analysis: example 3 ............................ 92 The averaged weights in the learning phase from the PSPICE transient analysis: example 4 ............................ 92 The averaged values of the weight from the PSPICE transient analysis 104 The averaged values of the weight in the 6oneuron recurrent neural network with two output neurons .................... 108 The pin assignment of the 4 neuron recurrent neural network chip . . 128 LIST OF FIGURES 2.1 The structure of a classical neuron ................... 9 2.2 Schematic diagram of a McCulloch— Pitts neuron ............ 11 2.3 Single layer feedforward neural network. 5' means sigmoid function 13 2.4 Two layer feedforward neural network .................. 15 2.5 (a) The circuit of one unit 1n Hopfield model. (b)A three neuron Hop- field neural network. ........................... 24 3.1 The schematic diagram of N MOS and PMOS ............. 28 3.2 The convention of NMOS and PMOS, D:drain, stource, G:gate, szulk 29 3.3 The circuit diagram of active resistors using PMOS .......... 33 3.4 The PSPICE circuit simulation result: Active resistors with different W/ L ratios ................................ 35 3.5 The circuit diagram of the cascaded active resistors using PMOS . . . 36 3.6 The PSPICE circuit simulation result: The cascaded active resistors with different W/ L ratios ......................... 37 3.7 Simple transconductance amplifier .................... 38 3.8 Current output of the simple transconductance amplifier ....... 40 3.9 Simple transconductance amplifier with load resistors ......... 43 3.10 Simple transconductance amplifier: Output current (131 -132) as func- tion of V1 for different values of V2 ................... 44 3.11 The circuit diagram of the wide range transconductance amplifier . . 45 3.12 The PSPICE circuit simulation result: the wide range transconduc- tance amplifier, output current as function of V1 for several values of V2, Resistor = 100K!) .......................... 46 3.13 The PSPICE circuit simulation result: the wide range transconduc- tance amplifier, output current as function of V1 for several values of V2, Resistor = 1M0 ........................... 46 3.14 The circuit diagram of sigmoid function generator ........... 48 xi 3.15 3.16 3.17 3.18 3.19 3.20 3.21 5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 5.9 5.10 5.11 5.12 5.13 5.14 5.15 5.16 5.17 The PSPICE circuit simulation result: Sigmoid function generator with different W/ L ratio of the active resistors (M10 and M11) in PMOS when the bias voltage is 1.1V ...................... The PSPICE circuit simulation result: Sigmoid function generator with different bias voltages when (W/ L) of the active resister (M10 and M11) is (4pm/19pm,4pm/21pm) .................... The circuit diagram of the modified Gilbert multiplier ......... The PSPICE circuit simulation result: the modified Gilbert multiplier with V2 = V4 = 2.5V, V(5) is the output voltage ........... The circuit diagram of the 3-D Vector Multiplier ............ The PSPICE circuit simulation result: the 3-dimensional vector mul- tiplier ................................... The implementation of the differential equation ............ The MATLAB simulation result: the converged weights example . . . The MATLAB simulation result: the converged weights example . . . The MATLAB simulation result: the converged weights example . . . The MATLAB simulation result: the diverged weights example . . . . The recurrent neural network with four neurons and two inputs, S means the sigmoid function generator .................. The block diagram of recurrent neural network with the modified re— current back-propagation: four neurons and two inputs ........ The circular trajectory ....... ' ................... Example 1 (the first figure): Input and target signals ......... Example 1 (the second figure): Em, and the weights ......... Example 1 (the third figure): Test phase with the initial condition inside the circle .............................. Example 1 (the fourth figure): Test phase with the initial condition outside the circle ............................. Example 2 (the first figure): Input and target signals ......... Example 2 (the second figure): Em, and the weights ......... Example 2 (the third figure): Test phase with the initial condition inside the circle .............................. Example 2 (the fourth figure): Test phase with the initial condition outside the circle ............................. Example 3 (the first figure): Input and target signals ......... Example 3 (the second figure): Em, and the weights ......... xii 49 49 51 52 54 55 56 80 80 81 81 83 87 88 93 93 94 94 95 95 96 97 97 5.18 5.19 5.20 5.21 5.22 5.23 5.24 5.25 5.26 5.27 5.28 5.29 5.30 5.31 5.32 5.33 5.34 5.35 5.36 Example 3 (the third figure): Test phase with the initial condition inside the circle .............................. Example 3 (the fourth figure): Test phase with the initial condition outside the circle ............................. Example 4 (the first figure): Input and target signals ......... Example 4 (the second figure): Em, and the weights ......... Example 4 (the third figure): Test phase with the initial condition inside the circle .............................. Example 4 (the fourth figure): Test phase with the initial condition outside the circle ............................. Two state trajectories .......................... The PSPICE transient analysis: V(5), V(6), V(7), and V(1) are 3:1, 1:2, (11, and 311 of the recurrent neural network, respectively ...... PSPICE transient analysis: weight waveforms, w;,- = V(z' j ) ...... PSPICE transient analysis: the test result, V(5), V(6), and V(1) are the input 1, the input 2, and the actual output ............. PSPICE transient analysis: the test result of the trajectory 1, V(5), V(6), and V(1) is the input 1, the input 2, and the actual output PSPICE transient analysis: the test result of the trajectory 2, V(5), V(6), and V(1) is the input 1, the input 2, and the actual output The recurrent neural network with six neurons, two inputs, and one output, 5 means the sigmoid function generator ............ Two state trajectories: the two output neuron case on 6—neuron recur- rent neural network ............................ The PSPICE transient analysis: V(7) and V(8) are $1 and 2:2, V(9) and V(1) are d1 and y1, V(134) and V(2) are d2 and y; ........ PSPICE transient analysis: weight waveforms of the 6-neuron recur- rent neural network with two output neurons, ng = V(z' j ) ...... PSPICE transient analysis: the test result of the 6—neuron recurrent neural network, V(7), V(8), V(1), and V(2) are 2:1, 2:2, yl, and y;, respectively ................................ PSPICE transient analysis: the test result of the 6-neuron recurrent neural network, V(7), V(8), V(1), and V(2) are $1, m2, 311, and 31;, respectively ................................ PSPICE transient analysis: the test result of the 6-neuron recurrent neural network, V(7), V(8), V(1), and V(2) are x1, x2, y1, and y2, respectively ................................ xiii 98 99 99 100 103 103 104 105 105 107 109 110 110 111 111 112 5.37 Testing phase with the circular trajectory: when the circular trajecroty 5.38 6.1 6.2 6.3 6.4 6.5 6.6 6.7 C.1 C.2 C.3 0.4 05 CG is 500K H z ................................. 113 Testing phase with the circular trajectory: when the circular trajecroty is 250K H z ................................. 114 2—D array configuration elements .................... 120 The array structure of the recurrent neural network .......... 122 The MAGIC layout of the weight cell, 112;, ............... 123 The MAGIC layout of the weight cell, wgo ............... 123 The MAGIC layout of the error; & 2,- cell ............... 124 The MAGIC layout of the sigmoid,- & input,- cell ........... 124 The MAGIC layout of the recurrent neural network .......... 125 Input, output and target signals ..................... 147 Em, and the weights ........................... 147 Input, output and target signals ..................... 148 Em, and the weights ........................... 148 Weights hit the rail voltage ....................... 149 Weghts hit the rail voltage ........................ 149 xiv CHAPTER 1 Introduction The brain is a highly complex, nonlinear, and parallel computing structure. Neural networks have been motivated from its inception by recognizing that the brain com- putes in an entirely different way than the conventional digital computer. Artificial neural networks are machines that are designed to model the way in which the brain performs a particular task or function of interest [1][2] [3] [20]. An important feature of neural networks is that they perform useful computations through a process of learning. To achieve good performance, neural networks employ a massive interconnection of simple computing cells referred to as neurons. In neural networks, interneuron connection strengths known as synaptic weights are used to store the knowledge. The knowledge is acquired by the neural network through a learning process. The procedure used to perform the learning process is called a. learning algorithm. A learning algorithm modifies the synaptic weights of the neural network in order to get a desired design objective. Neural networks have some unique attributes [48]: nonlinearity, input-output map- ping, adaptivity, the ability to learn from their environment, the ability to generalize from weak assumptions, fault tolerance, VLSI implementability. There have been a lot of architectures and learning algorithms in artificial neural networks [4]. One important artificial neural network is the feedforward neural network and the back- propagation learning algorithm. Typically, the multilayer feedforward neural network consists of an input layer, one or more hidden layers of computation nodes, and an output layer of computation nodes. The error back-propagation process consists of two processes: a forward pass and a backward pass. In the forward pass, input pat- terns are applied to the network, and their effects propagate through the network. In the backward pass, the desired patterns are presented to the network and the out- puts of the network are compared with the desired ones. The errors are calculated and propagated backward through the connection weights to minimize the errors. The synaptic weights are adjusted so as to make the actual responses of the network move closer to the desired responses. Multilayer feedforward networks have been ap- plied successfully to solve some difficult and diverse problems by training them in a supervised manner [1] [20] [48] . The neural networks are usually implemented using electronic components or sim- ulated in software on a conventional computer. In the forward pass and the backward pass of neural network, high computational requirement is needed. To meet the com- putational requirement, analog hardware implementation of the neural networks will be an ideal medium for real time learning and processing. The advantages of using analog VLSI as technology medium for special-purpose neural network implementa- tions include the inherent parallelism of the operations, fast speed on learning and processing, the compact size, and low power consumption of the elements performing the computational functions [5]. Some of the traditional analog design requirements such as accurate absolute com- ponent values, device matching, precise time constants, etc., are not major concerns in neural networks. This is primarily because computation precision of individual neurons is not important [6]. For learning neural networks, the effects of mismatches and offsets in the analog components can be greatly compensated by the learning hardware parameters directly on the implemented neural networks. The learning performance of the neural networks may still be affected by the analog precision of the implemented learning function themselves, depending on the nature of the algo- rithm used. The list of general-purpose and special-purpose neural chips available presently is quite diverse, and still growing. The direct implementation of biological circuitry has been done by Mead’s group [7] [38]. They have found parallels between the behavior of subthreshold analog CMOS VLSI and biological neural circuitry. They have implemented a wide variety of neural VLSI systems that have been successfully constructed, including retinas[8], cochleas[9], and other biological systems. Intel’s ETAN N chip [10] is an analog model of 64 analog neurons and 10240 analog synapses. It uses EEPROM technology to store synaptic weights and the learning algorithm is performed at external general computers. The network can simultaneously compute the dot-product of a 64 element analog vector with a 64x64 synaptic array at a rate in excess of 1.3 billion interconnections per second. All elements of the computation are done in the analog domain and strictly in parallel. AT&T has built an ANNA (Analog Neural Network Arithmetic and logic unit) chip [11] that is a hybrid analog-digital neural network chip. The chip implements 4096 physical synapses. The resolution of the synaptic weights is 6 bits, and that of the states (input and output of the neuron) is 3 bits. The chip uses analog computation internally, but all input /output is digital. The chip can be reconfigured for synaptic weight and input vectors of varying dimension, namely, 64, 128, and 256. The ANNA chip is implemented for the application of high-speed optical character recognition with a total of 136,000 connections on a single chip. An analog VLSI neural network processor was designed and fabricated for com- munication receiver applications [12]. A channel equalizer was implemented with a neural chip configured as a three-layer perceptron network. The number of neurons in the input layer, two hidden layer, and the output layer is 8, 12, 12, and 1, re- spectively. The whole network includes a total of 252 synapse cells, and the input layer consists of the switched-capacitor analog delay circuits. The synapse cell is re- alized with a wide-range Gilbert multiplier circuit. The neuron circuit consists of a linear current-to-voltage converter and a sigmoid function generator with a control- lable voltage gain. Network training is performed by the modified Kalman filtering algorithm and the learning process is done in the companion DSP board which keeps the synaptic weight for the chip. The chip requires the refresh hardware to maintain the weight values. Storage of adjustable analog weights is one of the most important problems faced in analog implementation of artificial neural networks. The storage form can be analog: it would thus have the properties of an analog memory cell. A commonly used storage technique is based on storing charge across a capacitor. The storage capacitor will slowly discharge. For today’s conventional technologies, the storage time is of the order of several milliseconds. To store the weight control voltage for longer periods, the charge stored on the capacitance needs to be refreshed once every few milliseconds. The natural decay of a capacitor’s charge is one of the inherent limitations of the analog storage of weights. We have designed a neural network which has two feedforward neural networks in one chip [13]. The first feedforward neural network has its learning circuits. The second neural network does not have the learning circuit. The weight of the second feedforward neural network, is fed from the first neural network through voltage followers. If we keep applying the input-target pairs to the first feedforward neural network, there is no need to make an external interface circuit for weight refreshing in the second feedforward neural network. A recent paper [14] which employs a stochastic perturbative algorithm uses a local analog memory technique which does not require external storages. The weight refresh is performed in the background, and does not interfere with the continuous- time network operation. The back propagation algorithm is one of the most popular methods for the design of neural networks. A major limitation of the standard back propagation algorithm is its focus on approximating static mappings. This static input-output mapping is well suited for static information processing problems, however, it is not suitable for dynamic temporal information processing. Time-varying signals are important in many of the cognitive tasks encountered in practice, e.g., in vision, speech, control, and signal processing [15]. It is necessary to provide the neural network with dynamic properties that make it responsive to time-varying signals. For a neural network to be dynamic, it must be given memory [48]. One way in which a neural network can assume dynamic behavior is to make it recurrent, that is to build feedback into its design. In the recurrent neural network, connections are allowed in both ways between a pair of units, and even from a unit to itself. Analog recurrent neural network learning on time-varying signals offers a wide range of attractive applications, e.g., for process control, identification of dynamic system, and adaptive signal processing. Several versions of gradient descent algorithms for supervised learning in dynamic recurrent neural networks exist [16]. Pearlmutter [50] has derived learning proce- dure which can learn nonfixed point attractor. The technique is called the back- propagation through time. Pineda [54] has studied the fixed point learning proce- dure using the error back-propagation algorithm. It is called the recurrent back- propagation learning rule. An on-line [55], but computationally expensive, procedure for determining the derivatives of the states with respect to the weight parameters has been discovered and applied to the recurrent networks. It is called real time recurrent learning. The above mentioned algorithms are implemented in software. Their analog hardware implementations for real—time operations have currently not been demonstrated. We have deigned and fabricated feedforward neural networks with the modified update law [13] [28] [47]. The test results have demonstrated the successful operations of the feedforward neural networks [13] [28] [47] [61][62]. The circuit designs of the neural network at this dissertation are based on the test results of previous fabrications and test results. In this dissertation, recurrent artificial neural networks with on-chip learning cir- cuit are implemented using standard CMOS technology. We have modified the learn- ing rule from the time-dependent recurrent back-propagation learning rule. We avoid the backward integration and the memory requirement by modification. Its forward instantaneous update scheme is suitable for an analog hardware implementation. This is the first successful demonstration of the implementation of the gradient descent al- gorithm in the recurrent neural network. In order to implement the recurrent neural network and its learning algorithm, we employ a wide-range transconductance am- plifier, an active resistor, a modified Gilbert multiplier, and a vector multiplier. Contributions of this dissertation are as follows: 1. A simple sigmoid function generator is designed using a wide-range trans- conductance amplifier. Its current output is converted to voltage via active resistors in order to achieve voltage-to-voltage operations. The sigmoid function generator is simulated using the PSPICE circuit simulator. Its characteristics are shown in this dissertation. 2. The modified Gilbert multiplier is designed and implemented as a CMOS circuit. Its voltage-to-voltage operation is achieved through active resistors. Its four- quadrant multiplication is verified in this dissertation. The performance of the multiplier using the PSPICE circuit simulator is shown in this dissertation . The vector multiplier is designed using the modified Gilbert multiplier. High dimensional multiplier is implemented through simple current summing nodes. The performance of the vector multiplier is presented using the PSPICE circuit simulator. . The modified recurrent back-propagation rule is presented. Its continuous-time modified update law from the time-dependent recurrent back-propagation learn- ing rule is shown in this dissertation. The modified algorithm is suitable for the analog CMOS implementation. . The MATLAB simulations show the conditions for the stable operation of the network. The damping factors in the modified equations are important for the stability of the recurrent neural network. We show the range of the parameters 'which ensure the successful learning in the learning phase. . The modified recurrent back-propagation rule is implemented with on-chip learning. Its learning rule is implemented in CMOS circuit and the PSPICE simulations demonstrate its learning capability. We have demonstrated the learning capability by training a circular state trajectory. In the circular trajec- tory generation experiment, the recurrent neural network can generate a limit cycle. We have performed experiments with different parameters for successful learning. At the trajectory recognition experiment, the simulation result shows that the recurrent neural network can distinguish different trajectories. . A two-dimensional scalable array configuration is designed for large-scale im- plementation. With the 2-D array configuration, the layout offers a simple and scalable VLSI architecture. We have designed 4-neuron recurrent neural net- work and its number of input and target is easily configurable. We show the subcell design, the floor plan, and its layout in this dissertation. This dissertation is divided into 7 chapters. Chapter 2 reviews the model of neuron and three artificial neural network models. McCulloch and Pitts model, a feedfor- ward neural network model, and Hopfield neural network model are investigated. In Chapter 3, basic CMOS circuits and analog neural subcircuits are discussed for the implementations of the artificial neural networks. Active resistors are designed to convert the current output to the voltage output and its characteristics are shown using the PSPICE circuit simulator. The designs of a transconductance amplifier, a modified Gilbert multiplier, and a vector multiplier are demonstrated. Chapter 4 reviews the algorithms for the temporal signal learning. 'We have reviewed the Pearlmutter’s algorithm, the classification of temporal trajectories, the Pineda’s al- gorithm, and the William and Zipper’s algorithm. The hardware requirement for the William and Zipper’s algorithm are investigated in this chapter. In Chapter 5, the modified back-propagation learning rule is presented. Its MATLAB simulations for stability are shown in this chapter and its implementation is explained. We have demonstrated that the learning scheme successfully learns the temporal signals by generating the circular trajectory and recognizing the different state trajectories in the recurrent neural networks. In Chapter 6, we present the 2-D array configuration of the modified back-propagation learning rule. With the 2-D array configuration, the layout offers a simple and scalable VLSI architecture. We show the layout and pin-configuration of the chip. Chapter 7 summarizes the conclusions of this research work. CHAPTER 2 Neural Network Models 2.1 The model of a Neuron The basic unit in the nervous system is specialized cell which is called neuron. A typical view of neuron is shown in Figure 2.1. Fromotheraxons Sy11apse\> s W Deodritec / :I‘ © . .... Axonlnllock Cell Body orSonn Synapse/ Figure 2.1. The structure of a classical neuron Most neurons share certain structural features that make it possible to distinguish four regions of the cell: the cell body or soma, the dendrites, the axon, and synapse. 10 The individual nerve cell transmits nerve impulses over a single long fiber (the axon) and receives them over numerous short fibers (the dendrites) [17]. The functioning of the brain depends on the flow of information through elaborate circuits consisting of networks of neurons. Information is transferred from one cell to another at specialized points of contact: the synapses. A typical neuron may have anywhere from 1,000 to 10,000 synapses and may receive information from something like 1,000 other neurons. Synapses are most often made between the axon of one cell and the dendrite of another. These synaptic contacts are the primary information processing elements in neural systems. The transmission of a signal from one cell to another cell at a synapse is a complex chemical process. At a synapse the axon usually enlarges to form a terminal button, which is the information-delivering part of the junction. The terminal button con- tains tiny spherical structures called synaptic vesicles, each of which can hold several thousand molecules of chemical transmitter. On the arrival of a nerve impulse at the terminal button, some of the vesicles discharge their contents into the narrow cleft that separate the button from the membrane of another cell’s dendrite, which is designed to receive the chemical message. Hence the information is relayed from one neuron to another by means of a transmitter. The “firing” of a neuron-the gener- ation of nerve impulses-reflects the activation of hundreds of synapses by impinging neurons. Some synapses are excitatory in that they tend to promote firing, whereas others are inhibitory and so are capable of canceling signals that otherwise would excite a neuron to fire. Equipped with a tree of filamentary dendrites, the neuron body aggregates synap- tic inputs from other neurons. The input currents are integrated by the capacitance of the cell body until a critical threshold potential is reached, at which point an out- put is generated in the form of a nerve pulse. This output pulse propagates down the axon, which ends in a tree of synaptic contacts to the dendrites of other neurons. 11 2.2 McCulloch and Pitts neural network model Due to the complexity and diversity of the properties of biological neurons, the task of compressing their complicated characteristics into a model is extremely difficult. Neural computational elements are nonlinear, and typically these are analog in nature. Many researchers have tried to model neural network systems with current knowledge of biological neurons since it is still under research. McCulloch and Pitts [18] proposed a simple model of a neuron as a binary thresh- old unit. Specifically, the model neuron computes a weighted sum of its inputs from other units, and outputs a one or a zero according to whether this sum is above or below a certain threshold: "10+ 1) = 5(2 w,.-n.~(t) + 91) (2-1) The diagram is shown in Figure 2.2. Here n, is either 1 or 0, and represents the state Figure 2.2. Schematic diagram of a McCulloch-Pitts neuron of neuron j as firing or not firing respectively. Time t is taken as discrete, with one time unit elapsing per processing step. 5 is the unit step function, or Heaviside 12 function: 1 if a: Z 0; 3(1') = (2.2) 0 otherwise. The weight 111,; represents the strength of the synapse connecting neuron i to neuron j. It can be positive or negative corresponding to an excitatory or inhibitory synapse respectively. It is zero if there is no synapse between j and i. The cell specific parameters 0,- is the threshold value for unit j; the weighted sum of inputs must reach or exceed the threshold for the neuron to fire. Though it is a simple model, a McCulloch-Pitts neuron is computationally a pow— erful device. McCulloch and Pitts proved that a synchronous assembly of such neurons is capable in principle of universal computation for suitably chosen weights wji. This means that it can perform any computation that an ordinary computer can, though not necessarily so rapidly or conveniently. Real neurons involve many complications omitted from this simple description. The most significant ones include [1]: 0 Real neurons are often not even approximately threshold device as described above. Instead they respond to their input in a continuous way. However, the nonlinear relationship between the input and the output of a cell is a universal feature. The hypothesis is that it is the nonlinearity that is essential, not its specific form. 0 Many real cells also perform a nonlinear summation of their inputs. There can even be significant logical processing (e.g., AND, OR, NOT) within the dendritic tree. 0 A real neuron produces a sequence of pulses, not a simple output level. Rep- resenting the firing rate by a single number like 11,-, ignores much information that might be carried by such a pulse sequence. 13 o Neurons do not all have the same fixed delay (t —» t+1). Nor are they updated synchronously by a central clock. In fact, Neurons are operating in asynchronous way. 2.3 Feedforward neural network model In a feedforward neural network, information flows in the forward direction only, and there are no feedback loops. The network is always stable, and its state depends on the inputs in a simple manner. Suppose that N neurons form a feedforward neural network with M inputs. Its structure is shown in Figure 2.3. Input Output layer la er Wu 9 y. «25 y. '21s 92 . .s . '/\" 9» It (A Yn Figure 2.3. Single layer feedforward neural network. 5 means sigmoid function 14 For this network, the input-output relationship can be expressed as 311 ( “’11 “’12 ° " WM 1'1 91 l 312 w21 w22 ' " 102114 $2 92 = S + (2.3) _yN‘ (_wN1 wN2 wNM. [3714‘ _0Nd) where 3;, 1 S i S M, is the input neuron, y,, 1 S j S N, is the output neuron, and wj; is the connection weight from ith input neuron to j th output neuron. The above equation is represented by Y = S(WX + o) (2.4) where X is an M-component column vector of inputs, W is an N x M synaptic weight matrix, and Y is the resulting N-component column vector of outputs. 9 is an N- component column vector of threshold and S is a differential, bounded, and strictly increasing monotone function. The threshold can be included into the weight matrix implicitly by using wgo = 0,- and 2:0 = 1. Thus the general equation is expressed as: . . I . , 1 \ Ill “’10 "’11 w12 ° ' ' 101M 31 3’2 wzo 1021 2022 ' ‘ ' sz _ yN-l - I wNo le wN2 “WM 1 I .‘W .) This type of network has only one layer of neurons and there is no connection between neurons themselves. It is called single-layered feedforward neural network or Perceptron [19]. Naturally, its practical applications are limited and interesting applications are emerged when many layers are interacted each other. This can be achieved through cascading two or more layers of such simple network, as shown in 15 Figure 2.4. Any layer between the input layer and output layer is called a hidden Figure 2.4. Two layer feedforward neural network layer. Thus, all but the output layer are hidden layer in internal layers. The operation of such networks duplicates that of Figure 2.3, that is, the outputs of each layer are produced from the weighted sum of the previous layer’s output. As the number of layers increases, the usefulness of the network also increases. The output from a multi-layered neural network can be expressed as Y = S(W‘S(W’ - - - S(WKX))) (2.6) where W" is a matrix of weights for kth layer, the size of this matrix depending on the number of neurons acting as source and destination for the layer. With the development of the back-propagation training algorithm, multi-layered feedforward neural networks overcome many of the limitations of single-layered feed- forward neural network [20]. The back-propagation algorithm uses a gradient-descent 16 search technique to minimize a cost function which is difference between the desired output and the actual output. Also, this approach is called supervised learning. For training the desired output, the input vector is applied to the neural network and its output vector is produced. The actual output is compared to the desired output and the error is passed in a backward direction to adjust the connection weights. The modification of weights is carried out from layer to layer in a backward fashion and this process is iteratively continued until its error is minimized. 2.3.1 The standard back-propagation algorithm The static equations of feedforward artificial neural networks [1][20] [48] [49] are given by "‘3th = 9(wiiv mini) (2-7) ym‘ = 3("8th) = S(g(w,-.-,xp,.-)) (2-8) where g() denotes the general synapse function. Usually, it is given as a dot-product multiplication between input vector and weight matrix. Here, p represents the index of patterns and w, is a connection weight from the (output of the) ith node to the (input of the) jth node and 319.6: which is an output of ith node at pth pattern. Observe that, in a feedforward network, 3110.)“ is the output of the jth neuron in the present layer and mm.- is the output of the ith neuron from the previous layer. 5' is a differentiable, bounded, and strictly increasing monotone function. S is in fact a diffeomorphism usually referred to as a sigmoid function. The desired patterns are presented to the neural network and the outputs of the networks are compared with the desired ones. The error signal at the output of neuron j is defined by ep’j = dpvj — ypvj (2'9) 17 The errors between the desired output (d) and the actual outputs (y) are calculated and propagated backward through the connection weights to minimize the total error. The total squared error is given by 1 1 E = XE» = —£Z(d,,,,- - 31m)2 = -ZZ¢§,,~ (2-10) in 2 p 1' 2 P 1' where p denotes the pth pattern and j denotes the jth output neuron. The error is a function of connection weight 20,-; and updating the weight in gra- dient system is defined as the delta rule: w’F. = wffl + A112,. (2.11) J8 where ijg denotes the changes in the weight 112,; at the kth iteration due to the input-target patterns. The delta rule defines the change due to applying 1) patterns such as 3E _ _ 820,-.- Aw], = -17 as, w (2.12) 77 E p a where r] > 0 is the (learning) rate. Observe that 17 is assumed to be sufficiently small in order for equation (2.12) to be truly gradient system. The method of gradient descent has the task of continually seeking the bottom point of the error measure. According to the chain rule, we may express this gradient as follows: 0E,D _ BEP . Bew- - 831m .Bnetw- _ 2.13 awn 36m 31111,,- Bnetm- 620,,- ( ) Differentiating Ep in equation (2.10) with respect to em, we get E a p (2.14) P1] aepd 18 Differentiating both sides of equation (2.9) with respect to ym, we obtain 36,, ,1- = —1 531m Next, differentiating equation (2.8) with respect to net”, we get 63/95 Finally, differentiating equation (2.7) with respect to 11),, yields 6111:th _ 89(w,-,~,x,,,.-) 6212,.- — 610,-.- Hence, the use of equation (2.14) to equation (2.17) in equation (2.13) yields 3531» 5900151 9312.5) — = —e 6"- net - aw.“ p1] J( p91) Cw]; Accordingly, the use of equation (2.18) in equation (2.12) yields 3E Aw '5 = —7] p 1 2p: 610,-.- 39(wjg, 3? i) = 7) 5 .— 2p: P] aw.“ where the local gradient 6,9,,- is defined by _ 6E? . 36m . 631105 8610.1 63110.1 67‘3th 5m = 6p.15§(netp.j) (2.15) (2.16) (2.17) (2.18) (2.19) (2.20) (2.21) (2.22) The local gradient points to required changes in synaptic weights. We may identify the local gradient in two distinct cases, depending on where in the network neuron j 19 located. For an output layer, we may use equation (2.9) to compute the error signal. Thus it is straightforward to compute the local gradient 6,9,5 using equation (2.22). Aw]; = 77 $0121 - yp.j)3§(n€tp.j)%tgfi?p'—i) (2-23) When neuron j is located in a hidden layer of the network, there is no specified desired response for that neuron. Accordingly, the error signal for a hidden neuron would have to be determined recursively in terms of the error signals of all the neurons to which that hidden neuron is directly connected. Suppose that neuron r,- is in the input layer, neuron y,- is in the hidden layer, and neuron 2,, is in the output layer. For the input-to—hidden connection weight wig, we must differentiate E, with respect to the wjg’s, which more deeply embedded. Using the chain rule, we obtain 8E, 6E, . flyw- 610,,- = Byw- Bwj; (2.24) = 3::-..i*:::.-33;i?’ (2..., = 4mmg—gjfl‘l (226) where the local gradient 6”- is given by ‘5“ = ’33,}: ' 6:32;; (2'27) = -35: .s;(net,,,~) (2.28) Using the chain rule again, we may express the first term of equation (2.28) by CE, _ CE, 66%); all/pd I: 5611.15pr (2.29) 20 = 2.30 -810; 1:63:91]: J' ( ) _ 62,, k anetp k ‘ “e”; kpanet ,. ayp, (2'31) = ‘2 ep kSk( net” )agmkj’ yp,,-) (2.32) k Byrd = W2 69(wa1 311! J') (233) I: By? J where , in the last line, we have used the definition of the local gradient 6M given in equation (2.22) with the index k substituted for j. Finally, using equation (2.33) in equation (2.28), we get the local gradient 6105’ for hidden neuron j, after rearranging terms, as follows: I a w '1 ,° 5m = Sj(netp.j) 2610.}: g( 0"] Sq“) (2°34) 1: 3110.1 Thus, for a hidden layer, the delta rule is given by , a .,:c 6 w -, ,~ ijs‘ = "€253 (”etpj)i('tgjw_11_)k 261" g( a; 1?”) (2°35) where 2:), is the summation over the next layer and 6p), is a back-propagated error from the next layer. 2.3.2 The modified update law Let’s consider the new update law. Define the total energy by the following equation: E. = (é 2w... — 1...)? + flat-wt) (2.36) 21 where a5,- is a sufficiently small ‘forgetting factor’, signifying damping, which is also important for global stability. Update law in continuous time system is expressed as ij, z tbjgdt = —17j536u%dt (2.37) The differential dt is the time—step in the integration of the derivative quantity, 112,3. If we let the differential dt be sufficiently small, the approximation becomes more accurate. 17,-,- is the learning rate of each weight. After the chain rule, the continuous time gradient descent dynamic update law is given by the differential equation as [25] [26]. , 090.0332: 3) wJ'i = "ii 2 :5p.15§(n6tp.j)——a—:for— '— "jiajiwji (2.38) 1» ,. If the neuron j is in an output layer, 5m = (dim - ym‘) (239) If neuron j is in any hidden layer, 69 wk I, y 9 - 5m = 2 610.1: (a J . N) (2°40) 1: 31m where k is the index for the elements in the immediate subsequent layer. The modified continuous time update law is obtained by removing the sig- moid derivative terms, namely S;(netp,_,-) [25] [27]. Also, we can have ij instead of the partial derivative ag(wa-, yp,j)/0yp,,- and mm.- instead of (9g(w,-,-, $11.5) / 810,-.- in equa- tion (2.38) and (2.40). It is shown that the derivative terms in the equation may be removed without loss in stability or convergence of the update law [25] [26]. The 22 modified update law is thus given by ‘in = 71:12 5 .1pr - njs'ajiwji (2-41) in For an output layer, the modified update law is obtained as 6M = (dim - 31:25) (2-42) For a hidden layer, the modified update law is given by 6p“, = Z 6p,kwkj (2.43) k where 6,9,,- 18 glven as 61,”, = ep,k (2.44) Observe that an essential difference between the modified (continuous time) up- date law and the conventional (discrete time) back-propagation is the absence of the derivatives of the sigmoid functions. This results in simplifying the learning rule with major payoff in implementations, both in software and hardware. 2.3.3 Learning methods We have written the update rule as sums over p patterns. For given training patterns, back-propagation learning may proceed in one of two basic ways: parallel learning or sequential learning [27] [28]. In parallel learning, weight updating is performed after the presentation of all the training patterns (i.e., batch mode). In hardware implementation of parallel learning, it requires excessive copies of the feedforward neural network equal to the number of patterns and additional terminals for each input-target pair. In parallel 23 learning, the number of input-target pairs is limited to the number of the copies in the neural network implementation. Although we have written the modified update rule as sums over all patterns p, they can be used incrementally. A pth pattern is presented at the input and then all weights are updated before the next pattern is considered. This clearly decreases the error measure (for small enough 1)) at each pattern. We refer to this approach as sequential learning. In sequential learning, the network receives p input-target pairs periodically as time-varying signal. If we continue to feed the input-target pairs to the network, the network will learn the input-target pairs by converging to a set of equilibrium weights. From an implementation point of view, the sequential learning is preferred over the parallel learning, because it requires less hardware component. Moreover, given that the patterns are presented to the network in a random manner, the use of sequential learning makes the search in weight space stochastic in nature. This randomization makes the search less likely for the back-propagation algorithm to be trapped in a local minimum. On the other hand, the use of parallel learning provides a more accurate estimate of the gradient vector, while sequential learning gets the time-averaged value in weight space. The relative effectiveness of the two learning modes depends on the problem [1] [48] . 2.4 Feedback neural network model Biological systems have some sort of feedback among various neurons. A more realistic neural network model should have such feedback path. In this section, We briefly introduce the Hopfield model [21][22] [23]. Extensions to the Hopfield model have generated interest in a new class of dynamic network models called recurrent neural networks which are capable of performing a wide variety of 24 computational tasks including sequence recognition, trajectory following, nonlinear prediction, and system modeling [24]. In the Hopfield model, each neuron has a nondecreasing sigmoid nonlinearity. Its output is fed back to all other neurons via synaptic weights. Each synaptic weight is denoted by T5,, which connects from the output of neuron j to the input of the neuron i. There is a symmetric requirement on the connections, namely, we must have ng = Tjg. The Hopfield model is shown in Figure 2.5. “1 V1 1‘7 v I V [5 'r'll—W—l [__I I I "H 7937 "H 'I 1|] g;- _ III—\NVL—l a“ Des/fl 1||—,/\/\/~——l 1—1 H? (b) Figure 2.5. (a) The circuit of one unit in Hopfield model. (b)A three neuron Hopfield neural network. Each unit 1' is consists of the circuit shown in Figure 2.5(a). u,- is the input voltage, 25 V.- is the output voltage, and the amplifier has the transfer function of V.- = g(u,-). I,- is the external input to the node i. The input of each unit is connected to ground with a resistor R and a capacitor C. The output of unit j is connected to the input of unit 1' with a resistor 12.3. The circuit equations are du; u; I or, equivalently Cgfl—ZT-V .1_ ~+I- (246) (It — j 13 J 12"“: 1 . where T-- __ i (2 47) I) R‘j ' 1 1 1 — = - + — 2.48 Rt P 21': '2' ( ) The equilibrium points of the above neural system are the roots of the n simulta- neous equations 1 0 = 271116 - E93104) + 1; (2-49) .7 The dynamic behavior of I the Hopfield model can be examined by considering the energy function of equation (2.46). Equation (2.46) can be rewritten as du; _ CE 7&- - ~37; (2.50) where the energy function, E, is given by V.‘ E = -§ZX:71.VM +22]. g:‘(v.-)dv— 23w.- (2.51) i j i i 26 The time derivative of the energy function along trajectories is d_E_ dt since C _>_ 0 and g,-(u.-) is a monotone nondecreasing sigmoid function. Therefore, this system is a gradient-descent system. That is, this energy function decreases along trajectories and its time-derivative equals zero at 35 = 0, which is an equilibrium point of this system. The Hopfield model is proposed as an associative memory or to solve optimization problems. |/\ CE du.‘ 2.3 5:53 3E {IV} dug Emma 141/.- as 2 2351—... I57) 1' _1_dgi(‘ui) BE 2 CHAPTER 3 Analog CMOS Circuit Blocks 3.1 CMOS transistor Research into some of the unusual electrical properties of semiconductors led to the development of the transistor, a device for controlling the flow of electrons in a solid crystal. Like a switch, a transistor can either allow or inhibit the flow of electric current in response to an external signal. Metal-Oxide—Silicon (MOS) Field Effect Transistors are commonly used in digital and analog electronics. A MOS transistor is formed by creating islands of semiconducting material, doped with either negative N-type or positive P-type charge carriers, in a substrate of the same material doped with charge carriers of the opposite type. The schematic drawing of an N-channel MOS transistor and P-channel MOS transistor is shown in Figure 3.1. By alternating the voltage applied to the gate, charge carriers from the source are either attracted toward the channel or repelled from the channel. A channel under the gate is formed since enough attracted or repelled charges are accumulated. The channel allows current flows all the way across the gate region. The first MOS electronic circuits employed p-channel (PMOS) transistors. As MOS technology advanced, n-channel (NMOS) transistors replaced PMOS transistors because they offered higher speed performance than PMOS. The need for reduced 27 28 NMOS PMOS source gate drain drain gate source _Hiflflifln n n p p n-well p-type substrate Figure 3.1. The schematic diagram of NMOS and PMOS power consumption led to the development of the larger but more powerful efficient Complementary MOS (CMOS) transistors. 3.1.1 MOS Models The n-channel and p-channel enhancement MOS devices along with the convention for the electrical variables are shown in Figure 3.2. In Figure 3.2, (a) and (b) shows the convention of the four terminal device respec- tively. If the bulk terminal of NMOS is connected to the lowest circuit level, usually V35 or GN D, the bulk the convention of (c) is used. This is equivalent to (d). In (d), the bulk terminal of PMOS is connected to the highest voltage of the circuit, VDD. The following model equations is restricted to the n-channel transistor. The p- channel model equation is identical with the exception of sign changes in some of the equations. The same model is used for the PMOS if all the voltages and currents are multiplied by -1 and the absolute value of the p-channel threshold is used. When the length or width of the MOS is greater than about 10pm, the substrate doping is low, and when a simple model is desired, the model suggested by Sah [29] and used in SPICE by Shichman and Hodges [30] is very appropriate. The dc model 29 vDs ["4 G vSD [3'26 ‘l' . Tis vas +13 vso — - + + S S (C) NMOS (d) PMOS Figure 3.2. The convention of N MOS and PMOS, D:drain, stource, G:gate, szulk introduced by Sah is given by VDs ID = [3((Vas - VT) -— ——2-—-)Vps (3.1) which was derived for small value of V05, 0 < V03 S VGs — VT. Small values of VDs correspond to the ohmic region of operation. The region is termed the ohmic, linear, or active region. In this equation, ,6 = the transconductance parameter W = K— = (poCoz)Z(amps/volt2), L L K = FOCoa: p0 = surface mobility of the channel, 0 t 0 Cox capacztance per umt area of the gate oxide = —, 03 £3 a II 30 L = channel length, W = channel width, VT = threshold voltage For V05 2 Vas — VT > 0, the current remains practically constant(independent of V05) at the obtained when the channel is pinched off. The equation is obtained by _ fl 2 ID —- 5(VGS — V7) (3.2) which is good for V03 > V63 — VT and veg > VT. The region of operation is termed the saturation region. When the MOSFET is operating in the saturation region, the MOSF ET is inherently a transconductance-type device with the voltage input, V03, and the current output, I D. If VGs — VT is zero or negative, then the MOS is in the cutoff region and the current becomes zero. ID = o, VGS — VT 3 o (3.3) The model based on the equations (3.1), (3.2) and, (3.3) is the simplest model. In many situations, this model is quite tractable for hand calculations and adequate for the analytical portions of the design. It can be shown theoretically and experimentally that the drain current in the saturation region increases slightly in a linear manner with V05. Defining A to be the coefficient that represents the linear dependence of I D on VDS, a more accurate expressions for the drain current in the saturation region is given by _ 5 2 ID — 5(Vas — VT) (1+ AVDs) (3.4) The coefficient A is quite small for long devices but increases considerably for very 31 short transistors. The W/ L ratio is the only geometrical design parameter available to the design engineer that affects the performance of MOS transistor. Assuming the parameter K and VT are constant, it can be shown that the device is electrically symmetric with respect to drain and source. The choice of which end of the channel to designate as source and drain is thus arbitrary. Since the MOS is a bi-directional device, the source for an n—channel transistor is always at the lower potential of the two nodes. For the p-channel transistor, the source is always at the higher potential. The threshold voltage, VT, is somewhat dependent upon the bulk-source voltage. The dependence can be approximated by VT = VTo + 7(V¢ - VBs - fl) (3-5) where V33 is the bulk-source voltage and V10, 7, and 45 are process parameters: VTo = threshold voltage for V33 = 0 7 = bulk threshold parameter ()5 = strong inversion surface potential Note that the change in V7 can be quite significant for large V33. The effect becomes even worse with larger A. This simple model has five electrical and process parameters that completely de- fine it. These parameters are K, VT, 7, A, and 43. They constitute the Level 1 model parameters of SPICE (Simulation Program with Integrated Circuit Emphasis) circuit simulator [31][32]. In many situations, this model is quite tractable for hand calcula- tions and adequate for the analytical portions of the design. With the Level 1 model, the simulation does not perform the short- and narrow-channel effects. 32 In the Level 2 model based on Meyer [33], the short- and narrow-channel effects are calculated. The Level 2 model differs from the Level 1 model both in its method of calculating the effective channel length (A effects) and the transition between the saturation and ohmic region. The Level 2 model offers improvements in performance which are particularly significant for short channel devices. The Level 3 model has been developed to simulate a semi-empirical model. Several empirical parameters (parameters not obviously related to or motivated by the device physics of the MOS transistor) are introduced in the Level 3 model. It simulates quite precisely the characteristics of MOS which have a channel length up to 2pm. The basic equations have been proposed by Dang [34]. It is useful to examine the differences among the three models [31]. The Level 1 model is elementary, the Level 2 model uses processing parameters and geometry, and the Level 3 model uses measured characteristics. Usually the Level 1 model is not suffiCiently precise because the theory is too approximated and the number of fitting parameters too small; its usefulness is in a quick and rough estimate of circuit performances. The Level 2 model can be used with differing complexity by adding the parameters relating to the effects needed to simulate with this model. However, if all the parameters are used, i.e., the greatest possible complexity is obtained, this model requires a great amount of CPU time for the calculations, and it often causes problems with convergence [35]. The Level 3 model takes less time and less errors on simulation than the Level 2 model. The only disadvantage of the Level 3 model is the complexity in the calculation of some of its parameters. 3.2 Transconductance amplifier The transconductance amplifier is a device that generates current as its output. If the output current is proportional to the difference between two input voltages, V1 33 and V;, the circuit is called a differential transconductance amplifier. Since the input terminal of the transconductance amplifier receives the voltage input, we usually need current-to-voltage conversion at the output stage of the amplifier. The current-to-voltage converter is implemented using operational amplifier and resistors. However, it is not practical to use operational amplifier inside the neural network implementation. Thus a simple and convenient current-to-voltage converter is needed. 3.2.1 Active resistors / Loads The current-to-voltage converter is implemented using two transistors. The active resistor is used to produce a dc-voltage drop or provide a resistance which is linear over a small range. The active resistor is achieved by simply connecting the gate to the drain. The active resistor is shown in Figure 3.3. _fi M, -I 1m- Figure 3.3. The circuit diagram of active resistors using PMOS Vss In this figure, there is a current source I m and two PMOS transistors. If we assume that M1 and M2 is a matched pair, i.e., they have the same conductance fl, and work in saturation region, we have the circuit equation by applying Kirchhoff ’s 34 current law at the VOUT node: 1m + 101 = 102 (3-6) I — B V V V 2 D1 — §( DD — OUT - T1) (3.7) _. 5 2 102 — 5(Vour - Vss - V12) (3-8) After some algebraic manipulations with the assumption, VT = V11 = V72, VOUT is given as 1 1 (Von + Vss) V = —I + 3.9 OUT (VDD _ V33 _ 2%) )6 IN 2 ( l _ 1 (Von + Vss) where G is the conductance of the active resister, G = (VDD — V35 — 2VT)fl (3.11) If we assume that V33 is 5V and V33 is 0V, then VOUT will be 2.5V when [m is zero. In Figure 3.4, the characteristics of current-to—voltage relation are shown with different W/ L ratios of the active resistor. In this graph, the W/ L ratio of the upper PMOS is assumed to be same as the W/L ratio of the lower PMOS. We can see that the resistance increases as the W/ L ratio of PMOS decreases. In Table 3.1, the calculated resistances at the (0A, 2.5V) point from Figure 3.4 are shown with different W/ L ratios. From the table, the value of resistance of active resistors ranges from 10K 0 to 1M9. If we need larger active resistors but we have a limitation of area in chip layout for active resistors, we can cascade active resistors. Two upper and two lower active 35 -mm.mm.m(nw1m Wham 212326 TWR‘M &w T ............................................................................................................. 1 .................... 29v: .WW. . ..... , WM ...... m m -20M 400A 0A D W“) 0 W12) A V03) ' W14) 0 W15) + V(10) x W") Figure 3.4. The PSPICE circuit simulation result: Active resistors with different W/ L ratios Table 3.1. Resistance with different W / L ratios for Active resistors I (W/L)M1 (W/L)M2 I Resistance, R = é = 52% I 4p/4p 4p/4p 20.61Kfl ' 4p/8p 411/8}; 47.03Kfl 411/1611 411/1611 95.85KQ 411/3211 411/32}; 192.941“) 411/6411 411/6411 392.041“) 411/ 128;: 471/128}: 786.05KQ 411/2561: 4p/256p 1573.1Kfl 36 resistors approximately have 9 times greater resistance than that of single upper and single lower active resistor with the same W/ L ratio. The circuit diagram of the two cascaded active resistors and its PSPICE simulations are illustrated in Figure 3.5 and 3.6. In Table 3.2, the resistances of the cascaded active resistors are shown. To make active resistors operate correctly, we have to consider the body effect of the transistor. To make two threshold voltages, VT; and V12, the same, M1 and M2 should have the same bulk-source voltage, V33, according to equation (3.5). - Von M1 » [.1 M2 ——(—>,L v VOW M3 ['1 M4 Figure 3.5. The circuit diagram of the cascaded active resistors using PMOS Vss That is why each transistor’s bulk terminal is connected to its source. The selec- tion of NMOS or PMOS as linear active resistors depends on the fabrication technol- ogy. If n-well is used for fabrication, PMOS is used as active resistors because we can isolate the bulk voltage of PMOS, i.e., the bulk voltage of N MOS is tied together to the p-type substrate. When p-well is used for fabrication, NMOS should be used as active resistors since we can isolate the p-well only. 37 °Awwwum2m:oam.mmdr Dds/Tine rm: muss 20.4210 Tm 27.0 0.0V r --------------------------------------------------------------- - ------ . ------------------ , ----1 ‘.0v 1. .................................. 2 any i ..................................... WM“ . NIL-W . OV‘: _ . ............... . ................... E . WIb-Gultou ' E wmm r ----- ' -------------------------------------------------------------------------------------- 4 ~20uA -15uA -10uA 0A 511A 1011A 1511A 2011A ovm) ovaz) was) vvm) oV(15) lln Figure 3.6. The PSPICE circuit simulation result: The cascaded active resistors with different W/ L ratios Table 3.2. Resistance with different W/ L ratios for the cascaded active resistor I (W/LlMl = (W/Lle (W/L)M3 = (W/L)M4 I Resistance, R = % = fl AI 411/4}; 4p/4p 208.81KQ 411/811 411/811 455.801“) 411/16}: 411/16}; 893.65KQ 411/3211 4p/32p 1730.41“! 411/64}; 4p/64p 3418.6Kfl 38 3.2.2 Simple transconductance amplifier The differential transconductance amplifier is one of the most versatile circuits in ana- log circuit design [36] [37]. The objective of the differential amplifier is to amplify the difference between two different voltages regardless of the common-mode value. The differential amplifier is characterized by its common-mode rejection ratio (CMRR) which is the ratio of the differential gain to the common-mode gain. Another char- acteristic affecting performance of the differential amplifier is voltage offset. If the terminals of the differential amplifier are connected together, the output offset voltage is the voltage which appears at the output of the differential amplifier. Let us consider the large- and small-signal characteristic of the CMOS differential amplifier [36]. Figure 3.7 shows a CMOS differential amplifier that uses n-channel MOS devices, M1 and M2, as the differential pair. M3 is a current source and the loads for M1 and M2 are obtained from a simple p-channel current mirror (M4 and M5). If M4 and M5 is matched, then the current of M1 will determine the current in - V00 M4 [>—<'——<] [Ir/15 Um I In: ’0'”. II»: II... v, 14511 M2]: v, V V I, I... vans ‘_] M3 . VSS VOUT Figure 3.7. Simple transconductance amplifier 39 M4. This current will be mirrored in M5. If V1 = V2, then the currents in M1 and M2 are equal. Thus the current in M5 is equal to the current in M2, causing Io,“ to be zero. If V1 > V2, then 131 increases with respect to 132 since 133 = 101 + 132. This increase in [31 implies an increase in 134 and 135. However, 13: is decreased with respect to 131, therefore, the only way to establish circuit equilibrium for IOUT is to become positive. It can be seen that if V; < V; then [OUT becomes negative. The large—signal characteristics can be developed by assuming that M1 and M2, the differential pair, are always in saturation. The relationship describing large-signal 21 21 VID=VGS1—Vcsz=‘/ BB] -‘/ :2 (3.12) 103 = IDl + [02 (3-13) behavior are given as where it has been assumed that M1 and M2 are matched (3 = 31 = [32). The solution for 131 and 132 is given by 101:— 2 2 133 + IDs2l/ID\I fl 3 VID (3.14) I =——— ——— - D” 2 2 \IID3 41,2,3 (315) where these relationships are valid only for V13 < ([2253. Figure 3.8 shows a plot of the normalized drain current of M1 and M2 versus the normalized input voltage. If we assume that the currents in the current mirror are identical, then IOUT can be found by subtracting 132 from 131. The output voltage V037 of the differential transconductance amplifier can be found by assuming that a load resistance R3 is 40 .......................................... up .......................................... ' 1 ' —> (IS/1...)”V... -1.41 -1 o 1 1.41 Figure 3.8. Current output of the simple transconductance amplifier connected from the output of the amplifier to ground. 2V2 VOUT = IOUTRL = (131 — IDZ)RL = ID3V1D\ fi— — fl ID 103 412,” 12;, (3.16) The differential-in, differential-out transconductance is written as _ alovr gmd-a—w; KI W (v,D=o)=\/flloa= —‘ Z3 1 (3.17) The small-signal voltage gain of the differential amplifier can be found by differenti- ating Eq.(3.16) with respect to V13 and setting V") = 0, giving av K I W Av = 0‘” = 31033,, = ‘l—lm—l-RL (3.18) 8V”) L1 We note several important deviations from ideal behavior of the differential am- plifier. The first is the mismatch between transistors. Not all transistors are created equal. Some are created with a higher transconductance property than are others. It causes current mismatch and shift in characteristic curve and results in voltage offset. 41 The designer must consider the worst case VT spread (specified by the process) in each [transistor and adjust current level and transconductance to meet the requirements. Typically, the voltage offset of a CMOS differential amplifier is 5 to 20 millivolts. The common mode gain of the CMOS differential amplifier is ideally zero, because the current-mirror load rejects any common-mode signal. Due to the mismatches in the differential amplifier, a common-mode response might exist. This mismatches consist of a non-unity current gain in the current mirror and geometrical mismatches between M1 and M2. Another important characteristic is the input common-mode voltage range. The input common-mode range is defined by the input voltage range over which both M1 and M2 remain in saturation. The highest input voltage at the gate of M1 (or M2) when V1 = V; is found to be V1 = VDD - V504 - V031 + V031 (3.19) For saturation, the minimum value of V331 is V051 = V031 - VT01 (3-20) Substituting and replacing the equations give the final result, 2104 V1=VDD- 34 - IVT04I + V101 (3-21) The last two terms are determined by the process and the equations now becomes [1 . Vl(ma.r) = V33 — f — IVT04|(maJ:) + V701(mzn) (3.22) As V1 approaches V33, M1 will be in the saturation region and close to cutoff. 42 Therefore, it makes more sense to relate Vl(min) to V3 343 when M3 is no longer in saturation. The gate voltage on Ml can be shown to be V1 = V051 + V033 (323) Set V053 = V053 — VT03 to get V1 = VGss + V051 - VT03 (3.24) 21133 V1(m2'n) = V3343 + + VT01(ma:r:) — V703(min) (3.25) [31 Third, the common-mode input voltage has a significant effect on the transfer function, particularly the output-signal swing. In this case, the swing limitation will be based on keeping both M2 and M5 in saturation. When V1 is taken above V2, the output voltage, VOUT, increases. The output voltage is given as VOUT = VDD — V305 (3-26) M5 is at the edge of saturation when V335 = V335 — |V1~05|. Using this relationship, the maximum output voltage is given as VOUT = VDD - V304 + [VT05I (3.27) 2104. 34 The minimum output voltage is found by determining when M2 is at the edge of Vour(ma$) = VDD - - IVTO4I + IVrosl (3-28) saturation. The minimum output voltage is given by VOUT(min) = V2 -- VT2 (3.29) 43 Table 3.3. The transistor sizes of the simple transconductance amplifier [Transistor [ W/ L ratio I M1 4,1/4p M2 4p/4/1 M3 1211/4}: M4 l5p/4p M5 l5p/4p To verify the limitation on the output voltage range, a PSPICE simulation has been performed. To see the output current, we attached two resistors at the end of the output terminal. Since two resistors have the same value of 100K 0, the output current is 131 - 132. The circuit diagram is shown in Figure 3.9 The W/L ratios of the transistors are shown in Table 3.3. Figure 3.10 shows the PSPICE simulation result of the differential transconductance amplifier. The influence of V2 upon [OUT is illustrated in this figure. shown in Appendix A.1. The PSPICE input file is 1 I111 R1 12:: Figure 3.9. Simple transconductance amplifier with load resistors 44 'WWWW-tflflmtm Warm 22:04:17 Team-1:210 5.” T ------------------------------------------------------- 1 ' 2.0V 2.5V ' 3.0V asv ' 4.0)! 45v Figure 3.10. Simple transconductance amplifier: Output current (131 — 132) as func- tion of V1 for different values of V2 3.2.3 Wide range transconductance amplifier A simple transconductance amplifier will not generate output voltage below VOUT(min), which, in turn, is dependent on the input voltages. We can remove this restriction by a simple addition to the simple transconductance amplifier, as shown in Figure 3.11. To overcome the problem, two extra current mirrors are usually added [38] [39]. By reflecting the currents of M1 and M2 to upper current mirrors, the output current is just the difference between 11 and 12. The major advantage of the wide-range amplifier over the simple circuit is that both input and output voltages can run almost up to V33 and almost down to V33, without affecting the operation of the circuit. The output current, IOUT, in the PSPICE simulation is shown in Figure 3.12 and Figure 3.13. 100K 0 resistors are used in Figure 3.12, and 1M fl resistors are used in 45 Ml] [He—<1 M5 M6 kiwi 1.7 . ”BIL... {1m LIE? Figure 3.11. The circuit diagram of the wide range transconductance amplifier Table 3.4. The transistor sizes of the wide range transconductance amplifier I Transistor [ (W/L) ratio ] M1 411/411 M2 4p/4p M3 12p/4p M4 1511/411 M5 15,1/411 M6 l5p/4n M7 1511/41: M8 4p/4p M9 4p/4p 46 “Whom-w Malta) Minn-1: m 21:55:” Tm: 27.0 Figure 3.12. The PSPICE circuit simulation result: the wide range transconductance amplifier, output current as function of V1 for several values of V2, Resistor = 100K (I 'mmmmtm When-12m 8:00:20 TW:27.0 5.“ T ............................................................................................................ 1 Figure 3.13. The PSPICE circuit simulation result: the wide range transconductance amplifier, output current as function of V1 for several values of V2, Resistor = 1M0 47 Figure 3.13. The output current represented in the difference current between R1 and R2. The PSPICE input file is shown in Appendix A.2 and the sizes of transistors are shown in Table 3.4. As the value of resistors increased, the saturated level of current is decreased and the current gain at the operating point is increased. 3.2.4 Sigmoid function generator The sigmoid function can be obtained by using a wide range transconductance ampli- fier and active resistors. The output of the transconductance amplifier is in current. Thus we attached diode-connected MOS transistors as active resistors to convert current to voltage. Using active linear resistors, we can achieve voltage-to-voltage operations. The complete circuit of sigmoid function generator is shown in Figure 3.14. The sizes of the transistors are shown in Table 3.5. The transconductance of the differential transconductance amplifier is proportional to the bias current. The (W/ L) ratios of the M1 and M2 transistors also control the transconductance. We can achieve a larger transconductance when we increase the width of the transistor M1 and M2. The PSPICE simulation results are shown in Figure 3.15 and Figure 3.16. The PSPICE input files are shown in Appendix A.3 and AA. In Figure 3.15, VOUT is shown with different W/ L ratios of the active resistor. If the W/ L ratio is decreased, the gain is that of the tanh like function and saturated level of high and low voltage is increased. In Figure 3.16, VOUT is plotted according to the different V3343 voltages. From this figure, it is obvious that the gain of the sigmoid function generator and the level of the saturated voltage are dependent on the V3 343 voltage. From the PSPICE circuit simulation, we have two parameters to control the char— acteristics of the sigmoid function generator. The W/ L ratio of the active resistor and the bias voltage are the controllable parameters. 48 Figure 3.14. The circuit diagram of sigmoid function generator Table 3.5. The transistor sizes of the sigmoid function generator [Transistor [ (W/ L) ratio [ M1 2411/41: M2 2411/4/1 M3 10p/4p M4 1511/4); M5 15p/4p M6 1511/4}; M7 1511/4” M8 1611/41; M9 16p/4p M10 4p/19p M11 411/2111 49 'M(SWWMMMNLW1JV:OMT.W(m1M Wit-um 23:56:28 Tamer-tin: 27.0 3.5V 1 """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" 1 I I A A ___I . V I O i O L; t v o I sov -: _4: i ‘ . I . ,. - : : I Misfit!“ _ / 1‘r f ‘ [I A mammal ~. 2.0V { ‘Wmm . - “farm WW 1.5V «3- -------------------------------------------------------------------------------------------------------------- 4 aw 2.2V 24V 20/ 2.0V aov 0V0) Wt") was) 'VM WW) v1 Figure 3.15. The PSPICE circuit simulation result: Sigmoid function generator with different W/ L ratio of the active resistors (M10 and M11) in PMOS when the bias voltage is 1.1V . 801mm mmmmwm(mw.mwom.w(m Min-mm 00:03:40 Twas kW 1' """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" 1 Figure 3.16. The PSPICE circuit simulation result: Sigmoid function generator with different bias voltages when (W/L) of the active resister (M10 and M11) is (4pm/19pm,4pm/2lpm) 50 3.3 The Modified Gilbert Multiplier To implement the multiplication of the neural networks, a multiplier is one of the most important component. Several different techniques are used in the multiplier design. Multipliers based on a modified Gilbert cell [40] [41] [42] are popular. The other technique is the use of the square-law characteristics in saturation region [43] [44] [45]. Others have implemented the multiplier based on the current-voltage characteristics in the non-saturation region [46]. We consider a modified Gilbert multiplier as a synapse circuit[38]. The circuit diagram of the modified Gilbert multiplier is shown in Figure 3.17. Assume that all transistors in Figure 3.17 are in saturation region, and are matched so that the transconductance parameters satisfy the equations flN = 3M1 = 5M2 and 5? = 5M3 = 5M4 = 5M5 = flMe. The output current is the difference between 1301412) and 1301413) since the current 1301421) and 130,122) are reflected by the current mirrors. Defining the output current 1+ = I301“) + 13(M5) and I. = 130143) + 130146), it can be readily shown that the differential output current I 3133 = 1+ — I. is given by 11m = (bran-V.) (WI _ 5pm. — v4)? _ WJI _ now. — V0") ”own 2101142) (3.30) The above equation can be approximated into IDIFF = \/2,BP(\/ID(M1) - \/ID(M2))(V3 - V4) (331) if the following condition is satisfied .BP(V3 - V02 << 1 21 D(M1) 51 3120/3 — V4)2 << 1 21mm) Also, the current I 3( M1) and I D(M2) are dependent on the voltage difference (V1 — V2). Since (Vl — V5) is given by K_%=‘/?_I_2LEQ_‘/M . (332) BN ,BN the equation (3.31) becomes IDIFF = t/flPflNU/s - V4)(V1 - V2) (333) This is the ideal characteristics of the approximated equation. ; A “71—3138 “Em 11.0 wt. t 7'11 3 E2 1+ V4—<[ M4 M3 ID—vs v3—<[ M5 M6 [o—w l—I E? v1—-[ m m [—vz (:EI—J M32 M11 |-"-[ [21112 vm —[ M15 M13 ]—"—HM14 . f v” Figure 3.17. The circuit diagram of the modified Gilbert multiplier The modified Gilbert multiplier takes the difference between two voltages (Va -— V4) and multiply that difference by a difference two other voltages (V1 - V;). In the small 52 Table 3.6. The transistor sizes of the modified Gilbert multiplier ratio ' ratio 4;: 4p 11 4 4 4p M12 4 15 4 13 15 4 14 15 4 M15 15;: 1 15 4;: M22 15 4 31 15 4 M32 15 4 '1-DVWWWIT. mwufl) 2.0V 3 va-zasv MW i M 1m - - - : vs-asv i -------------------------------------------------------------------------------------------------------------- 4 1.0V 15V 20V 2.5V 30V 3.5V 40‘! ova) Figure 3.18. The PSPICE circuit simulation result: the modified Gilbert multiplier with V2 = V4 = 2.5V, V(5) is the output voltage 53 signal range, the characteristic curve is approximately linear, and all four inputs carry information about multiplication. For the large-signal range, the multiplier is nonlinear. However, the nonlinearity does not cause any instability. Since the modified Gilbert multiplier has current outputs, we employed PMOS active resistors at the output stage of the multiplier to convert current to voltage. Thus, the voltage- to—voltage multiplier is achieved. The PSPICE circuit simulation result of the modified Gilbert multiplier is shown in Figure 3.18. In this figure, V2 and VI, are connected to Reference voltage, 2.5V. With different voltages of V3, the output voltage is shown according to the input voltage, V1. The output voltage shows four quadrant multiplication. The PSPICE input file is shown in Appendix A.5. 3.4 The Vector Multiplier In a dot-product operation, two vectors are multiplied to generate a scalar quantity. Let X = ($1,:c2,...,a:,,)T and Y = (y1,y2,...,yn)T be N x 1 vectors. Their dot- product operation is expresses as z = XT - Y = 223,315 (3.34) In a vector multiplier, the vector multiplication is given as Vout = E9041 — V,-2)(V,-3 - V14) (3.35) To implement the dot-product operation, we use the modified Gilbert multiplier to obtain the vector multiplication. There are identical Gilbert multiplier subcircuits in the vector multiplier and they are connected together at the terminals of the current mirror. The differential output currents from the multiplier are summed on 54 two current buses and they are converted to voltage through active PMOS resistors. One thus can construct larger dimensional vector multipliers by simply adding the Gilbert multiplier subcircuits. Figure 3.19 presents the 3-dimensional vector multiplier. In this figure, identical three Gilbert multiplier subcircuits are connected together via 1+ and I- current bus. ‘3 W ‘1 I W r115 :12” IT'S] 23—1“ 114-4111 unit-v1 vs—rlm asp-V4 w-fim Ink-v1 113-41¢ unlit-v0 I-vz '- VI-I m mI-in ‘- ”mi 23—12 I“ 3.2121” VG" V88 Figure 3.19. The circuit diagram of the 3-D Vector Multiplier The PSPICE simulation result is shown in Figure 3.20. In this figure, all V1,V2,V3 and V4 of the subcircuits are connected together. V2 and V4 are connected to 2.5V and V3 is varied from 0.5V to 4.5V. The output voltage obtained by the different voltages of V1 and V3 are shown in this figure. The output voltage gain and the saturated voltage are controllable by the V3 343 voltage and the (W / L) ratio of active 55 'sovmwonm. “(magnum m MI: was 01:30.36 Tumult: 27.0 4.0V 'r """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" 1 ssvé - mo. as. 3.5V . m 3. ............................................................................................................. 4 1 .5V 2.0V 2.5V 3.0V 3.5V 0 V(5) 1 V Figure 3.20. The PSPICE circuit simulation result: the 3-dimensional vector multi- plier resistors. 3.5 Implementation of the modified update law With a view towards a circuit realization, we have proposed the circuit to implement the modified update law [13] [28] [47]. To approach the concept of implementing the differential equation, let us first review the circuit shown in Figure 3.21. In this figure, there is a current source I m from the modified Gilbert multiplier and two PMOS transistors as active resistors. A capacitor is attached to the VOUT node to generate the dynamic behavior of the circuit. We assume that M1 and M2 are a matched pair (i.e., they have the same conductance ,3), work in the saturation region, and have the same threshold voltage. We have the circuit equation by applying l 1m —<—>H - k V § J} 1c our I... g c (multiplier) _fifi M2 I -1 I. Figure 3.21. The implementation of the differential equation Kirchhoff ’8 current law at the VOUT node, [IN + 101 = 102 + IC (336) 101 = .B(VDD — VOUT — VT)2 (3-37) 102 = 5(V0UT - V35 - VT)2 (3338) d 10 = C ‘2” (3.39) After some algebraic manipulations, we have a solution for VOUT as dV 1 2 fl = —IIN-£(VDD-Vss-QVT)V0UT+E(VDD"VSS‘2VT)(VDD+VSS) (3.40) dt C C C (IV 1 2G G 5th = 51m - —C—V0UT + 5(VDD + Vss) (3-41) where G is given as equation (3.11) G = 3(VDD - Vss - 2VT) 57 The last term can be treated as an AC-ground term. If we rewrite the equation with the shifted variable VOUT, V + V VOUT = VOUT - —DD—2——S—S (3-42) dV l 20 ——5th = 51m - ‘CTVOUT (3-43) With a new variable 76;, the reference voltage is shifted to 2.5V when VDD is 5.0V and V55 is 0V. The equation (3.43) is qualitatively same as the following differential equation (3.44). 13:2 = "(di - yi)$j " flawu‘ (3-44) Thus the learning parameter, 1], is related with the capacitor value in the learning circuit and the a parameter is related with the conductance of the transistor. CHAPTER 4 Learning Temporal Signals The supervised learning algorithm will be extended to incorporate the use of temporal processing. The standard feedforward network makes its topology implausible in a neurobiological sense. On the other hand, by permitting arbitrary connections, including the use of feedback, the recurrent network assumes a more neurobiologically plausible topology [48]. In this chapter, we review several learning methods in literature for temporal sig- nal. We consider a neural network consisting of the interconnection of N continuous- valued neurons, V5, continuous in both time and amplitude, and with the synaptic weights from neuron j to neuron i denoted by 212.3. Neurons are assumed to have the nonlinear function, 5', such as tanh. Unlike the standard feedforward network, the network will be permitted to have feedback connections among the neurons; that is, the network is recurrent. Three subsets of neurons are identified in the neural networks: 1. Input neurons: they receive stimuli directly from the externally applied pattern. 2. Output neurons: they supply the overall response learned by the network. 3. Hidden neurons: which are neither input nor output neurons. 58 59 Of particular interest is the ability of the recurrent network to deal with time-varying input or output through its own temporal operation. 4.1 Time-dependent recurrent back-propagation Pearlmutter [50] has developed an algorithm for training a general continuous-time recurrent network. The units evolve according to d i Td—yt = —yi+S(Zwijyj)+Ii (4-1) = —y,- + S(nJet.-) + I; (4.2) Some of the units are designed as input units and have input values 1;. Similarly, some units are used as output units with desired training values d;. The objective is to minimize the difference between the outputs of the neural networks and the desired outputs: =%/ 2b,. — dk]2dt (4.3) 0 1:50 where the sum is only over the units that are output units. We must minimize equation (4.3), subject constraints equation (4.1). For this purpose, we introduce the functional: t1 J = Ldt (4.4) = /°1§1 Zlyk—dklz'l’ZA (t) T_y_+y'_ 5(ngjyj) -—I,- dt (4.5) ‘0 21:60 where the constraints equation (4.1) is multiplied by Lagrange multipliers, A,(t). The variation of the integral[5l][52] leads to the well-known Euler-Lagrange equa- 60 tions for the functions y;(t), Ag(t) depending on time. For y;(t), we have (9L d 0L 531—507)” (4'6) and BL 5A: — (4.7) with the end-point conditions: 3L 5;“ — t1) — 0 (4.8) Notice that the equation (4.7) is in fact equivalent to equation (4.1). The equation (4.6) can be expressed as 0L , 67;- = 61¢(yk - dk) + A; —- z: Aijwj,‘ (4.9) ' 1 3L — = A,- . all.“ 7' (4 10) where 6;, is 1 if k E 0 and 0 if otherwise. 5'; is the derivative of S(net,~) in terms of netj. Finally, we obtain TX; = 6k(yk - d],) + A; — z Ang-wj; (4.11) J' The boundary conditions are given by 6L 6—3j.-(t = t1) = 0 = /\.‘(t = t1) (4-12) As for the back-propagation algorithm, we adapt the weights by gradient descent 61 method such as OJ 6ng t Aw.-, = -17 = 1] f ’ y,.5'{).,~dt (4.13) to where 17 is a small positive constant. Equations (4.1), (4.11), and (4.13) specify the learning rule for the recurrent net- work. Equation (4.11) is a backward integration in time and has a final value in equation (4.12). The learning algorithm proceeds as follows: 1. Starting from t = to to t = t1, integrate the activation equation (4.1) in forward time with initial condition. Record every state, 3],, and the derivative of Si, S’(net.-). 2. Make input signal halt. With the final value of A, in equation (4.12), start integrating the equation (4.11) in backward time. Record every co-state, A5. 3. With the data which are obtained in step 1 and step 2, update the corresponding weight change using equation (4.13), and repeat the computation. This algorithm is successfully used by Pearlmutter to learn temporal trajectories [50]. He has trained a recurrent neural network with no inputs, four hidden units, and two output units to follow the circular trajectory. In addition to the circular trajectory, he has trained a network with ten hidden units to follow the figure eight trajectory. Since this algorithm requires analog memories and backward integration, it has limitation for analog circuit implementation. 4.2 Classification of temporal trajectories The previous Pearlmutter’s algorithm is used for the identification of temporal trajec- tories. For the classification of temporal trajectories, Sotelino et al. [53] has modified the Pearlmutter’s algorithm. In Sotelino’s algorithm, the decision is taken immedi- ately at the end of the input signal. The output units play the same role as the 62 hidden units before the end of the input signal, and the task of the network is easier: the activation of the output units just has to pass through a point; they do not have to follow a trajectory. As in the Pearlmutter’s algorithm, the activation rule for V;- is given by TdV- d_t—— — —V- + $220014) + I (4.14) We have to minimize the functional $23M — 0.1 I.—.. + /: Ldt (4.15) 21:60 where L is the Lagrange function L= ZA(t)[r—+V.-— S(ij.-,V,—)— 1.]dt (4.16) The output units do not have to follow a desired time trajectory any more. They have to classify the signal by clamping output units on at time t1. The end—point term at time t1 is exactly the same as the one used in standard back-propagation algorithm for the output layer. The difference is that the dynamic is continuous. By the Euler-Lagrange equations, TX; = A; — ZAngwJ-g (4.17) i The end-point conditions at t = t1 are now _0_L+ 0: 8V- +8V~ %Z(V— 1),)2] (4.18) 2160 63 and we obtain Mt = t1) = —6,-(V,-(t = 11) - Di) (4J9) The gradient descent rule is obtained as same as the Pearlmutter’s algorithm. t1 862;; = 77/;0 VjSfikgdt (4.20) Am,- = -n The procedure of updating the weight is the same way in the previous section. The fundamental difference with Pearlmutter’s algorithm is the end-point condition, equation (4.19). The final value of the Lagrange parameters is no longer zero, but the difference to the target value at this time. This can be considered as the error signal that has to be back propagated in time. 4.3 Recurrent Back-propagation Pineda [54] has developed the back-propagation algorithm that can be extended to arbitrary networks. Let us consider N continuous-valued units, V}, with connection weight, w;,-, and activation function, 5'. Some of these units are used as input neurons and have input values of I.-. We define I,- = 0 if the neuron i is not the input neuron. Similarly some may be output neurons with desired values of D5. The dynamics of the network is based on the following differential equations dV; TE = —V£+S(Zwijvj)+1i (4°21) i -V.' + S(net,-) + I; (4.22) where 10.3 is a connection weight from jth neuron to the ith neuron. It is easily seen 64 that this dynamic rule leads to the right fixed point, where dVg/dt = 0, given by V.- = 5(2 w.,-V,-) + 1.- (4.23) 1' We assume that at least one such fixed point exists and is a stable attractor. An error measure for the fixed point is the quadratic one E = 12 E: (4.24) 2 k where D;c — V;c if K is an output unit; E), = (4.25) 0 otherwise Gradient descent method gives 6E _ aw. _nawr‘ — 17 2k: Eh awr’ (4.26) Awrs = On performing the differentiation in equation (4.23), one immediately obtains 0V; 620.. 0w" : S'(net,-) [6.3-V, + ngj 6V} ] (4.27) i where 6;, is the Kronecker 6 symbol. With solving for the derivatives, the result is 0V , gLijEZJ—j: = 6&5 (net;)V, (4.28) where ng = 651' - S'(net,-)w,~,- (4.29) 65 Inverting the linear equations (4.28) gives 8V)c 0w" = (L'l)k,S'(net,)V, (4.30) On substituting equation (4.30) into (4.26) one immediately obtains Aw" = 7754/. (431) where 6,. = S'(net,) Z Ek(L'l)k,. (4.32) 1: Equations (4.31) and (4.32) specify a formal learning rule. Equation (4.32) requires a matrix inversion to calculate the error signal 6,. Direct matrix inversion is not suitable for implementation of the neural networks. A local method for calculation of 6, is obtained by the introduction of an associated dynamic system. Consider the vector 2 whose components are defined in terms of the components of 6 according to 6,. = S'(net,)z, (4.33) so that z, = Z EAL-1))" (4.34) 1: Equations (4.32) and (4.33) imply that 2, satisfies 2 [4,52, = E; (4.35) Now observe that the solutions of equation (4.35) are the steady-state solution of dz,- TE 2 -- : Ln-z, + E; (4.36) 66 Using equation (4.29), 7% = —z.' + Z S'(net,)wn'zr + E.“ (4'37) Equations (4.31) and (4.33) lead to a learning rule of the form Aw" = nS'(net,.)z,V, (4.38) Equations (4.22), (4.37), and (4.38) completely specify the dynamics for an adap- tive neural network, provided that equations (4.22) and (4.37) are convergent. The whole procedure is 1. Use the activation equation (4.22) to find W3. 2. Compare with the targets to find the E53 from equation (4.25). 3. Relax the network equation (4.37) to find zfs. 4. Update the weights using equation (4.38). 4.4 Real-time recurrent learning Williams and Zipser [55] showed how to construct a learning rule for general recurrent networks that runs continuously. The network so trained is called a real-time recurrent network. The network operates at discrete time and the rule can be run on-line, learning while sequences are being presented, rather than the whole sequences are shown. Consider a network consisting of a total of N neurons and M input connections. Let x(n) denotes the M x 1 input vector at discrete time n, and let y(n + 1) be the corresponding output vector produced one step later at time n + 1. The input 67 vector a:(n) and one-step delayed output vector y(n) are concatenated to form the (M + N) x 1 vector z(n). Let A denote the set of indices 2' for which 9:;(n) is an external input, and let B denote the set of indices 2' for which 2,-(n) is the output of a neuron. We have at; n i i 6 A 2.0.) = ( l f (439) yd") if i 6 B Let W be the N x (M + N) weight matrix of the network. The net internal activation of neuron i is given by net-(n) = ngjzj(n) (4.40) yg(n+1) = S(net,-(n)) (4,41) Let d;(n) denotes the desired (target) response of neuron i at time step n. Let C denote the set of neurons that chosen to be visible neurons externally. The remaining neurons are hidden neurons. An appropriate error measure e(n) on neuron i at time step n is given by e(n) = dd") -ye(n) if i E C (4.42) 0 otherwise The instantaneous error measure at time step n is E(n) =l % E; e?(n) (4.43) :6 To minimize the error, we use the gradient descent method. For a particular weight wpq(n), we may define the incremental change Awpq(n) at time step 12 as follows: Awpq(n) = ”flaalZZ) (4.44) = nZek(n)ag:fn) (4.45) k P9 68 where 17 is the learning-rate parameters. The last derivative in (4.45) can now be found by differentiating the activation equation (4.41), 0y.-(n) 0w,” - — 1 = S’(net.-(n _ 1)) 6ipzq(n — 1) + Zwij 6.1/16(2) ) j P? (4.46) where 6;, denotes that 65,, = 1 when i = p, otherwise it is zero. It is natural to assume that the initial state of the network at time step n = 0, say, has no functional dependence on the synaptic weights; this assumption implies that 3.1/5(0) 0w“ = o (4.47) We now may define a dynamic system by a triply indexed set of variables, ngq, where i = 63M") P9 awpq (4.48) For every step n and all appropriate 2', p, q the dynamics of the system are governed by: n;q(n) = S'(net,-) 6;,zq(n — 1) + Z ngwgq (4.49) i with initial conditions 7r‘ = 0 (4.50) The real-time recurrent learning algorithm for training the recurrent neural net- work proceeds as follows: 1. For every time step n, starting from n = 0, use the activation equation (4.41) of the network to compute the output value of the N neurons. For the initial values of the weights, choose them from a set of uniformly distributed random numbers. 2. Use equation (4.49) and (4.50) to compute the variables 1r;Q(n). 69 3. Use the value of n;q(n) obtained in step 2, and the error signal e.-(n) expressed in equation (4.42), to compute the corresponding weight changes Aw..(n) = nze.(n)w;.(n) (4.51) 4. Update the weight mpg in accordance with wpq(n + 1) = "’99 + Awpq(n) (4°52) and repeat the computation. The use of the instantaneous gradient rather than the true gradient over the whole interval is analogous to that encountered in the standard back-propagation algorithm used to train a multilayer feedforward neural network, where weight changes are made after each pattern presentation. The practical differences between the real- time and non real-time versions are often slight. These two versions become more nearly identical as the learning rate 17 is sufficiently small [55]. The real-time version avoids any storage requirements and is especially simple to implement. 4.5 Hardware limitation of the real-time recur- rent learning rule At the Williams and Zipser’s algorithm, the learning rule runs on-line. The on-line version of the Williams and Zipser’s algorithm is appropriate for analog circuit imple- mentation since analog hardware usually doesn’t have memory capability. Though the learning equations of the real-time recurrent learning rule are simple and easy to implement with analog hardware, the Williams and Zipser’s algorithm requires a large number of 11' equations. 70 Table 4.1. Hardware requirements of the real-time recurrent learning rule Number of Number of Number of weights Number of 1r equations neurons, N inputs, M W = N(M + N) N2M + N3 2 2 8 16 3 2 15 45 4 2 24 96 5 2 35 175 6 2 48 288 7 2 63 441 8 2 80 640 9 2 99 891 10 2 120 1200 2 3 10 20 3 3 18 54 4 3 28 112 5 3 40 200 6 3 54 324 7 3 70 490 8 3 88 704 9 3 108 972 10 3 130 1300 2 4 12 24 5 4 45 225 10 4 140 1400 20 4 480 9600 50 4 2700 135000 100 4 10400 1040000 Consider a recurrent neural network with N neurons and M inputs. Under these conditions, the dimension of the weight matrix becomes N x (M + N). The imple- mentation of the real-time recurrent learning rule uses 1r equations (4.49) and weight update equations (4.45). The 7r equations take total N x (N x (M + N )) equations. Thus, the required number of 1r equations becomes N 2M + N 3. In table 4.1, the required number of weight equations and 7r equations for different number of neurons and inputs are shown. Nevertheless, because of its ease of imple- mentation, the real-time recurrent learning rule is used by many researchers working with small networks. 71 CHAPTER 5 Implementation of the Modified Recurrent Back-propagation 5.1 The modified recurrent back-propagation rule Let us consider N neuron recurrent neural network which is fully connected among neurons. Let y.- be the output of the neuron, ng be a connection weight from the j th neuron to the ith neuron, and 9:,- be the external inputs. Some of the neurons can be externally seen as output neurons, and some neurons receive the external inputs. The dynamics of the network in the Pearlmutter’s algorithm is expressed as equa- tion (4.1) 7.1—4 = -315 + 3(2 212,331,) + I.- J = —y,- + S(net,-) + I,- We propose the equivalent discrete model to the recurrent neural network as y,-(t+At) = S(net,—(t)) (5,1) 72 73 = 5(2 win/1'0) +4.0» (5.2) where 5' means the sigmoid function, and 2:;(t) represents the input value which may be zero if y.-(t) is not used as an input neuron in the network. The activation equation (5.2) is analogous to the equations in the feedforward neural network. In the Pearlmutter’s algorithm, the error is defined in the integral form as equation (4.3) 1 ‘1 E = 5/ Elf/k - dk]2dt ‘0 1:60 The error measure of the proposed network is defined at instantaneous time t such as E(t) = g; 6:“) + $2 2: Claw?)- (5.3) = $2140) - 4.0))” + $224.44 (5.4) where [C denotes the output neuron, dk(t) is the desired value of the output neuron k, and 0.3 is a small positive constant, which is damping factor. In the Pearlmutter’s algorithm, the following two equations (4.11) and (4.13) specify the learning rule. TX; = 61¢(yk — (1);) + A; - Z Ang-wj; (5.5) j 31 n , Aw,‘7 — -178ng — ”/40 y,S,-A,dt (5.6) We propose the modified recurrent back-propagation update law with equa- 74 tions (5.7) and (5.8). 1'; digit) = —z,(t) + Z: wjng-(t) + ek(t) (5.7) Lu?)- = nzs(t)yj(t) - mews-(t) (5-8) In the Pearlmutter’s algorithm, the forward integration from to to t1 is used in the weight update equation (5.6). If we consider that the weight update law changes the weights only at instantaneous time t, not at the interval from to to t1, the weight update equation becomes equation (5.8) with damping factor a. The modification from equation (5.6) to equation (5.8) is based on the derivation in Section 2.3.2. We use the continuous-time gradient descent method for modification and we re- move the sigmoid derivative term. In the implementation of the modified recurrent back-propagation update law, we set the r] parameter as small as possible and the integration is obtained by averaging of the weight value through time. Since we start the learning with arbitrary initial conditions and continue to learn the target with periodical waveform, the A equation (5.5) is modified to the 2 equation (5.7). For analog hardware implementation, the backward integration is impossible. We employ only forward instantaneous update in the 2 equation. Also, this updating rule is the special case of the Pineda’s algorithm. The Pineda’s algorithm assumes the activation equation is convergent and the error measure is cal- culated in the fixed point. The update rule at the modified recurrent back-propagation is the on-line version of the Pineda’s algorithm. It updates the weights at instanta- neous time t. The activation equation at neurons gives instantaneous response to the network at time t. Instantaneous error is measured at time t and used to update the weights, while the time-varying inputs and time-varying targets are changing with time. 75 Since we have showed the successful operations of the feedforward neural network with the modified algorithm that does not have the derivative of non-linearity term [13] [28] [47], the derivative of the non-linearity term is omitted. The omission of the derivative of the non-linearity term makes the analog hardware implementation simple. 5.2 Stability of the modified recurrent back- propagation If we do not guarantee that the network will be stable, it may not perform the task we wish it to learn. We need to investigate the stability of the learning rule. To consider the stability of the recurrent neural network with fixed point learning, we must consider the stability of the dynamic equation of the system and the 2 equation. It has been shown previously by Almeida [56] that if the dynamic equation of the system is stable, then the 2 equation is stable as well. Other stability requirements for the dynamic equation of the neural network deal with constraints on the weight matrix. Hopfield [21] has guaranteed that if the weight matrix is symmetric, he could find a Lyapunov function and global asymptotic sta- bility. We can not ensure that our weight matrix will be symmetric. Other type of constraints [57] [58] deal with the size of the weights in the network relative to the maximum gain of the sigmoid function. However, they consider only static patterns. The key pattern is given as a constant input to the network. There is no report on the stability analysis of the recurrent neural network when the pattern is a dynamic (temporal) pattern. In the modified recurrent back-propagation learning rule, we have two basic equa- 76 tions for the learning rule such as u30(1) = nze(t)yj(t) - Wow“) (59) 54(1) = iji(t)zj(t) + ek(t) - 4240) (5.10) Note that we have inserted the ,6 parameter to investigate the stability of the 2 equation. In equations (5.9) and (5.10), the damping factor a and [3 control the stability of the equations. We have simulated the learning rule in MATLAB. The network is given as 4- neuron neural network with threshold weights. Thus there are total 20 weights, 16 connection weights and 4 threshold weights. The experiment has two inputs and one target. In this experiment, two input values are given as 2:1(t) = sin(0.5 x 1r x t) (5.11) 3:2(t) = sin(0.25 x 1r x t) (5.12) The target value is given as d1(t) = 541(4) + an» (5.13) With input and target values, the differential equations (5.9) and (5.10) are solved with the activation equation, yi(t + At) = “In“: w,,-(t)y,-(t)) 311 312 313 114] “’10 w20 tbso lbw = tanh wu “’21 war "’41 \' dngu) 1012 1022 W32 I042 1010 wzo W30 1040 the weight update equation, dt 1013 7-023 was 1043 "’11 1021 W31 1041 1014 W24 tbs. 1044 1012 um was 1042 77 U213 1023 was 1043 We assume all Clij has the same quantity 0. 21 22 23 w" "’24 W34 1044 wlo wzo wso w4o b 311 312 313 314 = 712i(t)yj(t) - 1701050) ’ [1311 Wm 1021 1031 1041 d $1 32 d 1 312 313 314] 1012 11’22 w32 11142 "113 1023 was w43 W14 1024 1034 1044 (5.14) where tanh is a hyperbolic-tangent function. There are 16 differential equations for (5.15) and 4 differential equations for the 2 equation, ng(t) T’ dt = 244.240) + e.(t) — 324:) Table 5.1. The MATLAB simulation results 78 ] y(learning rate) a(weight update) 6(2 equation) Results ] 0.50 0.050 0.20 infinity 0.50 0.050 0.30 infinity 0.50 0.050 0.35 infinity 0.50 0.050 0.40 converge 0.50 0.050 0.50 converge 0.05 0.050 1.00 converge 0.10 0.050 1.00 converge 0.30 0.050 1.00 converge 0.50 0.050 1.00 converge 0.70 0.050 1.00 converge 0.90 0.050 1 .00 converge 1.10 0.050 1.00 converge 1.20 0.050 1.00 infinity 1.30 0.050 1.00 infinity 0.05 0.025 1.00 converge 0.10 0.025 1.00 converge 0.30 0.025 1.00 converge 0.50 0.025 1.00 converge 0. 70 0.025 1 .00 converge 0.80 0.025 1.00 infinity 0.90 0.025 1.00 infinity 0.10 0.010 1.00 infinity 0.20 0.010 1.00 infinity 0.30 0.010 1.00 infinity 0.40 0.010 1.00 infinity 0.50 0.010 1.00 infinity 31 “hi w21 wal w41 21 61 21 22 “’12 w22 w32 wt: 22 0 22 72. = - fl is “’13 was was was 23 0 23 _54. _w14w24w34w44sz4J L0‘ ‘24] (5.16) In MATLAB simulation, we have changed the parameters 7), a, and 6 to investi- gate the stability. The simulation results are summarized in Table 5.1. From the simulation results, the stable operation of the network is determined by the damping factors. If the 0: parameter and the 6 parameter are not big enough to 79 make the equations stable, the to values and the 2 values increase to infinity. The first set of data in Table 5.1 shows the stability dependence on the B parame- ter. With the fixed value of 17 = 0.5 and a = 0.05, the small 6 value drives the neural network into infinity values of weights and 2 values. The weights have converged to the constant values with ripples with B = 0.40, however, the weights have diverged to the infinity value when 6 = 0.35. The rest of the results in the table are obtained with the fixed value of 6 since equation (5.8) implies the B parameter is 1. From the results in the table, we can see that the small value of a, such as a = 0.01, makes the unstable operation in the neural network. Another observation on the a parameter suggests that a large a parameter is not desirable for successful learning. The large a parameter ensures the convergence of the weights. However, the large a parameter tends to drive the weights to near zero since the damping factor dominates the weight update equation. The 17 parameter usually deals with the speed of the learning equation. The large value of 7) causes the divergence of the weights and the 2 values. Once the differential equations are stabilized, the n parameter determines the success or failure of the learning. If the learning rate (7]) is not sufficient small, the ripple in the converged weights becomes significant since we are using the instantaneous learning in continuous time. To get the constant weight in learning phase, we need to have a very small 1] value. In the learning phase, we usually take the averaged value of the oscillating weights. If the weights are converged with large oscillation, we can not decide the constant weight values. From Figure 5.1 to Figure 5.4, we display four simulation results in Table 5.1. In Figure 5.1 and Figure 5.2, the effect of the a parameter is shown. The large a parameter causes the large magnitude oscillation of the converged weight. In Figure 5.3, it is shown that the constant converged weight can be obtained with the small 1] parameter. Figure 5.4 shows the diverged weights to the infinity when 1) has a large 80 Wow eta-0.5 alpha-0.025 beta-1.0 T r T ""“WW V‘\"‘IIIII‘UII Figure 5.1. The MATLAB simulation result: the converged weights example wm eta-0.5 chm-0.050 bob-1.0 (16 r’ w . V T ! I r I iii-W NI I III 04 j 1 x l I liaiiv'HWi‘Mdi‘z‘mmqu6666161166661avivivivtivmvm * ..1vvwvvvvvv) )11 11111 vvvvvvvvvrvvvvvvvvwviw ] ........ h ............................................................................. q )1 ‘ - ..... ......... a \‘I . . 4 4 n I n | \ t l \ t t t 1 111 I | t | \ 1 | l,| u \ I' ] ttta2i.“i{§§§§hf~\{~\\£a\\\\\\\\\\\\\\\\s\\~\\~~ t ................... i .......... 2 .......... . .......... i ......... i .......... i ........ .4 I”, "r ' .............................................................................................. ., 4‘ l l l J l l I i i 0 50 100 150 200 250 300 350 400 450 500 time Figure 5.2. The MATLAB simulation result: the converged weights example 81 W MA alpha-0.025 but-1.0 I V ....................................................................... .1... ~....o.......an----1.........---~~- ....................... .‘ l l l L i l l ‘0 0 500 1000 15“) 2WD 2500 3000 3500 4G!) Figure 5.3. The MATLAB simulation result: the converged weights example Wow m-os alpha-0.025 beta-1.0 1.5 I T l I I l I _1 .. .................................. .1 -1.5 l L L j 1 L l 0 50 100 150 200 250 300 350 400 m Figure 5.4. The MATLAB simulation result: the diverged weights example 82 value. 5.3 Implementation of the modified recurrent back-propagation The recurrent neural networks with learning capability are designed using the mod- ified recurrent back-propagation learning rule. This implemented recurrent network can be divided into two parts: a recurrent neural network and a learning network. Suppose that the recurrent neural network has four neurons. It is fully connected including self-feedback. There are 16 connection weights and 4 threshold weights in this recurrent neural network. The recurrent neural network is shown in Figure 5.5. Let us suppose that the recurrent neural network has two inputs, 2:1 and 2;. These inputs are connected through the neuron, m, and 314. Two neurons, yl and yg, will be visible with the target signal, d1 and d2. wgo denotes the threshold weight and 20,-, is the connection weight from the j th neuron to the ith neuron. The activation equation is given as , , r. . m . .1 yr “’10 mil w12 1013 um 0 5'1 312 wzo W21 1022 wzs 1024 , 0 = S y.2 + Vlogzcl (5.17) 313 wso war 1032 was 1034 $1 313 314 1040 1041 1042 w43 11744 a 1 $2 . ' ‘ 1' _ m . 1 In circuit implementation, activation equations (5.17) are implemented using the 6- dimensional vector multiplier. There are 5 multiplications between the weight matrix and the input vector and a multiplication for input. If there is an input signal to the neuron, the activation equation is implemented using whole 6 modified Gilbert 83 input], x, input '2, x2 w w w w w out, i” 14% I3 12 i u e to S , gwu $323 ngz $1112, “'20 S 01:2 .4» f» .w» .w” W w w w w w x2 I A 44 43 42 41 40 VIII Figure 5.5. The recurrent neural network with four neurons and two inputs, 5' means the sigmoid function generator 84 multiplier sub—circuits. If there is no input to the neuron, the 6-dimensional multiplier uses only 5 modified Gilbert multiplier sub-circuits by connecting the input terminal to the reference voltage, 2.5V. If there is no + or - sign in the dotted pair, V2 and V4 are 2.5V. Vth denotes the threshold voltage which is chosen to be 3.5V and Vlogicl has been chosen as the role of identity 1 in multiplication. Since the high saturated voltage of the sigmoid function generator is 3V, Vlogicl is selected to be 3V. The sigmoid function generator is designed to have the saturated voltage as 2V and 3V. The gain and the voltage characteristics are shown in Section 3.2.4. The bias voltage is selected as 1.1V. The learning network is implemented using weight update equations (5.8) and 2 equations (5.7). There are 20 differential equations for the weight update equation, I- q I- 1 “’10 wu w12 wls w14 21 wzo w21 w22 w23 w24 22 o o o o o — n Vt h yl y2 y3 y4 wso wsl w32 wss ws4 Zs w4o w41 w42 w4s w44 L Z4 "’10 wn w12 wls w14 wzo w21 w22 wzs w24 _,,a (5.18) wso wsl wsz wss ws4 w4o w41 w42 w4s w44 Weight update equations (5.18) are implemented using l—dimensional multiplier since the term no: is implemented using active resistors and capacitors. The learning rate, 17 is related with the capacitor value and the damping factor, a is related with the conductance of active resistors. Active resistors of the 1-dimensional multiplier uses cascaded PMOS as shown in 3.5 since the small a parameter is required for successful learning. 85 There are 4 differential equations for the 2 equation, u- 1 F q I- - 1- - I- -1 21 wn w21 wsl w41 21 31 21 52 1012 1022 w32 w42 22 , 82 22 1-2 = + Vlogzcl — Vbeta (5.19) 23 "’13 wzs wss w43 za 0 23 24 w14 w24 ws4 w44 Z4 1 0 24 L d h u L c u L d The differential equations for the 2: equation (5.19) are implemented using 6- dimensional vector multiplier. The 6—dimensional vector multiplier can be used as a fidimensional vector multiplier or a 5-dimensional vector multiplier depending on the existence of the target signal. If the neuron is not used as an output neuron, there is no target signal to receive. In the case of non-output neuron, the e.- is set to zero by connecting Vl 2 V2 together to prevent producing the multiplication since V1 is connected to the target signal and V2 is connected to the output signal. We have tried to implement the damping factor (the last term, -z) in the 2 equation using the active resistors. We have found that the use of the active resistors can not guarantee the stable operation in the PSPICE circuit simulation. We realized the damping factor using the negative self-feedback. Using the negative self-feedback, we can control the stability of the neural network with the associated voltage to the negative self-feedback. The associated voltage to the negative self-voltage is labeled Vbeta. Vbeta is selected to 3V. If we reduce this voltage to 2.7V or 2.6V, then the circuit operation turns to be unstable. When Vbeta is 2.7V, the weight values and the 2 values usually hit the rail voltage (5V or GN D). Vlogicl has been chosen as the same role of identity 1 in multiplication. It is selected to be 3V. In an analog circuit implementation, different time constants are used for equa- tions (5.17), (5.18), and (5.19). The time 3] and z spend settling is negligible compared to the rate of weight change 17. The learning rate (17) is designed to be slow compared to the speed of presentation of new training samples. 86 The circuit block diagram of the recurrent neural network with modified recurrent back-propagation learning circuit is shown in Figure 5.6. In this diagram, 6D denotes 6dimensional vector multiplier. S means the sigmoid function generator. y.- represents the state of neuron, z,- is the state of 2 equations, and 11);, is the connection weight from the j th neuron to the ith neuron. In the vector multiplier, the usual multiplication occurs in the form of 2,-(V1 - V2) x (V3 - V4). The dotted lines inside the vector multiplier show the multiplication pair. If V2 and V4 are Vref = 2.5V, (1:1 — 2.5V) x (yl — 2.5V) becomes 1:1 x yl since 2.5V is the virtual ground term. Vlogicl has been chosen as 3V and Vre f is the reference voltage which is 2.5V. Vth supplies the threshold voltage which is chosen to be 3.5V. Vbeta in the implementation of 2 equation is selected to be 3V. 5.4 Simulation results of 4 neuron recurrent neu- ral network 5.4.1 A circular trajectory generation We have trained a recurrent neural network with no input units, two hidden neurons and two output neurons. The neural network has fully connected weights. It has 20 weights, 16 connection weights and 4 threshold weights. Since the neural network has no inputs, all input terminals are connected to the reference voltage, 2.5V. It is trained to follow a circular trajectory. To learn the circular trajectory is the well-known problem [14] [50]. The desired state (I; and d; are plotted against each other in Figure 5.7. The target trajectory consists of a sine waveform and a cosine waveform. The trajectory is given as the continuous waveform to the neural network. We have run the PSPICE transient analysis. We performed the PSPICE circuit simulation using the M0818 2.0pm CMOS parameters. The PSPICE parameters are 87 “11'21'31'41 'Iz'n'n'u Figure 5.6. The block diagram of recurrent neural network with the modified recurrent back-propagation: four neurons and two inputs '88 Circular Trajectory Figure 5.7. The circular trajectory shown in Appendix A.1. We apply the target signal to the neural network and observe the output waveform and the weight waveforms. If the output is following the target closely and weights have converged to the constant values, we take the averaged value of the weights. The obtained weight values are used as a verification of learning. After learning, we set the weight values to the neural network and see the generation of the trajectory by the neural network. If the neural network generates the circular trajectory, the learning is successful. We found that the randomly initialized weight values didn’t fail to learn the trajec- tory. We have performed several experiments with different frequencies of waveforms, different values of capacitance, and different gains of active resistors in the weight update equation. The results of the experiment are summarized in Table 5.2. In Table 5.2, (W/Lmfifl'fio, denotes that (W/ L) ratios of 4 PMOS transistors in cascaded active resistors of l-D multiplier for weigh update equation. The multiplier with (W/L)=4/64 has higher gain than that of (W/L)=4/16. From the table, high gain of active resistor will be desirable for successful learning. High gain of the active 89 Table 5.2. The simulation results of the circular trajectory experiment (W/ Lmfiififor Target Cw C 2 Result Remark 4 / 64 1MHz 20pF 1pF fail 4 / 64 1MHz 50pF 1pF fail 4/ 64 1MHz 100pF 1pF fail 4 / 64 500KHz 20pF 1pF success 4 / 64 500KHz 50pF 1pF success example 1 4 / 64 500KHz 100pF 1pF success 4 / 64 250KHz 20pF 1pF small amplitude 4 / 64 250KHz 50pF 1pF success 4 / 64 250KHz 100pF 1pF success 4 / 64 lOOKHz 20pF 1pF fail 4 / 64 lOOKHz 50pF 1pF small amplitude example 3 4 / 64 100K Hz 100pF 1pF success 4/ 16 1MHz 20pF 1pF fail 4/16 1MHz 50pF 1pF fail 4/16 1MHz 100pF 1pF fail 4/16 500KHz 20pF 1pF small amplitude 4 / 16 500KHz 50pF 1pF success example 2 4 / 16 500KHz 100pF 1pF success 4 / 16 250KHz 20pF 1pF fail example 4 4/16 250KHz 50pF 1pF small amplitude 4 / 16 250KHz 100pF 1pF success 4/16 lOOKHz 20pF 1pF fail 4 / 16 lOOKHz 50pF 1pF fail 4 / 16 ' lOOKHz 100pF 1pF small amplitude 90 resistors implies the small value of the a parameter. From the table, Cw means the capacitor value of the weight update equation. In the weight update equation, a larger value of the capacitor implies a small learning parameter 1]. When the learning parameter is not small enough, the weight has a ripple in its converged value. Small capacitance causes oscillatory behavior on the converged weight values and makes the determination of the final constant weight difficult. With the small capacitance, it is observed that the output waveforms usually follow the target waveforms. However, the weight is oscillating and not approaching the constant value when the capacitance is small. A large value of capacitance (small learning rate) is needed for near constant value of the weights. When the target signal is very fast trajectory such as 1M H z, the learning is not successful. The reason is that the circuit can not respond to the high frequency tar- get. In this experiment, a target around 500K Hz has the best results. With 250K Hz target signal, the weight has a small oscillation in its converged value. Usually, we take the averaged value in the learning phase when the weights have oscillation. If the peak-to—peak value of the oscillated value is not small enough, the taken averaged value can not generate the exact trajectory. It results in a small amplitude trajectory generation. In Table 5.2, the Result has three cases in the testing phase such as success, small amplitude, and fail. The success case generates the circular trajec- tory closely, the small amplitude case generates the circle in small magnitude. The fail case can not generate the circular trajectory in the testing phase. Both successful and failed cases of Table 5.2 are illustrated from Figure 5.8 to 5.23. We show four examples that are marked in Table 5.2. First two examples are successful cases, the third example is the case of small amplitude, and the last example is the failed case. The obtained weights of each example are shown from Table 5.3 to 5.6. In these tables, ng denotes the connection weight from jth neuron to ith neuron and 1050 91 denotes the threshold neuron at the neuron 2'. Each example consists of four figures. In the first figure, three sub-plots are shown. V(7) is the first target signal, d1, and V(8) is the second target signal, d2. V(1) represents the actual output signal, yl, and V(2) represents y;. The second sub- plot shows d1 and y1, and the third sub-plot shows d; and y;. The trajectory shape is illustrated in the first sub—plot in the figure. The transient trajectory is shown in V(1) versus V(2) plot (yl versus yg plot). In the second figure, the error measure is shown in the first sub-plot. The error measure is given as Erma = $1/(d1 — 311)2 + (d2 — 312)2 (530) The value Em, has the sense of a root-mean-square normalized error. The second and the third sub-plot show the weight values. The weights are labeled as w.-,- = V(z' j) and wgo = V(6i). The final weight in the learning phase is obtained by averaging each weight at the end of simulation. The third figure and the fourth figure show the results in the testing phase with the weights that are obtained in the learning phase. Two experiments with different initial conditions are shown in the third figure and in the fourth figure. The third figure shows the test result when the initial is inside the circle and the fourth figure shows the test result when the initial condition is given at the outside of the circle. The first sub-plot in each figure shows the state trajectory. Except the failed case, all test results (two successful cases and a small amplitude case) show that the circular trajectory is a limit cycle. The neural network has learned the stable oscillation and generated the circular trajectory irrespective of the initial state. Table 5.3. The averaged weights in the learning phase from the PSPICE transient analysis: example 1 92 J' l 2 3 4 2021' 2041' wlj 1031‘ 2.7844 2.5706 2.5279 2.5285 2.4111 2.7708 2.5338 2.5313 2.5234 2.5674 2.5253 2.5248 2.5264 2.5619 2.5254 2.5252 2.5276 2.5262 2.5339 2.5338 Table 5.4. The averaged weights in the learning phase from the PSPICE transient analysis: example 2 j l 2 3 4 wlj 1021' 2031‘ 1041' 2.7751 2.5901 2.5188 2.5190 2.4051 2.7702 2.5223 2.5208 2.5147 2.5427 2.5152 2.5150 2.5164 2.5401 2.5152 2.5150 2.5286 2.5309 2.5225 2.5226 Table 5.5. The averaged weights in the learning phase from the PSPICE transient analysis: example 3 j l 2 3 4 Wu 1021' 1031‘ 1043' 2.7452 2.5772 2.5292 2.5294 2.4099 2.7536 2.5306 2.5289 2.5296 2.5614 2.5250 2.5248 2.5305 2.5587 2.5252 2.5249 2.5215 2.5276 2.5344 2.5342 Table 5.6. The averaged weights in the learning phase from the PSPICE transient analysis: example 4 j l 2 3 4 Wu 1122, 1031' 1041' 2.7452 2.5772 2.5292 2.5294 2.4099 2.7536 2.5306 2.5289 2.5296 2.5614 2.5250 2.5248 2.5305 2.5587 2.5252 2.5249 2.5215 2.5276 2.5344 2.5342 93 'mmmwz4ma-M):mu .r """""""""""""""""""""""""""""""""""""""""""""""""""""""""" 1 :- / / ‘- / / . ~ r x I m L .............................................................................................................. .' IW0) 0V0) ”V .r """""""""""""""""""""""""""""""""""""""""""""""""""""""""" 1 1.5V l‘ -------------------- r -------------------- w- ------------------------- fi --------------------- ‘l 1m 1m 1001. 1m 2001. am we The Figure 5.8. Example 1 (the first figure): Input and target signals 'mmmeWad-nmsu Wilma/1m comm Taiwan: 27.0 ' ' v 1.. ”42" v V V "I ' '- ‘ 1 I”; gnu." . fl," 9W") "'02) ‘VUS 'W“) °V(21) *Vm I"N2!” "“20 “'91) “K3” h m 100m 150m 2001. 0m WW) Him) 'VM HIM) WW) ”(01) WM ‘VM ”(64) Tina Figure 5.9. Example 1 (the second figure): Em, and the weights 94 °mmmwmmmdymu menm M1013 7W: 27.0 1W 1‘ """"""""""""""""""""""""""""""""""""""""""""""""""""""""""" 1 : : l l i E I l 2.0V I; """"""""""""" v """"""""""" 1 °°°°°°°°°°°°°°°°° °r ------------------ r --------------------- ‘l W 1W 2W 3.0V 4W 5W “M V") 3.5V 1' """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""" '1 I l I I m '. .............................................................................................................. : 9V") 3.5V r """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" 1 O I 1.5V l- """"""""""""""" v -------------------- 1 -------------------- r ------------------- r --------------------- 4 o- 21» «I Q. I. 10.- ovm The Figure 5.10. Example 1 (the third figure): Test phase with the initial condition inside the circle 'mmmwuma-I-I):mu acumen-comm M1305 TWO: 27.0 W T """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" '1 i : 5 E 2W ‘- -------------------- 1 -------------------- 1 ---------------------- r --------------------- r --------------------- ‘l W 10V 20V 3.0V 40V 50V "V(2) V(1) 3.5V r """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" 1 i : 1 I ,W i. .............................................................................................................. 3 0W1) 15V r """"""""""""""""""""""""""""""""""""""""""""""""""""""""""" 1 l I 1 1 1.5V V """"""""""""" v -------------------- a -------------------- r --------------------- r --------------------- 1 8 z a a r E Figure 5.11. Example 1 (the fourth figure): Test phase with the initial condition outside the circle 95 'mmmw (2.1-1).souceu-50p:m when-curl“ 13:41:31 TW:21.0 2W 4* -------------------- I ----------------- 1 ---------------------- r --------------------- r --------------------- 4 W 1.0V 4.W 5W ”VQ V(1) 3.5V r --------------------------------------------------------------------------------------- 1 5 E - r r v r r r r r 5 E i . : ~ ’ ‘ ‘ ~ ‘ ~ ~ ‘ : U 0 I I 0 I O I m L .............................................................................................................. ,- °V(7) 01m) 2.0V - - --------------------------------------------------------------------------------- 4 0W") "1112) “'03) 'W“) ovrm ‘WM "((23) 'VW) *Vt31) "((32) QWT ---------------------------------------------------------------------------------- 1 """"""""""""""""" r--"'--"-'--"--------'---r"-"-"-"'"""'""""I"""""°"-"”"uuu'i 1501- 200- m 501- 1 “1m “((3‘) HR") "mm ““43 ”1144) 'W“) "mm ‘W 'W“) The Figure 5.13. Example 2 (the second figure): Em, and the weights 96 'mwmw:4mmd)zm.¢ m an: M780 04:55:05 -00--0---- IIIIIIIIIIIIIIIIIIII ---,----------------------r--- 10V 20V -- ---- -, - - -- - tOV V(1) r--------------.----...---------..--------------------------.---------------.----------------------...--------- 35V 15V L-......-...---....c.-.-.---.-- n V(1) - -----,-----.- GUI --,-.--- ‘0' Test phase with the initial condition inside Figure 5.14. Example 2 (the third figure) the circle 'MWWW:4m¢-1o1):m3 Warm 00:54:34 -,- -- - 10V 40V 20V .----.'----------------------r-------- ova) +- -- -- -------- 0V \K1) 1 4' on. 1 --,-- 01- -c------- - -ucoonocoooo --------1 I I I I I I I I I I I - -,.----- eu- u-1-.--------- ----,----- 200 p- 00 Figure 5.15. Example 2 (the fourth figure): Test phase with the initial condition outside the circle 97 'mmmwz4mm-t);mu Mu an: 05mm 14:10:! Figure 5.16. Example 3 (the first figure): Input and target signals 'mmmmwz4ma-1-1):mu W MI: 0917!“ 14:10:32 Tm: 27.0 o L .......................................... '3 05’W1)Mm1» 100m 150m 200m 0: 50m 0 V133) 0 V131) ‘ V141) ' W42) ° V103) * W44) 1' W") V V62) * W“) I W“) The Figure 5.17. Example 3 (the second figure): Em, and the weights 98 °mmmmmz4m¢+nzmu W MI: W10!“ 01:20:40 701mm: 27.0 IN T """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" 1 : I l : 2 I o l i s ‘ O 5 I I 5 ~ 5 2W + """"""""""""""" v """""""""""" 1 """""""""""""""" r ------------------- r ------------------- 1i W I W 2W 3.0V 4 0V 5 W a V(2) V(1) 3.5V """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" 1 ‘m I. -------------------------------------------------------------------------------------------------------------- J 0V“) 3.5V r """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" 1 1,5V P -------------- r ------------- '1- ------------- 1 -------------- 1 -------------- 1 ------------- r ----------- r """" J M an 4|. 3|. .8 1” I2“ I“ Ovm The Figure 5.18. Example 3 (the third figure): Test phase with the initial condition inside the circle 'TMWWMWMWRJJMMnI Wm W10!” 01205 Team: 27.0 '2‘ Figure 5.19. Example 3 (the fourth figure): Test phase with the initial condition outside the circle 99 'Tho WWWFW mmmnoa Wmfllflm 00:52:40 5 1 +ocooooooooccooocoooo 2W ------------------ 1 ------------------ '1 """"""""""""""" I" ------------------- r ------------------- 4 10V 20V 3.0V 4W 5W ovm V(1) 3.5V r """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" 1 . / f " / /- / /‘ /‘ / /. E v \. x, \. \r \, \ x c x. 5 E W : .............................................................................................................. : ova) ova) 3.5V r """"""""""""""""""""""""""""""""""""""""""""""""""""""""""" 1 2 / * * r /‘ A » , /~ ,~ : E v v \, x \. v \r v V 5 15V I """"""" 1' """""""""" r """"""" r """"""" ‘1' """""""" ‘1 ----------- 1 ----------- 1 --------- ‘l 1 105m 170m 175m 100m 106m 1m 195m 200m 13Vfln levy» TIIII Figure 5.20. Example 4 (the first figure): Input and target signals °mmmw+mmm 21310.1: m MI: “#17100 00:52:40 Tm”: 27.0 1.0 1' ------------------------------------------------------------------------------------------------- 1 : : : : o : .............................................................................................................. . ° ownmmm) 1W 1' -------------------------------------------------------------------------------------------------------- 1 n n A n _ A A A : r - ' L " A V A 1 1 ~ - r 3 .----..l I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I L 2W 9W") ovum “’03) "’04) °V(21) °V(22) llW23) 'VQ‘) *VG") 'W32) W (b 5000 1001: n was) 0 V(34) 4 W41) v W42) 0 W43) 9 W44) I V(01) v V(02) A V(03) I V(04) The Figure 5.21. Example 4 (the second figure): Em, and the weights 100 'mmmwu m (211) : ”10¢ W «In: MOM 01:54:55 TM: 27.0 p 2 '2 : 9.0-0...." ‘SVI- -------------------------------------------------------------------------------------------------------------- J Figure 5.22. Example 4 (the third figure): Test phase with the initial condition inside the circle 'mmmwz4mm.1)zmm W an: MW 01 :53215 7W: 27.0 3.0V 1: """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" 1 Figure 5.23. Example 4 (the fourth figure): Test phase with the initial condition outside the circle 101 5.4.2 Trajectory recognition The 4-neuron recurrent neural network is trained to learn two trajectories. There are two inputs and one output in the 4-neuron recurrent neural network. The learning circuit employs the modified recurrent back-propagation learning rule. This circuit does not include the threshold parameters. The general procedures for learning and testing are the follows: 1. In the learning phase, the first trajectory is applied to the inputs of the neural network and an associated target is also applied to the neural network. After one period of the first trajectory waveform, the second trajectory is applied to the recurrent neural network with its target waveform. We keep supplying the input-target pair to the neural network. 2. We can distinguish two trajectories by assigning the different states of the out- put waveform. the first trajectory is assigned to the high (or low) state of the target and the second trajectory is assigned to the low (or high) state of the target. 3. Each input-target pair is applied to the neural network as continuous waveforms and the PSPICE transient analysis is performed. After the transient analysis, we measure the output waveform and the target waveform. If the output waveform is following close to the target waveform, it is considered as successful learning. 4. We measure the weight values. If the learning parameter (1]) is small, the weight values are converged to the near constant value. However, the weights have some ripples in their waveform since the capacitors on the learning circuit have limited value. We measure the averaged value of the weights. 5. After the learning phase, we perform the testing phase in the recurrent neural network. We set the weights of the neural network to the obtained weight values at the learning phase. We verify the learning by applying the input waveforms 102 to the recurrent neural network. In Figure 5.24, two state space trajectories are shown. The trajectory 1 consists of V1 and V2 waveform. V1 is applied to the input 1 of the neural network and V2 is applied to the input 2. The trajectory 2 consists of W1 and W2 waveform. W1 and W2 are applied to the input 1 and input 2 of the recurrent neural network, respectively. We have performed the PSPICE transient analysis with different parameters. The results of the transient analysis in PSPICE simulation are shown in Figure 5.25 and v1 ........................... Trajectory 1 1 Y 0.5 ........................... -0.5 0 v1 Trajectory 2 _0.5 , ........................... -1 L 0 0.5 1 1111 Figure 5.24. Two state trajectories 103 ' Tho wmmm (4 man). cal-11111.25“ 0501K:11_0_2.dt Wan: WWII 10:22:00 Tm 27.0 1m 1 o vm o V(1) Figure 5.25. The PSPICE transient analysis: V(5), V(6), V(7), and V(1) are 1:1, 2:2, d1, and 311 of the recurrent neural network, respectively °mmmmw(4m).mtw.mamm_o_2u Wants/10100 10:22:00 TW:27.0 3.0V I ------------------------------------------------------------------------------------------------ 1 g i i : I I L , - - I- '- I u I . . - in5r- _ _ “777‘ wrfi‘ “ _ - . I w . FW - --- l I , s i : l : 2.0V 1 -------------------------------------------------------------------------------------------------------------- 3 ° W") ° W12) ‘ W13) ' W“) ° W21) ‘ W22) " V123) " V94) 3.0V 1' """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" 1 i I 1 I 2.0V I: -------------------------- r ------------------------- w- -------------------------- 1 ------------------------- I 00 50m 100m 1501. 20000 '3 WI”) 0 W32) 4 W33) ' V134) ° W“) 1 W42) " V143) ' W44) rm Figure 5.26. PSPICE transient analysis: weight waveforms, ng = V(z' j ) Table 5.7. The averaged values of the weight from the PSPICE transient analysis 104 j 1 2 3 4 wh- ng 1031' 1041' 2.5883 2.5132 2.4733 2.5125 2.5042 2.5131 2.5140 2.5092 2.1633 2.5148 2.6207 2.5151 2.5008 2.5136 2.5158 2.4912 Wmmm 21:21115 Figure 5.27. PSPICE transient analysis: the test result, V(5), V(6), and V(1) are the input 1, the input 2, and the actual output ' The no“! WW (4 mm): 1111_0.ci 105 'mmmwu mun): 111L050 Milan-rune!“ 21:12:44 7min: 27.0 WT """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" 1 E 1.0V 1 -------------------------------------------------------------------------------------------------------------- 0V1.) 3.5V r """""""""""""""""""""""""""""""""""""""""""""""""""""""""""" 1 f V V V V V v V ‘ 15v '1 ------------------------------------------------------------------------------------------------------------- 4 200 41- 000 000 101- 121- 1413 10m avm The Figure 5.28. PSPICE transient analysis: the test result of the trajectory 1, V(5), V(6), and V(1) is the input 1, the input 2, and the actual output 'mmmw«mm.ou Mn. nn: M0100 21:10:00 Tm: 27.0 A A A A A A A A 1.5V '1 ------------------------------------------------------------------------------------------------------------- 1 21.. It. 01- 01. 10m 133 “In 1013 0V0) The Figure 5.29. PSPICE transient analysis: the test result of the trajectory 2, V(5), V(6), and V(1) is the input 1, the input 2, and the actual output 106 Figure 5.26. In Figure 5.25, input waveforms, the target waveform of the trajectory, and the actual output waveform is shown. The weight values in the transient analysis are shown in Figure 5.26. The averaged weight values which are obtained in the learning phase are summa- rized in Table 5.7. After setting these weight values to the recurrent neural network, the testing phase is performed. The result is shown in Figure 5.27, 5.28, and 5.29. In Figure 5.27, the waveform which is used at the learning phase is used as a test waveform. In Figure 5.28 and Figure 5.29, the trajectory 1 and the trajectory 2 is applied to the neural network to test the learning, respectively. In this simulation, we use 250K H 2 signal for $1 and 500K H z signal for $2. The (W/ L) ratio of the active resistor in 1-D multiplier of the weight update circuit is given as 4/ 64. Other simulations with (W/ L)=4/ 16 have the same results as those of (W/ L)=4/ 64. The capacitor of the weight update equation is given as a 100pF capacitor and the capacitor of the 2 equation is given as a 1pF capacitor in figures. We have changed the capacitor value of the weight update equation from 20pF to 200pF in other simulations. The test results are almost same as the case of 100pF capacitor. The simulation results show that this recurrent neural network succeeds to learn the different trajectories. 5.5 Simulation results of 6 neuron recurrent neu- ral network The 6-neuron recurrent neural network with learning capability is implemented us- ing the modified recurrent back-propagation learning rule. We have performed the PSPICE circuit simulation with MOSIS 0.5pm technology for future development. The SPICE parameters for MOSIS 0.5pm HP process are shown in Appendix A.2. 107 The recurrent neural network has 6 neurons. It is fully connected including self- feedback and there are 36 connection weights in this recurrent neural network. The recurrent neural network is shown in Figure 5.30. The recurrent neural network receives two inputs, 2:1 and 32. These inputs are connected through the neurons, 3’5 and ye. Two neurons, yl and y2, are visible to outside with the target signals, d1 and (1;, respectively. output 1 output 2 Figure 5.30. The recurrent neural network with six neurons, two inputs, and one output, 5 means the sigmoid function generator We follow the general procedure for learning and testing as shown in the 4-neuron recurrent neural network. The 6-neuron recurrent neural network is trained to learn two trajectories. In Figure 5.31, two state space trajectories are shown. The trajec- tory 1 consists of V1 and V2 waveform. The trajectory 2 consists of W1 and W2 waveform. V1 and W1 are applied to 1:1 and V2 and W2 are applied to 1:; of the Table 5.8. The averaged values of the weight in the 6-neuron recurrent neural network with two output neurons 108 j 1 2 3 4 5 6 401,- 2.5481 2.4543 2.5318 2.5326 2.1609 2.7469 452, 2.4893 2.5948 2.5277 2.5280 2.7394 2.1480 w3j 2.5217 2.5201 2.5210 2.5204 2.5253 2.5213 45., 2.5210 2.5196 2.5207 2.5204 2.5250 2.5191 «.55,- 2.4830 2.5547 2.5200 2.5199 2.6287 2.3962 40.,- 2.5542 2.4545 2.5123 2.5152 2.3975 2.6496 recurrent neural network. In this simulation, we distinguish two trajectories by assigning two different states (ON/ OFF ) at two output neurons. If the recurrent neural network receives the tra- jectory 1, one of the output neurons will be ON state. If the trajectory 2 is applied to the input of the neural network, the other output neuron will be ON. We have performed the PSPICE transient analysis. The results of the transient analysis in PSPICE simulation are shown in Figure 5.32 and Figure 5.33. In Figure 5.32, input waveforms, the target waveform, and the actual output of the neural network are shown. The actual output waveform tries to match the target waveform. The weight values at the end of simulation are shown in Figure 5.33. We measure the averaged values of the weights. The averaged weight values are summarized in Table 5.8. After the learning phase, we perform the testing phase. We set the weights of the neural network to the obtained weight values at the learning phase. We verify the learning by applying the input waveforms to the neural network. The test result is shown in Figure 5.34, 5.35, and 5.36. In Figure 5.34, the waveform which is used at the learning phase is used as a test waveform. In Figure 5.35 and Figure 5.36, the trajectory 1 and the trajectory 2 is applied to the neural network to test the learning, respectively. 109 Figure 5.31. Two state trajectories: the two output neuron case on 6-neuron recurrent neural network 110 'NWMWCWM:Q_IWP§A WMMIIII 00:0021 Twat: 5W 1" -------- - ----- 1 w: ............................................................................................................ .' 0W7) 5.0V 1" -" -------------------------------------------------------------------------------------------- 1 i i w: .............................................................................................................. J ova) IN 1' ------------------------------------------------------------------------------------------ 1 f fl A _ _ A Q i | 1 g U U V E L ........................................................................................................... J DWI) 0V(1) 5.0V 1' """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" 1 i ' fl (1 A ' I I . "'L \J“ = V M; w+ ----------- r """"""" r --------- '1- """""""" 'r """"""" 1 ---------- 1 ------------- 1 ----------- 'I 1 170m 1000: 10400 10000 1021- 10000 2001- Q. 1 0 V(134) 0 V(2) Figure 5.32. The PSPICE transient analysis: V(7) and V(8) are 3:1 and 2:2, V(9) and V(1) are (11 and y1, V(134) and V(2) are d; and y2 'MWWWIWM:QJM&A Wmmmm 7-Way» "5v L ............................................................................................................. J D W“) O V02) 4 V03) V W“) 0 V(15) 0 W16) I V(21) V V(22) A V(23) 3 W24) 0 W25) 0 W28) 254V 1' -------------------------------------------------------------------------------- 1 ”a, ; $7 :31 m .I. ............................................................................................................. II 0 W31) ’ W32) 4 W33) ' W34) ° V135) 1 W33) ' W41) " W42) A W“) ' W44) ‘3 V145) ° W“) 3.0V 1 ---------------------------------------------------------------------------------------------------- 1 : I {K I m ¢—-—-—\ -\__ x v 2;:— F—LT—tdr A W 2.0V I """""""" r """""" 1 ----------- 1 ------------- r ---------- r ----------- 1r ------------ 1 ------------ I 1041. 1000- 1001- 1”. 104m 1m 1m 2001. 102:- 0V1") "KM “1153) "1(9) °V155) ”/1501 I“WM 'V182) *1/03) “((04) I“V(GES) 0W“) The Figure 5.33. PSPICE transient analysis: weight waveforms of the 6-neuron recurrent neural network with two output neurons, 20,-, = V(z' j ) 111 ‘WWWWIWM-fl mum 03_1) When-I: 0311000 0420:10 Tm 27.0 am I --------------------- ----------------------------------- 1 I 1 Figure 5.34. PSPICE transient analysis: the test result of the 6-neuron recurrent neural network, V(7), V(8), V(1), and V(2) are 2:1, 0:2, yl, and y2, respectively 'mmmwcma-z-azmtmag) WM: 031000 ”27:00 7W 27.0 5.0V? ----------------------------------------------------------------------------------------- 1 Figure 5.35. PSPICE transient analysis: the test result of the 6-neuron recurrent neural network, V(7), V(8), V(1), and V(2) are :01, 2:2, yl, and 312, respectively 112 'MWWWIWM:wm¢L1) Figure 5.36. PSPICE transient analysis: the test result of the 6-neuron recurrent neural network, V(7), V(8), V(1), and V(2) are 2:1, 0:2, yl, and y2, respectively As shown in the figures, the output of the neuron yl always goes to high state and the output of the neuron y; goes to low state when the trajectory l is applied to the neural network. Since we take the averaged value of the weights, the actual output waveform in the testing phase is a little different from the output waveform in the learning phase. However, it is very close to the output waveform in the learning phase. The result shows that the approximation of the averaged weight value is quite acceptable in the test results. If the output waveform of the testing phase is not saturated enough to classify the input waveforms, we can attach the buffer amplifier at the output of the recurrent neural network. The buffer amplifier can be a comparator or a double inverter with the threshold voltage of 2.5V. In this simulation, 250K H z and 500K H z sine waveforms represent the state tra- 113 jectory. The capacitor of the weight update equation is implemented using a 200pF capacitor and the capacitor of the 2 equation is assigned to lOpF. In the testing phase, we have applied the input trajectory that is not one of the learned trajectories. In this testing simulations, we applied the circular trajectory with different frequencies. The circular trajectory is shown in Figure 5.7. It consists of the sine waveform and the cosine waveform with same frequency. The test results are shown in Figure 5.37 and Figure 5.38. With the circular tra- jectory of 500K H z, the neural network classifies it as a trajectory 1. The simulation result is shown in Figure 5.37. With the circular trajectory of 250K Hz, the neural network can not classify the input trajectory. The output states of the neural net- work is changed as the input trajectory is applied. The simulation result is shown in Figure 5.38. Futher exhaustive testing is needed to characterize the behavior for other arbitrary signals. 'WWWWImQ-z-azhaLm-Lu W IIII: 0005/00 00:02:37 Tm: 27.0 5.0V T -------------------------------------------------------------------------------------------- 1 : I w : .............................................................................................................. 0 V(7) 5.0V 1' ---------------------------------------------------------------------------------------- 1 Figure 5.37. Testing phase with the circular trajectory: when the circular trajecroty is 500K H z 114 'NMWWImm-azm1maJ) Wmm 05:54:00 TW:27.0 INT """"""""""""" - """"""""""""""""""""" 1 Figure 5.38. Testing phase with the circular trajectory: when the circular trajecroty is 250K H z 5.6 Hardware implementation considerations 5.6.1 Hardware requirements of the modified recurrent back-propagation learning rule The advantage of the implementation of the modified recurrent back-propagation learning rule over the modified recurrent real-time learning rule is that the modified recurrent back-propagation learning rule requires less learning circuitry. In section 4.5, we have investigated the hardware requirement of the real-time recurrent learning rule. The number of the learning circuitry increases in order of N 3 as the number of neuron is N. However, in the modified recurrent back-propagation learning rule, it needs only N equations for the 2 equation when the number of neuron is N. The 115 modified recurrent back-propagation learning rule is an efficient and economical way to implement the recurrent neural network. 5.6.2 Offset voltage adjustment In the realization of the multiplier circuit, there is an offset current about lpA. If active resistors have larger resistance, this offset current has become a significant offset voltage. To compensate the offset voltage, we usually change the W/ L ratio of active resistors. The W/ L ratios of the upper PMOS are adjusted with the W/ L ratios of the lower PMOS. If the offset voltage is lower than 2.5V, the resistance of the lower PMOS is increased by decreasing its W / L ratio, or the resistance of the upper PMOS is decreased by increasing its W/ L ratio. If the offset voltage is higher than 2.5V, the opposite way is performed to get the near zero offset voltages. 5.6.3 The learning rate The learning parameter r] is related with the capacitor value in the learning circuit. In the PSPICE circuit simulation, the value of the capacitor is given from 101) to 2001). With this value, on-chip implementation of the capacitor will not be appropriate. If the network is small and the chip has enough pins to connect each weight value to outside, the external capacitors will be used. If the capacitor value is the main concern for design, we need to develop the multiplier as a low current multiplier. In PSPICE circuit simulation, the output current of the multiplier ranges from OpA to 50pA. If we reduce the current level to the nano-A range, the capacitor value of several pico-F will be enough. One way of achieving the low-power multiplier is to use the sub-threshold design [38]. However, the design of a low-power multiplier includes several factors to be considered. The matching problem of the transistors, the poor linearity of the multiplication, and the slow speed of the circuit need to be overcome 116 in the sub-threshold design. 5.6.4 Weight refresh In order to provide the two phases (the learning phase and the testing phase), the neural chip has two modes, the learning mode and the test mode [27] [47]. The neural chip executes the learning mode with its input-target pair and the weight values are taken by the Analog-to—Digital (A/ D) converter with external interface. In the testing phase, the obtained weights are written onto the capacitor which holds analog weight voltage using Digital-to—Analog (D/ A) converter. The weight values have to be refreshed since the capacitor always discharges. The interface circuit for A / D converter, D / A converter, and refresh circuit is costly. If we employ the second recurrent neural network (the slave neural network) whose weight values are taken from the first neural network (the master neural network), we don’t need the refresh interfaces and the operations of two modes. The weight voltages are transferred through the voltage follower. The weight averaging also is achieved through the low-pass filtering with voltage follower. The slave neural network can be used for any testing works or any applications as long as the master neural network executing the learning tasks with its continuous input-target pair. 5.6.5 Temperature effects The temperature dependence of CMOS components is an important performance characteristic in analog circuit design. The PSPICE simulations of the recurrent neural networks are valid only for limited ranges about room temperature. In the PSPICE simulation of the modified Gilbert multiplier and the sigmoid function gen- erator with different temperatures, the change of the characteristics due to the change of temperature is much larger in the simulation of the sigmoid function generator. If 117 we tune the bias voltage of the sigmoid function generator via external pin in the neu- ral chip, we can achieve proper operations in the wide range of temperature. In the PSPICE simulations of the trajectory generation and the trajectory recognition, the neural chip shows the successful operations over the temperature range of 0 to 50°C. Other modifications are necessary for extreme temperature ranges. In Appendix C, the PSPICE simulation results of the trajectory generation with different tempera- ture are shown. With the temperature range of 0 to 50°C, the weights have converged as shown in Appendix C.1 and C2. However, the weights hit the rail voltage of the circuit with extreme temperature such as 125°C 0r -50°C as shown in Appendix C.3. 5.6.6 Future work In this implementation, we don’t include the update for time constant parameters. The time constant parameters give another degree of the freedom to the solution space of the neural network. Time constants of the equations are controlled by the capacitor value and the amount of current to the capacitor. The current can be controlled by the bias voltage. We need to develop the modified back-propagation learning rule to the time constant parameters. We need an efficient architecture for practical applications. Experiments with small-scale problem have proved as fruitful in many areas of science and engineering. However, not every phenomenon encountered in dealing with small models can be usefully scaled up [19]. We have seen many interesting demonstrations of neural net- works solving problems of very small scale but not doing so well when those problems were scaled up. Control domains are the most natural application for continuous-time recurrent neural networks. Signal processing and speech recognition and generation are also domains to which the recurrent neural network might be naturally applied. Certainly there is no reason to use a recurrent network when a feedforward layered neural 118 network suffices. Almeida [59] pointed out that one should not expect a major increase in the performance of a perceptron in every situation with feedback. In most cases, the best network structure will probably turn out to have feedback only in a small group of units. Ljung [60] also mentioned that, for system identification, the identifier must be chosen to have a small number of parameters, i.e., fewer parameters for a neural network. This is because the more parameters we use, the higher is the random influence on the model. We need to investigate the characteristics of new architectures such as partially connected recurrent neural network not the fully connected neural network. CHAPTER 6 2-Dimensional Scalable Array Configuration The recurrent neural network and its learning algorithm is implemented on a single analog CMOS chip. The floor plan of the recurrent neural network is organized in the 2-D array configuration. With the 2-D array configuration, the layout offers a sim- ple and scalable VLSI architecture for implementing a fully interconnected recurrent neural network. 6.1 Subcell design The multiplication between two quantities is the basic circuit of the recurrent neural network. Since the modified Gilbert multiplier generates the current output, we employ the current bus to collect the analog current outputs. The collected currents are converted to the voltage output through active resistors. To support the current bus and active resistors, we divide the modified Gilbert multiplier into two parts. The first component is the multiplier subcircuit which generates the current output, 1+ and I-. The second component consists of a current mirror and active resistors. The modified Gilbert multiplier cell and the current 119 114—Ci F— V3 V3—4 F— V4 it LI [. 1355C: v...—| PM {4 M. Vl—u + —» 1+ 1+ _, X CA Vour VZ—b .. ._, 1- I— ——> + - V3 V 4 The Current Minor and Active Resistors The Modified Gilbert Multiplier Subcell Figure 6.1. 2-D array configuration elements mirror and active resistors are shown in Figure 6.1. In this figure, we label the modified Gilbert multiplier subcell as a X component. The current mirror and active resistors are labeled as a CA component. The X component receives four voltage inputs and generates current outputs, 1+ and I-. The current output is proportional to the multiplication between (V1 — V2) and (V3 — V4). Since the neural network chip operates from ground to VDD of 5V, the virtual ground voltage becomes 2.5V. We label this virtual ground as reference voltage, Vref, 2.5V. If V2 and V4 terminals of the X component are connected to the reference voltage, the multiplication between V1 and V3 occurs. If V1 and V2 are connected together, then there is no multiplication on the X component since 121 (V1 — V2) produce the zero term. These current outputs are applied to the 1+ and I- of the CA component. The CA component converts the current inputs to the voltage output. There are two variations in the X component, X1 and X2. The X2 component has higher conductance than that of the X1 component. We use large (W/ L) ratio of the transistor M1 and M2 in the X2 component. The X2 component is used at the ll-dimensional multiplier for weight update equation. The CA component has three variations, CA1, CA2, and CA3. We have designed the voltage gain of the learning circuit is a little higher than that of the recurrent neural activation circuit. The C A1 component is used at the recurrent learning circuit and the CA2 component is used at the 2 equation generator. The CA3 component is used at the weight update equation and has the highest voltage gain overall. 6.2 Implementation of Floor Plan The floor plan of the VLSI recurrent neural network is organized in the 2-D array of weight interconnections. The block diagram of a fully connected 4-neuron recurrent neural network is shown in Figure 6.2. The 2—D array of weights and two boundary cells are shown in this figure. In Figure 6.2, we implement 4-neuron recurrent neural networks with maximum two input neurons and two output neurons. The number of inputs or outputs can be reduced by connecting the input terminal to the Vref or connecting the target terminal with the output terminal. The main function of the weight cell, w,-,- in Figure 6.2, is generating the activation term, 112.331,, of the neural network and the 2 term, 10.32;, in the learning equation. The upper X 1 component generates the 70,-, 2.- current output and the lower right X 1 component generates the my, current output. These current outputs are connected 122 VIII 2.5V 2 2' Q: I v I," 1 I +— J r— — H y 0., wnl‘ —‘ y W121i '-' P— . . i. 0 "I0 | 'u 2 “'12 PM A. '13 1 I“ '14 H k “Ml L "toyo '11,: '12,: F I 1 HI I I IT T I FL - 1 ' 1 2.5V F a r v I“ 0 —--4 —+ I—b-1 1 + . . r w, ' ‘72, ’ W22 if '12, i w“ _l—l “M y; wnyo 'an1 LL 1 1 1 1 4 I I I .L ' 1 L .L , I i I I 1 f "I z, #30 H :i—o I. :H) I. :L-o H CLO H W) '30 “'31 “'32 “’33 “'34 M Y: wnyo - I 1 4 1 1 I 1 r L L I l ' L I 1 I I I 1 I— "2 u 2‘ L u—o -—0 -—o 2‘ v—O ~ - We -—u—o L, -—4-«> I. u—r—Q ’YQ -—n-—o 1+ "MI '40 '41 "a '43 "u m Y4 "do F r 1 1 r 1 1 4 1 .L ' L ' l ' I 14 - ‘ 44 - 14 - ‘ toj - . J— I I1— i|__ x, t l g‘: mu. :2 m... M; ., w: z. ,1 1 2 {t 1136le Z. l Iqj 1') 24 1+ r CA 1+ I- ll I yl l . - . “085‘ M 1+ ‘ Inmatqaml j? 0—UV1 1- ; 111.1, 11 1:3 I‘ VI If H‘ y y, d 1- r “'3': w: 0m " ' '° fl 1 V] v 1 '3 an IE 93 [E Luv] x2 03 [E1 1+I- 1+ 1- 1+ 1- [+1- v2 1+ '- lfif- F" V W Ml ' I "I I I "no I 2' "by: "by: The array structure of the recurrent neural network 123 III I 2. II E; :IIIINI=: . I IIIIIHII-I—I-I—:—I LEI I?“ [111 11 fl° “- Figure 6.3. The MAGIC layout of the weight cell, an; _lil—j" II'fII Figure 6.4. The MAGIC layout of the weight cell, wgo 124 Figure 6.5. The MAGIC layout of the error.- & 2.- cell Figure 6.6. The MAGIC layout of the sigmoid.- & input,- cell 125 Figure 6.7. The MAGIC layout of the recurrent neural network 126 to the current buses and are collected in the error; & 2; cell and in the sigmoid; & input; cell. Also the update rule for weight itself is performed with z; and y,- terms by the lower left X2 component and the C A3 component. The capacitor is not included in this cell. The capacitor is located in the outside area of the array structure or weight can be connected to the external capacitor through the pins of the chip . The threshold weight cell, w;o, is shown in Figure 6.2. Since the threshold weight is not used at the 2 equation, the generation of w;,-z; circuit is removed. Other terminals such as V2 and V4 are connected to the reference voltage, 2.5V. The 2-D array of weights is connected with the boundary cells. Two boundary cells are designed to support the weight array. One is an error; & 2; cell and the other is a sigmoid; & input; cell. The error; & 2; cell collects the current of w;,-z; from the weight cell and the C A2 component generates the z; voltage with collected currents. Also, it receives target waveform and output waveform to generate the error term. If the neuron i is not used as an output neuron, the upper X 1 component in the cell can be blocked not to generate any output by tying the t; and y; terminals together. The error; 81. 2; cell also has the negative self-feedback component (the lower X1 cell) to ensure the convergent operation. Other terminals, such as V4 in the upper X 1 component and V1 in the lower X1 component, which is not displayed in the cell diagram are connected to the reference voltage, 2.5V. The sigmoid; & input; cell collects the current of w;,-y,~ from the weight cell to gen- erate the net; of the neuron 2'. These currents are converted to the voltage through the CA1 component and the output of the CA1 is applied to the sigmoid func- tion generator. The sigmoid function generator is implemented using the wide-range transconductance amplifier and its output becomes the state of the recurrent neural network. Also, this cell can receive the input waveform via the X1 component. If the neuron is used as an input neuron, the X1 component is activated to get the input 127 waveform. If the neuron is not used as an input neuron, the V1 terminal of the X1 component is connected to the reference voltage, 2.5V. The V2 and V4 terminals of the X 1 cell are connected to the reference voltage. This current-bus output arrangement combined with the boundary cells offers a simple and scalable VLSI architecture for implementing a fully interconnected recur- rent neural network with learning circuit. Figure 6.3 shows the MAGIC layout of the weight cell, w;,-. It has two X1 com- ponents, one X3 component, and one C A3 cell. It has been drawn using Scalable CMOS (SCMOS) technology and its size is 252) x 240). If 1A = 1p, the actual size becomes 252nm x 240pm. The actual size is dependent on the fabrication technol- ogy. In Figure 6.4, the layout of the w;o cell is shown. The size of the w;o cell is 252A x 224).. The other cells such as error; & z; and sigmoid; 85 input; has smaller size than the weight cell since they use fewer components. In Figure 6.5, the layout of the error; & 2; cell is displayed. Figure 6.6 shows the layout of the sigmoid; & input; cell. Each size is 220) x 214) and 247) x 178A, respectively. In Figure 6.7, the whole chip layout is displayed. 4 x 5 weight array is located in the center of the chip and the lower and right boundary cells are surround the weigh array. The lower boundary cells are error; & 2; cells and right boundary cells are sigmoid; & input; cells. The chip has 40 pins, 34 analog pads and 6 VDD and GND pads. It is designed to fabricate via MOSIS Tiny chip. In Table 6.1, the pin assignment of 40-pin tiny chip is shown. 128 Table 6.1. The pin assignment of the 4 neuron recurrent neural network chip Pin 1 2 3 4 5 Signal 2:1 x2 y1 y2 VDD Pin 6 7 8 9 10 Signal wu 1.012 1013 w“ GN D Pin 11 12 13 I4 15 Signal wlo 2.021 1.022 1023 VDD Pin 16 17 18 19 20 Signal 1024 wzo Vbias,mul Vref,mul Vth # Pin 21 22 23 24 25 Signal unused Vlogicl 11231 1032 GND Pin 26 27 28 29 30 Signal w33 w34 1.030 w.“ VDD Pin 31 32 33 34 35 Signal 1042 1043 to.“ 1.040 GND Pin 36 37 38 39 40 Signal t1 t2 unused Vref ,si g Vbias,sig CHAPTER 7 Conclusion A recurrent neural network with a modified recurrent back-propagation learning rule is implemented using analog CMOS technology. In order to implement the recurrent neural network and its learning algorithm, we employ a modified Gilbert multiplier, an active resistor, and a wide-range transconductance amplifier. The sigmoid function generator is designed using the transconductance amplifier. The limitation of the output voltage is resolved by using the wide—range transconduc- tance amplifier. The output of the transconductance amplifier is current. To convert the current output to the voltage output, we use an active resistor. To get appropriate ranges of resistance of the active resistor, we performed the PSPICE circuit simulation with different W/ L ratios. By adjusting the W/L ratio of the transistors, we can get a proper resistance value for converting the current output to the voltage output. The modified Gilbert multiplier uses voltage signals for its inputs where its output is current. We attached active resistors to get voltage-to-voltage operations. In the small-signal range, the characteristic curve is approximately linear, and its four- quadrant multiplication is verified through the PSPICE circuit simulation. Since the modified Gilbert multiplier cell generates the current output, the vector multiplier is designed on the current bus to collect the currents from the modified 129 130 Gilbert multiplier cells. The dimension of the vector multiplier can be increased by simply placing the modified Gilbert multiplier cell on the current bus. The adjusted active resistor converts this current to the voltage for voltage-to-voltage operations. We have reviewed four learning algorithms for temporal signal learning. The Pearlmutter’s algorithm converts a network evolving through time into a network whose activation is flowing through a number of layers (space). The requirements of the forward and backward integration make analog hardware implementation dif- ficult. For the classification of temporal trajectories, Sotelino et al. has developed the modified version of the Pearlmutter’s algorithm. This algorithm has the same problems as the Pearlmutter’s algorithm. The recurrent back-propagation algorithm by Pineda assumes that the activation equation of the neural network is convergent and the error is measured in the fixed point. The real-time recurrent learning by William and Zipser has on-line updating rule. However, its hardware requirements are so massive that we can not build a large network economically. We have modified the Pearlmutter’s algorithm and the Pineda’s algorithm for the modified recurrent back—propagation. Its forward instantaneous update scheme is suitable for an analog hardware implementation. We have built a 4-neuron recurrent neural network and a 6-neuron recurrent neural network. We have implemented the modified recurrent back-propagation learning rule using standard CMOS circuit and performed the PSPICE circuit simulations. In the 4-neuron recurrent neural network simulations, we have verified its func- tions by generating a circular trajectory. Simulation results show that the output signal is following the target signal and weights are convergent. The circular trajec- tory is generated by the recurrent neural network as a limit cycle. In the trajectory recognition experiment, we trained the neural network to recognize different trajec- tories. Its learning phase and test phase results show that the modified recurrent back-propagation learning rule is successful in learning and in testing the temporal 131 signals. In the 6-neuron recurrent neural network simulations, we have built two-neuron output neural network. We trained the neural network to learn different state tra- jectories and the PSPICE circuit simulations show the recurrent neural network has learned the temporal signals for classification. A two-dimensional scalable array configuration is designed for large-scale imple- mentation of fully connected recurrent neural network with learning. With the 2-D array configuration, the layout offers a simple and scalable VLSI architecture. We have built a 40-pin tiny chip using MOSIS’s SCMOS technology. APPENDICES APPENDIX A SPICE parameters 132 A.1 The SPICE parameters: MOSIS 2.0 pm OR- BIT ANALOG process N MOS P MOS LEVEL 2 2 TPG -1 LD 133.300000E— 09 102.100000E-09 VTO .8577 -.8721 KP 59.272000E—06 16.129000E-06 GAMMA .5361 .5972 PHI .7 .7 LAMBDA .03084 .03942 RSH 12.93 .1019 IS 10.000000E- 15 10.000000E- 15 PB .4 .9 PBSW .4 .9 CJ 134.000000E—06 334.000000E—06 CJ SW 61 1 .000000E— 12 397.000000E— 12 MJ .535 .585 MJ SW .2 .127 CGSO 174.800000E-12 133.890000E-12 CGDO 174.800000E- 12 133.890000E— 12 C GBO 345.820000E- 12 401 .740000E— l2 NSUB 6.617000E+15 8.212000E+15 NFS 93.830000E+09 607.200000E+09 TOX 39.500000E—09 39.500000E—09 XJ 200.000000E-09 200.000000E—09 UO 678 184.5 UCRIT 6.778000E+03 207.200000E+03 UEXP .0875 .4362 VMAX 48.300000E+03 999.900000E+03 DELTA 2.779 3.097 133 A.2 The SPICE parameters: MOSIS 0.5 pm HP process N MOS P MOS LEVEL 3 3 TPG -1 LD 47.290000E—09 35.070000E-09 VTO .6566 - .9213 KP 196.470000E-06 48.740000E-06 GAMMA .5976 .4673 PHI .7 .7 RSH 35.12 .11 IS 10.000000E— l5 10.000000E-15 PB .99 .99 PBSW .99 .99 CJ 562.000000E-06 935.000000E—06 CJ SW 50.000000E— 12 289.000000E— 12 MJ .559 .468 MJ SW .521 .505 CGSO 305. 150000E- 12 239.220000E- 12 CGDO 305. 150000E- 12 239.220000E— 12 CGBO 402.390000E- 12 375.790000E- 12 N SUB 139.200000E+ 15 85.120000E+15 N FS 590.900000E+09 650.000000E+09 TOX 9.600000E—09 9.600000E—09 XJ 200.000000E-09 200.000000E-09 UO 546.2 135.5 VMAX 200.800000E +03 254.200000E+03 DELTA .691 .2875 THETA .2684 .1807 ETA .03718 .0245 KAPPA .02898 7.958 APPENDIX B PSPICE Input Files 134 B.l Simple transconductance amplifier * Simple trans-conductance amplifier: Vbias = 1.3V (transl.cir) vdd 80 0 5.0 vs: 90 0 0.0 v1 1 0 v2 2 0 vbias 3 0 1.3 vdum 6 66 0.0 m1 5 1 4 90 n w=4u I=4u m2 6 2 4 90 n w=4u l=4u m3 4 3 90 90 n w=12u I=4u m4 80 5 5 80 p w=15u l=4u m5 80 5 6 80 p w=15u l=4u r1 80 66 1000K r2 66 0 1000K N63J SPICE LEVEL2 PARAMETERS .MODEL N NMOS LEVEL=2 PHI=0.700000 TOX=3.9500E-08 XJ=0.200000U TPG=1 + VTO=O.8577 DELTA=2.7790E+00 LD=1.3330E-07 KP=5.9272E-05 + UO=678.0 UEXP=8.7500E-02 UCRIT=6.7780E+03 RSH=1.2930E+01 + GAMMA=0.5361 NSUB=6.6170E+15 NFS=9.3830E+10 VMAX=4.8300E+04 + LAMBDA=3.0840£-02 CGDO=1.7480E-10 CGSO=1.7480E-10 + CGBO=3.4582E-10 CJ=1.34E-04 MJ=0.535 CJSW=6.11E-10 + MJSW=0.200 PB=0.40 .MODEL P PMOS LEVEL=2 PHI=0.700000 TOX=3.9500E-08 XJ=0.200000U TPG=-1 + VTO=-0.8721 DELTA=3.097OE+00 LD=1.0210E-07 KP=1.6129E-05 + UO=184.5 UEXP=4.3620E-01 UCRIT=2.0720E+05 RSH=1.0190E-01 + GAMMA=0.5972 NSUB=8.2120E+15 NFS=6.0720E+11 VMAX=9.9990E+05 + LAMBDA=3.942OE-02 CGDO=1.3389E-10 CGSO=1.3389E-10 + CGBO=4.0174E-10 CJ=3.34E-04 MJ=0.585 CJSW=3.97E-10 + MJSW=0.127 PB=O.90 .probe v(6) V(66) i(rl) i(r2) i(vdum) .dc v1 0.0 5.0 0.05 v2 0.5 4.5 0.5 .end 135 B.2 Wide range transconductance amplifier * Wide trans-conductance amplifier: (wtran1.cir) vdd 80 0 5.0 vs: 90 0 0.0 v1 1 0 v2 2 0 vbias 3 0 1.3 m1 5 1 4 90 n w=4u l=4u m2 7 2 4 90 n w=4u l=4u m6 80 5 5 80 p w=15u l=4u m7 80 5 6 80 p w=15u l=4u m5 80 7 7 80 p w=15u I=4u m4 80 7 8 80 p w=15u I=4u m8 8 8 90 90 n w=4u |=4u m9 6 8 90 90 n w=4u l=4u m3 4 3 90 90 n w=12u l=4u r1 80 6 1000K r2 6 0 1000K N63J SPICE LEVEL2 PARAMETERS .MODEL N NMOS LEVEL=2 PHI=0.700000 TOX=3.9500E-08 XJ=0.200000U TPG=1 + VTO=0.8577 DELTA=2.7790E+00 LD=1.3330E-07 KP=5.9272E-05 + UO=678.0 UEXP=8.7500E-02 UCRIT=6.7780E+03 RSH=1.2930E+01 + GAMMA=0.5361 NSUB=6.6170E+15 NFS=9.3830E+10 VMAX=4.8300E+04 + LAM BDA=3.0840E-02 CGDO=1.7480E-10 CGSO=1.7480E-10 + CGBO=3.4582E-10 CJ=1.34E-04 MJ=0.535 CJSW=6.11E-10 + MJSW=0.200 PB=0.40 .MODEL P PMOS LEVEL=2 PH|=0.700000 TOX=3.9500E-08 XJ=0.200000U TPG=-1 + VTO=-0.8721 DELTA=3.097OE+00 LD=1.0210E—07 KP=1.6129E-05 + UO=184.5 UEXP=4.3620E-01 UCRIT=2.0720E+05 RSH=1.0190E-01 + GAMMA=0.5972 NSUB=8.2120E+15 NFS=6.0720E+11 VMAX=9.9990E+05 + LAMBDA=3.9420E-02 CGDO=1.3389E-10 CGSO=1.3389E-10 + CGBO=4.0174E-10 CJ=3.34E-04 MJ=0.585 CJSW=3.97E-10 + MJSW=0.127 PB=0.90 .probe v(6) i(r1) i(r2) .dc v1 0.0 5.0 0.05 v2 0.5 4.5 0.5 .end 136 B.3 Sigmoid function generator 1 * Soma (SIGMOIDAL FUNCTION) circuit with adjusted W/L, Vbias=1.1V: ORBIT, N63J (sm1-20u.cir) vdd 80 0 5.0 vs 90 0 0.0 vbias 3 0 1.1 v2 2 0 2.5 v1 1 0 xmainl 80 90 1 2 3 8 main xactl 80 90 8 arl xmain2 80 90 1 2 3 18 main xact2 80 90 18 ar2 xmain3 80 90 1 2 3 28 main xact3 80 90 28 ar3 xmain4 80 90 1 2 3 38 main xact4 80 90 38 ar4 xmain5 80 90 1 2 3 48 main xact5 80 90 48 arS .subckt main 80 90 1 2 3 8 m1 5 1 4 90 n w=24u |=4u m2 6 2 4 90 n w=24u |=4u m3 4 3 90 90 n w=10u |=4u m4 80 6 7 80 p w=15u |=4u m5 80 6 6 80 p w=15u |=4u m6 80 5 5 80 p w=15u |=4u m7 80 5 8 80 p w=15u |=4u m8 7 7 90 90 n w=16u |=4u m9 8 7 90 90 n w=16u |=4u .ends .subckt ad 80 90 8 m10 80 8 8 80 p w=4u |=4u mll 8 90 90 8 p w=4u |=4u .ends .subckt ar2 80 90 8 m10 80 8 8 80 p w=4u |=12u mll 8 90 90 8 p w=4u I=12u .ends 137 .subckt ar3 80 90 8 m10 80 8 8 80 p w=4u I=20u mll 8 90 90 8 p w=4u |=20u .ends .subckt ar4 80 90 8 m10 80 8 8 80 p w=4u I=28u mll 8 90 90 8 p w=4u I=28u .ends .subckt ar5 80 90 8 m10 80 8 8 80 p w=4u l=36u mll 8 90 90 8 p w=4u l=360 .ends N63J SPICE LEVEL2 PARAMETERS .MODEL N NMOS LEVEL=2 PHI=0.700000 TOX=3.9500E-08 XJ=0.200000U TPG=1 + VTO=0.8577 DELTA=2.7790E+00 LD=1.3330E-07 KP=5.9272E-05 + UO=678.0 UEXP=8.7500E-02 UCR|T=6.7780E+03 RSH=1.293OE+01 + GAMMA=0.5361 NSUB=6.6170E+15 NFS=9.3830E+10 VMAX=4.8300E+04 -l- LAMBDA=3.0840E-02 CGDO=1.7480E-10 CGSO=1.7480E-10 + CGBO=3.4582E-10 CJ=1.34E-04 MJ=0.535 CJSW=6.11E-10 + MJSW=0.200 PB=0.4O .MODEL P PMOS LEVEL=2 PHI=0.700000 TOX==3.9500E-08 XJ=0.200000U TPG=-1 + VTO=-0.8721 DELTA=3.0970E+00 LD=1.0210E-07 KP=1.6129E-05 + U0=184.5 UEXP=4.3620E-01 UCR|T=2.0720E+05 RSH=1.0190E-01 + GAMMA=0.5972 NSUB=8.2120E+15 NFS=6.0720E+11 VMAX=9.9990E+05 + LAMBDA=3.9420E-02 CGDO=1.3389E-10 CGSO=1.3389E-10 + CGBO=4.0174E-10 CJ=3.34E-04 MJ=0.585 CJSW=3.97E-10 + MJSW=0.127 PB=0.90 .probe V(8) v(18) v(28) v(38) v(48) .dc v1 1.5 3.5 0.01 .end 138 BA Sigmoid function generator 2 * Soma (SIGMOIDAL FUNCTION) circuit with different bias with (4u/19u,4u/21u): ORBIT, N63J (sm2-20u.cir) vdd 80 0 5.0 vs 90 0 0.0 vbias 3 0 v2 2 O 2.5 v1 1 0 xmain2 80 90 1 2 3 18 main xact2 80 90 18 ar2 .subckt main 80 90 1 2 3 8 m1 5 1 4 90 n w=24u |=4u m2 6 2 4 90 n w=24u |=4u m3 4 3 90 90 n w=10u |=4u m4 80 6 7 80 p w=15u |=4u m5 80 6 6 80 p w=15u |=4u m6 80 5 5 80 p w=15u |=4u m7 80 5 8 80 p w=15u |=4u m8 7 7 90 90 n w=16u |=4u m9 8 7 90 90 n w=16u |=4u .ends .subckt ar2 80 90 8 m10 80 8 8 80 p w=4u |=19u mll 8 90 90 8 p w=4u l=21u .ends N63J SPICE LEVEL2 PARAMETERS .MODEL N NMOS LEVEL=2 PH|=0.700000 TOX=3.9500E-08 XJ=0.200000U TPG=1 + VTO=0.8577 DELTA=2.7790E+00 LD=1.3330E-07 KP=5.9272E-05 + UO=678.0 UEXP=8.7500E-02 UCR|T=6.7780E+03 RSH=1.2930E+01 + GAMMA=0.5361 NSUB=6.6170E+15 NFS=9.3830E+10 VMAX=4.8300E+04 + LAMBDA=3.0840E-02 CGDO=1.7480E-10 CGSO=1.7480E-10 + CGBO=3.4582E-10 CJ=1.34E-04 MJ=0.535 CJSW=6.11E-10 + MJSW=0.200 PB=0.40 .MODEL P PMOS LEVEL=2 PHI=0.700000 TOX=3.9500E-08 XJ=0.200000U TPG=-1 + VTO=-0.8721 DELTA=3.0970E+00 LD=1.0210E-07 KP=1.6129E-05 + U0=184.5 UEXP=4.3620E-01 UCRIT=2.0720E+05 RSH=1.0190E-01 + GAMMA=0.5972 NSUB=8.2120E+15 NFS=6.0720E+11 VMAX=9.9990E+05 + LAMBDA=3.9420E-O2 CGDO=1.3389E-10 CGSO=1.3389E-10 + CGBO=4.0174E-10 CJ=3.34E-04 MJ=0.585 CJSW=3.97E-10 + MJSW=0.127 PB=0.90 139 .probe v(18) .dc v1 1.5 3.5 0.02 vbias 0.9 1.4 0.1 .end 140 B.5 Modified Gilbert Multiplier * l-D Vector Multiplier: ORBIT, N63J (gmul1.cir) vdd 80 0 5.0 vs: 90 0 0.0 v110 V220 vref 10 0 2.5 vbiasl 11 0 1.5 xgl so 90 1 102 10 11 535:1 .subckt gill 80 90 1 2 3 4 5 7 xsgl 80901234567sgil xre 80 90 6 7 linrel .ends .subckt sgil 8O 90 1 2 3 4 5 13 14 m1 7 1 6 90 n w=4u |=4u m2 8 2 6 90 n w=4u |=4u m3 9 3 12 80 p w=15u |=4u m4 9 4 11 80 p w=15u |=4u m5 10 3 11 80 p w=15u |=4u m6 10 4 12 80 p w=15u |=4u m7 80 7 9 80 p w=15u |=4u m8 80 7 7 80 p w=15u |=4u m9 80 8 8 80 p w=15u |=4u m10 80 8 10 80 p w=15u |=4u m11 13 11 90 90 n w=4u |=4u m12 11 11 90 90 n w=4u |=4u m13 12 12 90 90 n w=4u |=4u m14 14 12 90 90 n w=4u |=4u m15 6 5 90 90 n w=12u |=4u .ends .subckt linrel 80 9O 6 77 vdum 77 7 0.0 m31 80 7 7 80 p w=4u |=12u m32 7 90 90 7 p w=4u |=12u m21 8O 6 6 80 p w=11u |=14u m22 80 6 77 80 p w=11u |=14u .ends N63J SPICE LEVEL2 PARAMETERS .MODEL N NMOS LEVEL=2 PHI=0.700000 TOX=3.9500E-08 XJ=0.200000U TPG=1 141 + VTO=0.8577 DELTA=2.7790E+00 LD=1.3330E-07 KP=5.9272E-05 + UO=678.0 UEXP=8.7500E-02 UCRIT=6.7780E+03 RSH=1.2930E+01 + GAMMA=0.5361 NSUB=6.6170E+15 NFS=9.3830E+10 VMAX=4.8300E+04 + LAM BDA=3.0840E-02 CGDO=1.7480E-10 CGSO=1.7480E-10 + CGBO=3.4582E-10 CJ=1.34E-04 MJ=0.535 CJSW=6.11E-10 + MJSW=0.200 PB=0.40 .MODEL P PMOS LEVEL=2 PHI=O.700000 TOX=3.9500E-08 XJ=0.200000U TPGz-l + VTO=-0.8721 DELTA=3.097OE+00 LD=1.0210E-O7 KP=1.6129E-05 + U0=184.5 UEXP=4.3620E-01 UCR|T=2.0720E+05 RSH=1.0190E-01 + GAMMA=0.5972 NSUB=8.2120E+15 NFS=6.072OE+11 VMAX=9.9990E+05 + LAMBDA=3.9420E-02 CGDO=1.3389E-10 CGSO=1.3389E-10 + CGBO=4.0174E-10 CJ=3.34E-04 MJ=0.585 CJSW=3.97E-10 + MJSW=0.127 PB=0.90 .probe V(5) i(xgl.xre.vdum) id(xg1.xre.m31) id(xg1.xre.m32) .dc v1 1 4 0.02 v2 1.5 3.5 0.25 .end 142 B.6 3-D Vector Multiplier "‘ 3-D Vector Multiplier: ORBIT, N63J (vm3-20u.cir) vdd 80 0 5.0 vs 90 0 0.0 V110 V220 vref 10 0 2.5 vbiasl 11 0 1.1 xgl80901102101111021011110210115gil3 .subckt gil3 80 9O 1 2 3 4 5 11 12 13 14 15 21 22 23 24 25 7 xsg180901234567sgi| xsg2 80 90 11 12 13 14 15 6 7 sgil xsg3 80 90 21 22 23 24 25 6 7 sgil xre 80 90 6 7 linre3 .ends .subckt sgil 80 90 1 2 3 4 5 13 14 m1 7 1 6 90 n w=4u |=4u m2 8 2 6 90 n w=4u |=4u m3 9 3 12 80 p w=15u |=4u m4 9 4 11 80 p w=15u |=4u m5 10 3 11 80 p w=15u |=4u m6 10 4 12 80 p w=15u |=4u m7 80 7 9 80 p w=15u |=4u m8 80 7 7 80 p w=15u |=4u m9 80 8 8 80 p w=15u |=4u m10 80 8 10 80 p w=15u |=4u mll 13 11 90 90 n w=4u |=4u m12 11 11 90 90 n w=4u |=4u m13 12 12 90 90 n w=4u |=4u m14 14 12 90 90 n w=4u |=4u m15 6 5 90 90 n w=12u |=4u .ends .subckt |inre3 8O 9O 6 77 vdum 7 77 0.0 m31 80 7 7 80 p w=4u |=12u m32 7 90 90 7 p w=4u |=12u m21 80 6 6 80 p w=11u |=14u m22 80 6 77 80 p w=11u |=14u .ends 143 N63J SPICE LEVEL2 PARAMETERS .MODEL N NMOS LEVEL=2 PHI=0.700000 TOX=3.9500E-08 XJ=0.200000U TPG=1 + VTO=0.8577 DELTA=2.779OE+00 LD=1.3330E-07 KP=5.9272E-05 + U0=678.0 UEXP=8.7500E-02 UCRIT=6.7780E+03 RSH=1.2930E+01 + GAMMA=O.5361 NSUB=6.6170E+15 NFS=9.3830E+10 VMAX=4.8300E+04 + LAMBDA=3.0840E-02 CGDO=1.7480E-10 CGSO=1.7480E-10 + CGBO=3.4582E—10 CJ=1.34E-04 MJ=0.535 CJSW=6.11E-10 + MJSW=0.200 PB=0.40 .MODEL P PMOS LEVEL=2 PH|=O.700000 TOX=3.9500E-08 XJ=0.200000U TPG=-1 + VTO=-0.872l DELTA=3.0970E+00 LD=1.0210E-07 KP=1.6129E-05 + U0=184.5 UEXP=4.3620E-01 UCR|T=2.0720E+05 RSH=1.0190E-01 + GAMMA=0.5972 NSUB=8.2120E+15 NFS=6.0720E+11 VMAX=9.999OE+05 + LAM BDA=3.9420E-02 CGDO=1.3389E-10 CGSO=1.3389E-10 + CGBO=4.0174E-10 CJ=3.34E-04 MJ=0.585 CJSW=3.97E-10 + MJSW=0.127 PB=0.90 .probe .dc v1 1.5 3.5 0.02 v2 1.5 3.5 0.25 .end APPENDIX C PSPICE simulation results with different temperature 144 C.1 Trajectory generation with 50°C °m:4mmd):m Wham “31:10 Twat) m 1 . Figure C.1. Input, output and target signals 'm:smp1-1):ao_u Warm 03:31:13 1mm) 1.0 .cn-oooo...--.-od- D It. m cm may mm mm om om um vch-gsm Wm Figure 0.2. Em. and the weights 145 C.2 Trajectory generation with 0°C ‘m:¢m¢tot):u Tm 0.9 o-ouo-nood Figure C.3. Input, output and target signals 'm:4mp-t-t):efl_0& mum 1m 1m” 1111' I AAAAAA v' vvv I." .I . I ' ‘mmmwl'((1'{((|//'.I_IAJ 1".“ ~ D on. m °VI3I ’VDII ‘VI‘II 'W‘I 0m ’VWI 'VP‘I vvgtm 'VINI Figure C.4. Em. and the weights 146 C.3 Trajectory generation with 125°C and -50°C 'wtaztmfl-t-nzefl” ”The M1: MIDI 01:37:“ I“ 18.0 W T """""""""""""""""""""""""""""" ""1 aw i O to. I”. am ‘VINI ‘VI‘II 'W‘I ovum ’WuI 'VIII '11:.an 'VI‘I Figure C .5. Weights hit the rail voltage °mmmmm=m Twas! n: i- H" '° W "(11) 0V0!) MINI WWI 0W2!) om ”m 'Vfl) HIM) 'VM T 7 '_"v 7-- ' __v'v ' __vvv __v v W; --------------------------- r -------------------------- 'I' ------------------------- w --------------------------- 4. 1601- m. h m m- nm we» win) was: om) om» we» vm‘m WM Figure C.6. Weghts hit the rail voltage BIBLIOGRAPHY BIBLIOGRAPHY [1] John Hertz, Anders Krogh, and Richard G. Palmer, Introduction to the Theory of Neural Computation, Addison-Wesley Publishing Co., 1991. [2] Judith E. Dayhoff, Neural Network Architecture, Van Nostrand Reinhold, New York, 1990. [3] Patrick K. Simpson, Artificial Neural Systems: Foundations, Paradigms, Applications, and Implementations, Pergamon Press, New York, 1990. [4] R. P. Lippmann, “An Introduction to Computing with Neural Nets”, IEEE Acoustics, Speech and Signal Processing Magazine, 4(2), April 1987, pp. 4-22. [5] S. Eberhardt, R. Tawel, T. Brown, T. Daud, and A. Thakoor, “Analog VLSI Neu- ral Networks: Implementation Issues and Examples in Optimization and Supervised Learning”, IEEE Transaction on Industrial Electronics, Vol. 39, No. 6, December 1992, pp. 552-564. [6] Eric A. Vittoz, “Analog VLSI Signal Processing: Why, Where and How?”, Analog Integrated Circuits and Signal Processing, July 1994, pp. 27-44. [7] M. Maher, S. DeWeerth, M. Mahowald, and C. Mead, “Implementing Neural Architec- tures using Analog VLSI Circuits”, IEEE Transaction on Circuits and Systems, Vol. 36, No. 5, May 1989, pp. 643-652. [8] Caver Mead, “Adaptive Retina”, in Analog VLSI Implementation of Neural Networks, C. Mead and M. Ismail, Eds. Boston: Kluwer Academic Publishers, 1989. [9] R. F. Lyon and C. Mead, “An Analog Electronic Cochlea”, IEEE Trans. Acoust. Speech Signal Proc., Vol. 36, No. 7, July 1988, pp.1119-1134. [10] M. A. Holler, S. Tam, H. Castro, and R. Benson, “An Electrically Trainable Artificial Neural Network (ETANN) with 10240 Floating Gates Synapses”, International Joint Conference on Neural Networks, 1989, Vol. 2, pp.191-196, Washington D.C. [11] E. Sackinger, B. E. Boser, J. Bromley, Y. Le Gun, and L. D. Jackel, “Application of the ANNA Neural Network Chip to High-speed Character Recognition”, IEEE Trans- actions on Neural Networks, Vol. 3, May 1992, pp.498-505. 147 148 [12] J. Choi, S. H. Bang, and B. J. Sheu, “A Programmable Analog VLSI Neural Network Processor for Communication Receiver”, IEEE Transactions on Neural Networks, Vol. 4, May 1993, pp.484-495. [13] Hwa—Joon Oh and Fathi M. Salam, “Analog CMOS Implementation of Neural Network for Adaptive Signal Processing”, Proc. of IEEE International Symposium on Circuit and Systems 1994, London, England, May 30 - June 2, 1994, pp.503-506. [14] Gert Cauwenberghs, “An Analog VLSI Recurrent Neural Network Learning a Continuous-Time Trajectory”, IEEE Transactions on Neural Networks, Vol. 7, No. 2, March 1996, pp.346-361. [15] Bart Kosko, Editor, Neural Networks for Signal Processing, Prentice-Hall, Inc., New Jersey, 1992. [16] Barak A. Pearlmutter, “Gradient Calculations for Dynamic Recurrent Neural Network: A Survey”, IEEE Trans. on Neural Networks, Vol. 6, No. 5, September 1995, pp.1212- 1228. [17] Charles F. Stevens, “The neuron”, Scientific American, September 1979. [18] Warren S. McCulloch and Walter Pitts, “A Logical Calculus of Ideas Immanent in Nervous Activity”, Bulletin of Mathematical Biophysics 5, 1943, pp.115-133. [19] Marvin L. Minsky and Seymour A. Papert, Perceptrons, Expanded edition, The MIT Press, 1990. [20] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, Parallel Distributed Processing: Explorations in the Microstructure of Cognition, MIT Press, 1986, Vol. I. [21] J. J. Hopfield, “Neural Networks and Physical Systems with Emergent Collective Com- putational Abilities”, Proc. Natl. Acad. Sci. U.S.A. Vol. 79, pp. 2554-2556, 1982. [22] J. J. Hopfield, “Neurons with Graded Responses Have Collective Computational Prop- erties Like Those of Two-State Neurons”, Proceedings of the National Academy of Sciences, USA 81, pp.3088-3092. [23] J. J. Hopfield and D. W. Tank, “Computing with Neural Circuits: A Model”, Science 233, pp.625-633. [24] Don R. Hush and Bill Horne, “Progress in Supervised Neural Networks: What’s new since Lippman?”, IEEE Signal Processing Magazine 10, 1993, pp. 8-39. [25] Fathi M. Salam, “Learning Algorithms for Artificial Neural Nets for Analog Circuit Implementation”, Computing Science and Statistics, Proc. of the 22nd Symposium on the Interface, May 16-19, 1990, pp.169-177. 149 [26] Fathi M. Salam, “A Modified Learning Rule for Feedforward Artificial Neural Nets for Analog Implementation”, Memorandum No. MSU / EE/ S 90/02, Department of Electrical Engineering, Michigan State University, Jan. 26 1990. [27] Myung-Ryul Choi, Implementation of Feedforward Artificial Neural Networks with Learning using Standard CMOS Technology, Ph. D. Dissertation, Dept. of Electrical Engineering, Michigan State University, 1991. [28] Hwa—Joon Oh and Fathi M. Salam, “A Modular Analog Chip for Feedforward Networks with On-Chip Learning”, Proc. of IEEE 36th Midwest Symposium on Circuit and System, Detroit, Aug.l993. pp.766-769 [29] C. T. Sah, “Characteristics of the Metal-Oxide-Semiconductor Transistor”, IEEE Trans. on Electron Devices, vol. ED-11, July 1964, pp.324-345. [30] H. Shichman and D. Hodges, “Modeling and Simulation of Insulated-Gate Field Effect Transistor Switching Circuits”, IEEE Journal of Solid State Circuits, Vol. SC-3, No. 3, Sep. 1968, pp.285-289. [31] A. Paolo and M. Giuseppe, Semiconductor Device Modeling with SPICE, McGraw-Hill, Inc. 1988. [32] Paul W. Tuinenga, SPICE: A guide to circuit simulation and analysis using PSpice, Prentice-Hall, New Jersey, 1988. [33] J. E. Meyer, “MOS Models and Circuit Simulation”, RCA Review, Vol. 32, 1971. [34] L. M. Dang, “A Simple Current Model for Short Channel IGFET and Its Application to Circuit Simulation”, IEEE Journal of Solid-State Circuits, Vol. 14(2), 1979. [35] R. Kielkowski, Inside SPICE: Overcoming the Obstacles of Circuit Simulation, McGraw-Hill, 1994. [36] Phillip E. Allen and Douglas R. Holberg, CMOS Analog Circuit Design, Holt, Rinehart and Winston (HRW), 1987. [37] Randall L. Geiger, Phillip E. Allen, and Noel R. Strader, VLSI Design Techniques for Analog and Digital Circuits, McGraw-Hill Publishing Company, 1990. [38] Caver Mead, Analog VLSI and Neural Systems, Addison-Wesley Publishing Company, 1989. [39] M. Steyaert and W. Sansen, “High Performance Operational Amplifiers and Compara- tors”, in Analogue-Digital ASICs: Circuit techniques, design tools and applications, edited by R. Soin, F. Maloberti and J. Franca, Chapter 3, pp.41-64., Peter Peregrinus Ltd. 1991. 150 [40] Barrie Gilbert, “A Precise Four-quadrant Multiplier with Subnanosecond Response”, IEEE Journal of Solid-State Circuits, Vol. SC-3z365, 1968. [41] S. Qin and R. Geiger, “A :I:5-V CMOS Analog Multiplier”, IEEE Journal of Solid-State Circuits, Vol. 80-22, No.6, December 1987, pp.1143-1146. [42] J. N. Babanezhad and G. C. Temes, “A 20-V Four-quadrant CMOS Analog Multiplier”, IEEE Journal of Solid-State Circuits, Vol. SC-20, No. 6, Dec. 1985, pp. 1158-1168 [43] K. Bult and H. Wallinga, “A CMOS Four-quadrant Analog multiplier”, IEEE Journal of Solid-State Circuits, Vol. SC-21, No. 3, June 1986, pp. 430-435. [44] J. Pena-Finol and J. A. Connelly, “A MOS Four-quadrant CMOS analog Multiplier using the Quarter-square Technique”, IEEE Journal of Solid-State Circuits, Vol. SC-22, No. 6, Dec. 1987, pp.1064-1073. [45] D. C. 800 and R. G. Meyer, “A Four-quadrant NMOS Analog Multiplier”, IEEE Journal of Solid-State Circuits, Vol. 8017, No. 6, Dec. 1982, pp.1174-1178. [46] C. W. Kim and S. B. Park, “New Four-quadrant CMOS Analog Multiplier”, Electron Letters, Vol. 23, No. 24, Nov. 1987, pp. 1268-1270. [47] Hwa—Joon Oh and Fathi M. Salam, “4x4x2 Neural network Design Using Modular Neural Chips with On-chip Learning”, Proc. of IEEE International Conference on Neural Networks (ICNN), Orlando, Jun.28-Jul.2, 1994, pp.2070-2073. [48] Simon Haykin, Neural Networks: A Comprehensive Foundation, Macmillan College Publishing Company, Inc., 1994. [49] J acek M. Zurada, Introduction to Artificial Neural Systems, West Publishing Company, 1992. [50] Barak A. Pearlmutter, “Learning State Space Trajectories in Recurrent Neural Net- works”, Proc. of International Joint Conference on Neural Networks (IJ CNN) 1989, June 18-22, 1989, Vol. 11, pp.365-372. [51] D. Kirk, Optimal control theory: An introduction, Englewood Cliffs, NJ: Prentice-Hall, 1970. [52] Frank L. Lewis, Optimal Control, John Wiley & Sons, NY, 1986. [53] Luis G. Sotelino, Marco Saerens, and Hugues Bersini, “Classification of Temporal Trajectories by Continuous-Time Recurrent Nets”, Neural Networks, Vol. 7, No. 5, pp.767-776, 1994. [54] Fernando J. Pineda, “Generalization of Back-Propagation to Recurrent Neural Net- works”, Physical Review Letters, Vol. 59, Number 19, 9 Nov., 1987, pp.2229-2232. 151 [55] R. J. Williams and D. Zipser, “A Learning Algorithm for Continually Running Fully Recurrent Neural Networks”, Neural Computation 1, 1989, pp.270-280. [56] L. B. Almeida, “A Learning Rule for Asynchronous Perceptrons with Feedback in a Combinatorial Environment”, In IEEE First International Conference on Neural Networks (San Diego 1987), Eds. M. Caudill and C. Butler, Vol. II, pp.609-618. [57] Kiyotoshi Matsuoka, “Stability Conditions for Nonlinear Continuous Neural Networks with Asymmetric Connection Weights”, Neural Networks, Vol. 5, pp. 495-500, 1992. [58] L. Jin, P. N. Nikiforuk, and M. M. Gupta, “Absolute Stability Conditions for Discrete- time Recurrent Neural Networks”, IEEE Trans. on Neural Networks, Vol. 5, No. 6, November 1994, pp.954-964. [59] L. B. Almeida, “Backpropagation in Non-Feedforward Networks”, in I. Aleksander (Ed.), Neural Computing Architectures, Cambridge, MA: MIT Press, 1989, pp.75-91. [60] L. Ljung, “Issue in system identification”, IEEE Control System Magazine, Vol. 11, Jan. 1991, pp.25-29. [61] Hwa-Joon Oh and Fathi Salam, “Analog CMOS Feedforward Artificial Neural Network with On-Chip Learning: Test Results”, Proc. of 1993 International Symposium on Nonlinear Theory and Its Applications, Hawaii, Dec. 5 - 10, 1993. [62] Fathi Salam and Hwa-Joon Oh, “Real-Time Tracking Control Using Modular Neural Chips with On-Chip Learning”, Proc. of IEEE International Conference on Neural Networks (ICNN), 1996, June 2-6, Washington D.C. "IIIIIIIIIIIIIIIIIIIII