Q I: v J as... 9‘ .35. (.‘J A; 2.. :.: .IL ..I. a (41" ‘ . 171:3... . 3.7.3:... ’19. L 51 L... 2.1.4. .. 1%.»; .. a . «1.1.. s erzik ‘1):- i5 5.», , fig“? iii: .P HNHIIHIIHIHHIJIIIlNllllflHfl 301555 6628 This is to certify that the dissertation entitled Bifurcations in Brain Dynamics presented by Eugene Izhikevich has been accepted towards fulfillment of the requirements for Ph.D. degree in Mathematics Asa/W (Frank C. H oppensteadt ) Major professor Date April 18, 1996 MSU is an Afllnnative Action/Equal Opportunity Institution E1 EH5: LIBRARY Michigan State University PLACE IN RETURN BOX to remove thie checkout from your record. TO AVOID FINES return on or betore dete due. DATE DUE DATE DUE DATE DUE ‘13 h [1w 2 3 KW 3 j [—7 MSU le An Nflrmetive Action/Ewe! Opportunity lnetitmion W ‘ i. BIFURCATIONS IN BRAIN DYNAMICS By Eugene M. Izhikevich A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Department of Mathematics 1996 ABSTRACT BIFURCATIONS IN BRAIN DYNAMICS BY Eugene M. Izhikevich Mathematical models of the brain are studied with the assumption that the con- nections between neurons are weak. This leads to weakly connected systems, which are called Weakly Connected Neural Networks (WCNNs). Local dynamics of the WCNNs is studied using bifurcation theory. First it is proved that the WCNNS could have interesting local dynamics with possible applications to neurocomputers only near bifurcations. Then it is shown that near the bifurcations the WCNNs can be significantly simplified and reduced to canonical models. Derivation and analysis of the canonical models for multiple (quasi-static) saddle- node, pitchfork and Andronov-Hopf bifurcations and multiple cusp singularities is presented. Mathematical analysis of the canonical models suggests a new neural network paradigm —— non-hyperbolic neural networks. It also sheds some light on possible synaptic organizations of the brain. In particular, it reveals the relationship between synaptic architectures (anatomy) and dynamical properties (function) of networks of neural oscillators. A part of this dissertation (Chapters 2 and 7) received the SIAM Student Paper Prize in applied mathematics for 1995. DEDICATION To April 29. The day of birth of two people who have had a huge impact on my life for the last three years. iii ACKNOWLEDGMENTS This work could not be accomplished without guidance and wisdom of my Teacher Frank C. Hoppensteadt. It is a great fortune and pleasure to work with him. I am very grateful for this opportunity and for all kinds of investments he made in me and my research. It seems to me that for the last three years we have developed a relationship which goes beyond an academic level and I hope he feels the same way. I am proud to be called a student of Frank C. Hoppensteadt. I would like to mention my former scientific supervisor, professor of Moscow State University, Georgii G. Malinetskii, who introduced me to the Dynamical System Theory. I am especially grateful to Sheldon E. Newhouse for his contribution to my un- derstanding of Dynamical System Theory and for his patience to my ”active” style of learning. I am thankful to the Society for Industrial and Applied Mathematics (SIAM) for the high appreciation of this work. Finally, I would like to thank my wife Tatyana Izhikevich (Kazakova) for her understanding and support. iv Contents List of figures .................................. 1 Introduction 1.1 Overview .................................. 1.2 Models in Mathematical Biology ..................... 1.3 Neurobiological Background ....................... 1.4 Neural Network Types .......................... 1.4.1 Olfactory Bulb .......................... 1.4.2 Networks of Excitable Elements ................. 1.5 Generic Equations eeeeeeeeeeeeeeeeeeeeeeeeeeee I Derivation of Canonical Models 2 Weakly Connected Neural Networks 2.1 Hyperbolic Equilibrium .......................... 2.2 Non-Hyperbolic Equilibrium ....................... 2.3 The Center Manifold Reduction ..................... 2.4 Canonical Models ............................. 11 13 14 17 20 21 21 24 25 3O 2.5 2.6 2.4.1 Multiple Saddle-Node Bifurcation ................ 2.4.2 Multiple Cusp Singularity .................... 2.4.3 Multiple Pitchfork Bifurcation .................. Multiple Andronov-Hopf Bifurcation .................. 2.5.1 Equality of Frequencies and Attention ............. Discussion ................................. 2.6.1 Adaptation Condition and Psychology ............. Singularly Perturbed WCNNs 3.1 3.2 3.3 3.4 3.5 3.6 3.7 Basic Definitions ............................. Motivational Examples .......................... Reduction to Regular Perturbation Problem .............. Center Manifold Reduction ........................ Canonical Models ............................. 3.5.1 Multiple Quasi-Static Saddle-Node Bifurcations ........ 3.5.2 Multiple Quasi-Static Pitchfork Bifurcations .......... 3.5.3 Discussion of the case ,u = (9(5) ................. Multiple Quasi-Static Andronov-Hopf Bifurcations ........... Conclusion ................................. 3.7.1 Synaptic Organizations of the Brain ............... Weakly Connected Maps 4.1 Hyperbolic Fixed Points ..... ‘ .................... vi 31 33 34 39 44 44 46 50 60 61 63 63 65 67 4.2 Non—hyperbolic Fixed Points ....................... 68 4.2.1 Multiple Saddle-Node Bifurcations ............... 69 4.2.2 Multiple Flip Bifurcations .................... 70 4.3 Connection With ODE .......................... 72 II Analysis of Canonical Models 74 5 Multiple Saddle-Node Bifurcation 75 5.1 Saddle-Node on a Limit Cycle ...................... 75 5.2 The VCON ................................ 78 5.3 Preliminary Analysis ........................... 79 5.4 Case c at 0 ................................. 80 5.5 Adaptation Condition ls Satisfied .................... 82 6 Multiple Andronov-Hopf Bifurcation 85 6.1 Complex Synaptic Coefficients cij .................... 85 6.2 Oscillator Death and Self—ignition .................... 88 6.3 Synchronization and Convergence .................... 91 7 Multiple Cusp Singularity 94 7.1 Extreme Values of Parameters ...................... 95 7.1.1 Global behavior .......................... 95 7.1.2 Strong Input From Receptors .................. 97 7.1.3 Extreme Psychological Condition ................ 98 vii 7.2 Canonical Models as a GAS-Type NN .................. 99 7.3 Symmetric Synaptic Connections .................... 101 7.4 Hebbian Learning Rule For Synaptic Matrix C ................................. 103 7.5 Bifurcations for r = 0 ........................... 105 7.5.1 Stability of the Origin ...................... 106 7.5.2 Stability of the Other Equilibria ................. 108 7.6 Bifurcations for r 75 0 (two memorized images) ............. 112 7.6.1 The Reduction Lemma ...................... 112 7.6.2 Recognition: Only One Image Is Presented ........... 115 7.6.3 Recognition: Two Images Are Presented ............ 117 7.7 Bistability of Perception ......................... 118 7.8 Quasi-Static Variation of Parameter b .................. 121 8 Quasi-Static Bifurcations 124 8.1 Stability of the Equilibrium ....................... 124 8.2 Dale’s Principle and Synchronization .................. 127 8.3 Further Analysis of the Andronov-Hopf Bifurcation .......... 130 8.4 Proofs of Theorems 54 and 57 ...................... 132 8.4.1 Proof of Theorem 54 ....................... 134 8.4.2 Proof of Theorem 57 ....................... 135 9 Non-Hyperbolic Neural Networks 138 viii 9.1 Problem 1 ................................. 9.2 Problems 2 and 3 ............................. 10 Synaptic Organizations of the Brain 10.1 Neural Oscillators ............................. 10.1.1 Multiple Andronov-Hopf Bifurcation .............. 10.1.2 Type A and B Neural Oscillators ................ 10.2 Dale’s Principle and Connectivity .................... 10.3 Classification of Synaptic Organizations ................. 10.4 Learning Dynamics ............................ 10.5 Memorization of Phase Information ................... 10.6 Synaptic Organizations .......................... 11 Discussion 11.1 Canonical Models and Normal Forms .................. 11.2 Synaptic Connections ........................... 11.3 Mathematical Conditions and Biology .................. 11.4 Co—dimensions of the Models ....................... 11.5 List of Canonical Models ......................... Bibliography 142 144 147 149 151 153 156 158 161 166 173 179 List of Figures 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 Summary of bifurcations and canonical models for regularly perturbed weakly connected systems ........................ Summary of multiple quasi-static bifurcations and canonical models for singularly perturbed weakly connected systems ........... Relationship between models in mathematical biology ......... A “typical” neuron. It receives signals from other neurons via synapses and sends the output signal to the other neurons through its axon. . . a. Multiple Attractor (MA) neural network. b. Globally Asymptoti- cally Stable (GAS) neural network. ................... Schematic representation of the Olfactory Bulb (OB) anatomy. Activ- ity of the mitral cells is denoted by X,- and granule cells by Y,-. . . . . An excitable system which mimics a neuron dynamics. The neuron can be near threshold (a), hyperpolarized (b) or depolarized (c). Basic principles of the non-hyperbolic neural network functioning. The input is given as a parameter which perturbs the non—hyperbolic equi— librium. Local bifurcations of the equilibrium affect global dynamics of the network. .............................. 5 8 10 12 14 15 2.1 3.1 3.2 3.3 4.1 4.2 5.1 5.2 5.3 5.4 5.5 7.1 Synaptic connections between neurons having different natural fre- quencies are functionally insignificant. Therefore, the network can be divided into subnetworks (pools) of oscillators having equal or E-close natural frequencies ............................ Possible intersections of nullclines of the relaxation neuron (3.1). . . . An excitable system. There are initial conditions for which the system (3.1) generates an action potential, or spike (dotted line) ........ Intersections of nullclines which do not correspond to an excitable sys- tem. a. A relaxation oscillator with non-zero amplitude. b. A relax- ation oscillator with zero amplitude .................... Thalamo-cortical interactions ...................... Bifurcations of a mapping. a. Saddle-node bifurcation for cipg < 0. b. Saddle-node bifurcation for cipi > 0. c. Flip bifurcation. ....... Phase portraits for various 6 ....................... Saddle—node bifurcation on a limit cycle ................. An intersection of nullclines in a relaxation system which exhibits saddle-node bifurcation on a limit cycle ................. Dynamic behavior for various 0,- ..................... Co—existence of a local attractor and a global limit cycle ....... Global flow structures of the canonical models. a. System (7.1) for 01 = 02 = —1 is bounded. b. System (7.1) for 01 = +1, 02 = —1 is not bounded. ............................... xi 38 47 48 49 66 77 77 81 84 7.2 7.3 7.4 7.5 7.6 7.7 7.8 7.9 Phase portrait of (7.2) working in the extreme psychological regime. a. Allb,>0. b. b1>0, b2<0. ..................... Phase portrait of the canonical model (7.13) of weakly connected neural network near multiple pitchfork bifurcation point for different values of the bifurcation parameter b. a. b < —61. b. —61 < b < —/32. c. —32 < b < —/32 +(51—fizl/2- d- —/32 +(51— fizl/‘Z < b- ------ Bifurcation diagram. ........................... Every equilibrium point :l:\/b+ 5k k becomes an attractor after the sequence of the pitchfork bifurcations. Every time b crosses —fik + (6. — 3k)/2, s < k, the {Es-direction becomes stable. .......... Phase portrait of the canonical model on the stable invariant plane span(§1,€2). The first image is presented as an input onto the net- work. a. Input is weak, i.e. lal < a*. b. For lal : (1* there are fold bifurcations. c. For [a] > a* the canonical model is globally asymptot- ically stable ................................. Phase portrait of the canonical model on the stable invariant plane span(§l,§2). The input is a mixture of two images {I and £2. a. Overall input is weak. b. Strong input and weak contrast. There is a co-existence of two attractors. ..................... Bistability of perception .......................... Bifurcation diagrams for quasi-static variation of parameter b. a. The contrast c = 0. b. The first image is dominant. c. The second image is dominant. e d. Stroboscopic presentation of the phenomenon for b < —fi,b = —,B and b > —5, respectively. ............... xii 99 107 110 110 116 120 9.1 10.1 10.2 10.3 10.4 10.5 10.6 Depending upon the input (7‘1 or T2) the center manifold is tangent to the corresponding memorized vectors (2)1 or v2) ............ Schematic representation of the neural oscillator. It consists of exci- tatory (white) and inhibitory (shaded) populations of neurons. For simplicity only one neuron from each population is pictured. White arrows denote excitatory synaptic connections, black arrows denote inhibitory synaptic connections ..................... The neural oscillators (dotted boxes) are connected into a network. The mitral cell makes contacts with other mitral cells and may have contacts with other granule cells ..................... A network of two neural oscillators. Open boxes are local populations of excitatory neurons, shaded circles are local populations of inhibitory neurons. The real numbers a1,a2,a3,a4 are entries of the Jacobian matrix of i-th neural oscillator. The real numbers 31,32,33,s4 denote the strength of synaptic connections ................... Differences in dynamic behavior of type A and B neural oscillators. See text .................................. Complex numbers v1,—v2,v3,—v4 as vectors on the complex plane. For simplicity we depict —v2 with zero real part. a. Type A neural oscillator. b. Type B neural oscillator .................. Synaptic configurations that can exhibit the phenomenon described in Theorem 60. Open boxes depict excitatory neurons and shaded circles depict inhibitory neurons. A vertical pair of excitatory and inhibitory neurons is one neural oscillator ..................... xiii 145 149 154 10.7 10.8 10.9 Possible values of synaptic connections cij for different synaptic con- figurations Sij satisfying Dale’s principle. For synaptic configurations that are not explicitly depicted the possible values of c,,- may occupy all shaded areas. a. Type A neural oscillator. b. Type B neural oscillator157 Open boxes are excitatory neurons, shaded circles are inhibitory neu- rons. If there is an arrow between two neurons, then the synaptic contact is possible. a. The synaptic organizations that cannot memorize phase informa- tion. b. The synaptic organization that can either learn or unlearn phase information (but not both). If the network has more than two oscilla- tors, then the Dale’s principle will be violated during the learning. c. The synaptic organization that can learn phase information. d. The synaptic organization that can both learn and unlearn phase information ................................. c,,- and cj, must be inside the shaded area between —v4 and —v2 . . . 10.10For different choices of the plasticity rates 0 the oscillatory neural net— work can learn (k > 0), unlearn (k < 0) or passively forget (k = 0) phase information. Here k = 014—002. Negative value of 02 corresponds to increasing of strength of the inhibitory synapses. Positive value — to decreasing of the strength. ...................... xiv 168 169 Chapter 1 Introduction To study the brain is a challenge not only for neurobiologists but also for mathe- maticians. It is important to understand how it works. It might help us to build a new generation of computers —— neurocomputers. We would understand the nature of many brain diseases and could find ways of curing them. Another noble goal of studying of brain models is possible minimization of wet lab experiments that sacrifice animals. 1.1 Overview In this work we develop bifurcation theory for the Weakly Connected Neural Networks (WCNNs) of the form X,- = F,(X,-,/\) + 56,-(x1,...,x,.,,\,e), 5 << 1, (1.1) where X,- is the activity of the i-th neuron and A is a bifurcation parameter. Historically, the weakness of connections (condition 5 << 1) arises as a techni- cal auxiliary assumption needed for studying networks of oscillators. Although the assumption has strong neurophysiological justification, its importance has eluded re- searches so far. For our work it has paramount importance since it allows us to study (1.1) without even knowing the functions F,- and G.- that describe the real brain dynamics. Indeed, since little is known about chemical and electro—physiological processes taking place inside real neurons, we do not have detailed information about the dynamical system (1.1). Nevertheless, we would like to know its dynamic behav- ior. Especially, we are interested in the regimes where (1.1) has interesting neuro— computational properties, i.e. when it can serve as a prototype for a new generation of computers — neurocomputers. We discuss these issues here. In Chapter 2 we consider WCNN (1.1) at an equilibrium point and study its local dynamics using bifurcation theory. First we show (Section 2.1) that the WCNN must be near a multiple bifurcation point in order to have non-trivial dynamics. This is necessary (but not sufficient) condition for the WCNN to be interesting from the neuro—computational point of view. In biological terms this corresponds to the case when neurons are near thresholds and are susceptible to external perturbations. Then, using the Center Manifold Theorem (Section 2.3), we show that near mul- tiple bifurcations the WCNN (1.1) can be significantly simplified and reduced to canonical models. In Chapter 2 we derive canonical models for multiple saddle- node, pitchfork and Andronov-Hopf bifurcations and multiple cusp singularities (see flowchart in Figure 1.1). In Chapter 3 we analyze local dynamics of singularly perturbed weakly connected systems of the form {flxz’ : Fi(‘Xian9)‘) +€Pf(X’ Y’A’E) 2:1 .. . n [1 <<1 5 <<1 (1'2) K, = Gi(Xia K7 A) + €Qi(Xa Y) A) E:) Such systems describe dynamics of networks of relaxation oscillators or excitable systems or any other systems which have several time scales. We show (Section 3.3) that in many cases (1.2) can be reduced to (1.1). The cases when this reduction is impossible correspond to quasi-static bifurcations in (1.2). In Chapter 3 we derive canonical models for multiple quasi—static saddle-node, pitchfork and Andronov-Hopf bifurcations (see flowchart in Figure 1.1). In Chapter 4 we study local dynamics of weakly connected mappings of the form X; H F(.X,',)\)+EG,'(X,)\,€), €<< 1. We derive canonical models for multiple saddle-node and flip bifurcations. Then we reveal the relationship between these canonical models and the canonical models for weakly connected systems of ordinary differential equations. One of the principal achievements of this work is derivation of canonical models for the WCNNs. The canonical models have few non-linear terms, nevertheless, they. capture the qualitative behavior of all WCNNs of the form (1.1) or (1.2), including possibly the real brain. Thus, in order to study the neuro—computational properties of the brain it is reasonable to study canonical models first. In Chapter 5 we continue to analyze multiple saddle-node bifurcation. In par- ticular, we are interested in how much the canonical models can tell us about the dynamics of the original WCNN (1.1). We study canonical models for multiple Andronov-Hopf bifurcation in Chapters 6 and 10. In particular, we find conditions under which the model can operate as a gradient (Hopfield-type) neural network (Section 6.3). Since the pitchfork bifurcation is a particular case of cusp singularity we study both canonical models in Chapter 7. We show that they can operate as classical neural networks. We concentrate our efforts on the case when the models are taught to recognize only two images (Section 7.6). In this case behavior of the models is somehow close to behavior of the human brain, especially when the two images are presented simultaneously. We illustrate a psychological phenomenon — bistability of ii : fi($i, A) + 591(3) Aapve) Linear uncoupled . hyperbolic Drift E ii :- Drrfil'i non-hyperbolic Canonical model for multiple eigenvalues a pair Of pure imaginary > Andronov-Hopf bifurcation z:- = arzi + bizilzil2 + Z Cijzj one zero Adaptation condition NO Non-linear uncoupled f.(0,A) + €9i(0,/\a/),0) = we?) ' x" = fies“ + 5940”,“) YES 0 Canonical model for multiple ” 7f ; saddle-node bifurcation 1'17 ’_.. .. 2 5r,- — r, +b,.r, +:r,- +5 cum] = 0 B/iit 7g 0 Canonical model for multiple Canonical model for multiple pitchfork bifurcation AND cusp singularity x;=b;$;:l:l‘?+ZCij$j (Bi-ZTg-l-nggiIi-I’d-Zcijftj Figure 1.1: Summary of bifurcations and canonical models for regularly perturbed weakly connected systems PIX:=F1'(X1'3YI1A )+€B(X,YSA9#)€) KI: Gi(XiaYia A)+€Qi(XaYa’\9/19€) Dxc‘Fi hyperbollc y: : 911%,)‘1/1) + 591133» A: #75) V non—hyperbolic Canonical model for multiple quasz— —statzc eigFIB/alpres a pair Of A Andronov- Hopf bifurcation O 1'1' 1' . . ' pure imaginary {74:01: + A viz)zz + b ZzIZdZ + 21'- 1 61‘ij one vi=dii(R +Sl4'31'l2 +Tivi) zero L' . (5) 2 0(5) ; F’ . G" > 0 mear, hyperbolic, uncoupled u - y 22—» 1:,- 2 F531,. l< 0 92' = HIGLiEu 2 Canonical model for multiple 2 0(5 ) Andronov—Hopf bifurcation zf = a,z,- + bizilzil2 + Zcijzj i! . . . . fl # 0 Canonical model for multiple quasz-statzc Fxx f saddle—node bifurcation _ 0 { $1: _yi + $12 + 231:1 Cijmj But 9: = (11(331—7‘1) FIN 7e 0 Canonical model for multiple quasi-static Canonical model for multiple pitchfork bifurcation quasi- -statzc cusp singularity { x: = —yi i x? + ELI Cijmj unfOIding { xi = ”91’ :1: at; + 23:1 Cijrrj yi = agar; 315 = 021118 — 7‘1) Figure 1.2: Summary of multiple quasi-static bifurcations and canonical models for singularly perturbed weakly connected systems perception using these models (Section 7.7). Canonical models for multiple quasi-static saddle-node and pitchfork bifurcations are studied in Chapter 8. There we study how some biological constraints (such as Dale’s principle) affect dynamics of the networks. Studying the canonical models suggests a new neural network type — non-hyper— bolic neural networks, which we discuss in Chapter 9. This new type utilizes the fact that local dynamics near a non-hyperbolic equilibrium can determine global dynamics of a system. This justifies our restriction to consider only local dynamics of the WCNNs. The main result for our WCN N Theory is in Chapter 10. In it we analyze the rela- tionship between synaptic organizations (anatomy) and dynamical properties (func- tion) of the brain. In particular, we show that there are some synaptic organiza- tions that have especially rich dynamic behavior. Comparison of our findings with neurophisiological data shows that these organizations (anatomical structures) are ubiquitous in the brain. We consider this not as a lucky coincidence, but as a Sign that the theory of weakly connected neural networks developed in this dissertation is promising for study of the human brain]. 1.2 Models in Mathematical Biology Unlike in physics, there is no expectation that mathematical modeling in neurobiology could give exact quantitative results. Indeed, neurophysiological processes do not have known conservation laws or symmetry. We do not know and probably will never know all the nature laws that govern brain activity. Thus, we will possibly never be able to write an equation or a system of equations that describes completely a brain’s 1Chapters 2 and 7 received SIAM Student Paper Prize in applied mathematics for 1995 dynamics. Nevertheless, it would be interesting and helpful to have a model that somehow reflects and simulates the brain’s computational abilities. Most models in neuroscience can be divided into the following groups: e Ordinary Language Models are used by biologists to explain how the human brain or some of its structures might work. These models are not mathematical and, hence, are suitably imprecise. e Comprehensive Models are the result of an attempt to take into account all known neurophysiological facts and data. Usually they are too cumbersome and are not amenable to mathematical analysis. A typical example is Hodgkin’s and Huxley’s (1954) model. e Empirical Models occur when one tries to construct a simple model reflecting one or more important neurophysiological observations. A typical example is Hopfield’s (1982) network of McCulloch-Pitts (1943) neurons, based on the fact that neurons could be bistable elements. Although amenable to mathematical analysis, these models are far from reality. e Canonical Models arise when one studies a dynamical system in critical pa- rameter regimes. A typical example of a critical regime is a bifurcation, also referred to as being a phase transition. The major advantage of canonical mod- els is that they describe qualitatively all dynamical systems including the real brain. Their drawback is that they are useful only when the brain operates near the critical regime. Several examples of canonical models are presented in this dissertation. The division above is artificial. There are no exact boundaries between the model types. For example, the Hodgkin-Huxley model could be classified as an empirical Ordinary 'II t t' Em irical Language A 1 US ra 1011 ¢ Mgdels Models 0 .E is” E '15. Comprehensive critical regime _ Canonical Models T Models Figure 1.3: Relationship between models in mathematical biology model because it reflects our knowledge of membrane properties in 1950’s, which turned out to be far from complete. The canonical models are another example: They might be considered as empirical models too because they can be analyzed without resort to computers and / or because they illustrate some basic neurophysiological facts such as bistability. Each of the model types has its own advantages and drawbacks. Neither of them is better or worse than the others. For further discussion see chart in Figure 1.3. Most biologists use the ordinary language models since they are precise where data are known and appropriately vague otherwise. These models do not require knowledge of mathematics. Typical example of such a model is given in the next section as an introduction to the neurophysiology. Using comprehensive models could be a trap: The more neurophysiological facts are taken into consideration during the construction of the model, the more sophis- ticated and immense the model becomes. As a result, the model can become useless since it cannot be reasonably analyzed even with help of a computer. Moreover, as it usually turns out later, the model is not ‘comprehensive‘ at all, i.e. it is only an illusion that the model does include all essential neurophysiological information. Most mathematicians and physicists who study brain functions use empirical models. Usually one has an idea borrowed from an ordinary language model and tries to construct a simple formula that illustrates the idea. The simpler the model the better. As a result, one has a system of ordinary differential or difference equa- tions that might have some computational properties but could be irrelevant to the brain. We believe that if we completely understand the brain we would be able to explain how it works using a simple empirical model. Since we are far away from un- derstanding the brain, invention of empirical models capable of performing something useful is more an art rather than a science. Some successful models are discussed in the review article by S. Grossberg (1988). There are few examples of using the canonical models since there are only two of them known so far. First is the Voltage Controlled Oscillator Neuron (VCON, see Hoppensteadt 1986) model, which can arise when one considers a weakly connected network of excitable oscillators (see e.g. Ermentrout and Kopell 1986). The other canonical model occurs when one considers a weakly connected network of oscillators near Andronov—Hopf bifurcation. We discuss these models as well as derive and analyze other canonical models in subsequent chapters. 1.3 N eurobiological Background It is believed that basic functional unit of the brain is a neuron. Neurons can be different in shape, size and function, but it is possible to describe typical neuron attributes. A typical neuron consists of a cell body, a dendritic arborization and a long axon (see Figure 1.4). Due to nonlinear properties of membranes, neurons can generate action potentials, or voltage spikes —— the basic mechanism of communication between neurons and brain structures. 10 input /\ ._ dendrites \\ synapses / output Figure 1.4: A “typical” neuron. It receives signals from other neurons via synapses and sends the output signal to the other neurons through its axon. A simplified description of neuron activity is the following: Its dendrites receive input signals from other neurons via synapses — the contacts between neurons; after spatio—temporal processing, which takes place in the dendrites and the cell body, a neuron might generate a response, the action potential, that propagates along the axon to other neurons and serves as an input onto their dendrites. Neurons can have synaptic contacts with many thousands of other neurons. How the neurons communicate is still a mystery. The signal could be encoded as the number of spikes per unit of time or as inter-spike intervals. It is also feasible that the signal can be encoded as phase differences since most of the neurons tend to generate the action potentials repeatedly. Using various chemicals it is possible to suppress nonlinear membrane properties responsible for generation of the action potentials. In this case one action poten- tial of a neuron induces very small changes in the membrane potential of another neuron (i.e. the excitatory post-synaptic potential (EPSP) is less than 1 percent of the amplitude of the spike). This observation lies at the foundation of a theory of weakly connected neural networks. Nevertheless, in normal conditions even small 11 perturbations of membrane conductance may evoke substantial changes in membrane potential because neurons are usually very close to thresholds and can easily generate action potentials. Thus, we can characterize a neural network of the human brain as being a highly nonlinear and extremely sensitive system. We will see in the next chapter that these are desirable properties for a brain to have. 1.4 Neural Network Types It is believed that our memories correspond to (meta-stable) attractors in our brain’s phase space, which is huge since the human brain has more than 1011 neurons (Shep- herd 1983). Convergence to an appropriate attractor is called recognition. There are many Neural Network (NN) models that can mimic recognition by association processes. In general, NN is a network of interconnected simple elements, usually called neu- rons, that performs computational tasks, such as recognition by association, memo- rization, etc., on a given input (key) pattern. The pattern may be temporal or spatial or both. In the N N types discussed here, the input pattern does not depend upon time, i.e. input is quasi-static. Most of the NNs dealing with static input patterns can be divided into two groups according to the way in which the pattern is presented: MA —type (Multiple Attractor NN.) The key pattern is given as an initial state of the network. And the network converges to one of many possible choices (Hopfield 1982, Grossberg 1988). GAS —type (Globally Asymptotically Stable NN.) The key pattern is given as a pa- rameter which controls the location and shape of a unique attractor (Hirsch 1989). 12 Input I Input 3 O . 1 Image 1 ,. _ Image 2 a. ...................... [Ml-“\‘x l l npu : _. . .— e <—" ........ I Image 3 a b Figure 1.5: a. Multiple Attractor (MA) neural network. b. Globally Asymptotically Stable (GAS) neural network. In either case the N N must converge to an equilibrium. Each of MA—type network attractors corresponds to a prior memorized image. Each input pattern, considered as an initial state, lies in a domain of attraction of some of the attractors (see Figure 1.5a). A ”Good” MA-type NN is one that can memorize many images; It does not have spurious (or false) memory, i.e. the attractors that do not correspond to any of the previously memorized images; All its attractors have ”large” attraction domains. A typical example of the MA-type NN is the Hopfield network (Hopfield 1982), although it is not a ”good” network since it has spurious memory. The GAS—type NN has only one attractor that depends upon the input pattern as a parameter (see Figure 1.5b). Various inputs can place the attractor on various locations in the network’s phase space. Learning in such a NN consists of adjusting the connections between the neurons so that the network can realize the mapping to be memorized. of the attractor. { Patterns } { Prescribed locations } A ”good” NN of this type should memorize many such mappings so that if the key pattern to be recognized is near (in some metric) to a previously memorized pattern, then the resulting attractor is in the prescribed location. All attractors usually considered in these NNs are equilibrium points, although 13 there are many attempts to understand the role of limit cycles and chaotic attractors in brain functioning (Baird 1986, Eckhorn et.al 1988, Gray 1994, Hoppensteadt 1989, 1986, Izhikevich and Malinetskii 1992,1993, Kazanovich and Borisyuk 1994, Skarda and Freeman 1987, Tsuda 1992). We will show in subsequent chapters that the canonical models that we derive can work as both MA and GAS-type NNs. Notice that the process of recognition in the NN types presented above is essen- tially a global phenomenon. But the canonical models describe only local dynamics of WCNN near some equilibrium. Studying the canonical models suggests a new NN type, which we discuss below. 1.4.1 Olfactory Bulb Let us consider the well studied brain system comprising the Olfactory Bulb (OB). The neurons of OB receive signals from olfactory receptors, process information and send it to other parts of the brain. Anatomically the OB consists of neural oscil- lators (see Figure 1.6). The processing of information by the OB has been studied neurophysiologically (see, for example, Shepherd 1976,1983 and Skarda and Freeman 1987) and using mathematical models (see, for example, Baird 1986, Erdi eta] 1993, Izhikevich and Malinetskii 1993 and Li and Hopfield 1989). It is believed that in the absence of the signals from olfactory receptors the OB’s activity is chaotic and of low amplitude. W. Freeman called this background chaotic activity the dormant state. The signals from receptors make the chaotic activity co- herent and periodic. Each inhaled odor excites its own pattern of spatial oscillations. From a mathematical point of view, each odor is represented by a limit cycle in the OB’s phase space. To study the limit cycles requires global information about the 14 Olfactory Nerve Granule cell Figure 1.6: Schematic representation of the Olfactory Bulb (OB) anatomy. Activity of the mitral cells is denoted by X,- and granule cells by Y,. OB dynamics, but to study how the chaotic attractor corresponding to dormant state changes requires only information about local behavior near the attractor. We hypothesize that it is possible to predict to which limit cycle the OB’s activity will be attracted simply by studying the local bifurcations of the attractor. Thus, one could say that the future of the OB dynamics is determined by local events. This is the spirit of our non-hyperbolic NNs approach. Unfortunately, it is difficult to study chaotic attractors. In this thesis we assume that when input from receptors is absent, the attractor corresponding to the dormant state is an equilibrium point. In this case one can think of the low-amplitude chaos as being noisy perturbations of an equilibrium. The limitation of this approach is that we neglect possible role of deterministic chaos in information processing (Izhikevich and Malinetskii 1992,1993). 1.4.2 Networks of Excitable Elements The example of bifurcations in the olfactory bulb discussed above is vague. Below we present a precise mathematical illustration of the non-hyperbolic phenomenon that we study. action f potential a b c Figure 1.7: An excitable system which mimics a neuron dynamics. The neuron can be near threshold (a), hyperpolarized (b) or depolarized (c). Consider an excitable system with nullclines as depicted in Figure 1.7a. Suppose the input to this system shifts one of the nullclines, say C = 0, to the right (excita- tion) or to the left (inhibition). Dynamical systems with such attributes mimic some important properties of real neuron dynamics and are studied in Chapter 3. When the input is inhibitory (see Figure 1.7b), the equilibrium point is asymptoti- cally stable. If the initial state of the system is close to the equilibrium, the dynamics converge to it. Neurobiologists might say that such neuron is hyperpolarized and silent. When the input is excitatory (see Figure 1.7c), the equilibrium becomes unstable. Neurobiologists say that the neuron is depolarized and can generate action potentials or spikes. The spike is a global phenomenon in the sense that it is observable on the macro—level. But it is caused by a local event — temporary loss of stability by the equilibrium. This is possible because the equilibrium (the intersection of the nullclines) in the excitatory systems (Figure 1.7a) is nearly non-hyperbolic. One can say that excitatory systems can perform a trivial pattern recognition task: They can discriminate between excitatory and inhibitory inputs. Now consider a network of such excitatory elements (neurons). Suppose the in- put is inhibitory and strong. Then all neurons are hyperpolarized. Activity of the 16 no input input #2 non-hyperbolic equilibrium Figure 1.8: Basic principles of the non-hyperbolic neural network functioning. The input is given as a parameter which perturbs the non-hyperbolic equilibrium. Local bifurcations of the equilibrium affect global dynamics of the network. network converges to some equilibrium. The network is silent. Next suppose the input is shifted quasi—statically towards excitation until the global equilibrium loses its stability. The dynamics of the network might produce some macroscopic changes. For example, some of the neurons can generate action potentials whereas the others remain silent. The active neurons can send their signals to other parts of the brain and trigger various behavioral responses, such as attack or escape. This active re- sponse of the neural network depends on how the equilibrium loses its stability, which in turn depends on the input and the connections between the neurons. Thus, global behavior of the network crucially depends on local processes taking place when the equilibrium becomes non-hyperbolic. We can summarize the above as the following: NH -type (Non-Hyperbolic NN.) The input pattern is given as a parameter which perturbs a non-hyperbolic equilibrium (see Figure 1.8). To summarize, we can say that even local information about the brain’s dynamic behavior can be useful for understanding its neuro—computational properties. In subsequent chapters we derive canonical models for general weakly connected neural networks. The canonical models describe brain’s local activity, so we analyze them not only from MA and GAS—type N N point of View, but also from NH-type NN 17 point of View. 1.5 Generic Equations Before writing equations let us introduce some useful definitions. Among them there are two that cannot be strictly defined (like notion of the set in mathematics). By local population of neurons we mean a set of strongly interconnected neurons that are close to each other, have approximately the same pattern of synaptic con- nections and dynamic behavior. For instance, it could comprise those neurons in a cortical column, in an olfactory bulb glomerulus, etc. We can think of activity of the population as the number of action potentials per unit time, or as an amount of chemical neurotransmitter released by synaptic termi- nals, or any other physiological observable. Another point of view is that the activity of the population of neurons is a variable from a (possibly infinite-dimensional) Ba- nach space that describes all electro—physiological and neuro—chemical properties of the neurons: Spatial distributions of membrane potentials and neurotransmitters; activity of receptors, ion channels and pumps, etc. Of course, these are not the precise definitions required for mathematical modeling, but they give us some flexibility in interpretation of results. Indeed, any unfolded and explicit definition of them means a deliberate restriction of the set of phenomena that could be described by our mathematical modeling. Finally, the following abuse of notation is widely accepted in the neural network literature: We will call a population of neurons simply a neuron and the activity of the population the activity of the neuron. Let M be a manifold and let X,- E M denote the activity of the i—th neuron. 18 Suppose that the dynamics of each X,- can be described by the dynamical system X,=F,(X,~,A), AEA, X,-€M, i=1,...,n. (1.3) where the Banach manifold A is a parameter space and n is the number of neurons. We will assume that the unknown functions F,- describing the real neuron activity are as smooth as it is necessary for our computations. Equations (1.3) considered as a system describe dynamics of uncoupled neural network because the dynamics of each neuron X,- depends only upon itself and does not depend upon the other neurons. The basic assumption of the theory of weakly connected neural networks is that the contribution of activity of one neuron to activity of another one is very small, say, of order 8 << 1. Definition 1 A Weakly Connected Neural Network (WCNN) is a dynamical sys- tem of the form Xi=Fi(4Xia/\)+€Gi(‘x’19"-3‘¥ns’\ip75)a A36 M7 i=1,...,n, (14) where C,- describes synaptic connections between the neuron X,- and the other neurons X1, . . . ,Xn and receptors p E ’R; ’R is a Banach manifold and parameter 5 << 1. Note that the activity of i-th neuron X,- depends strongly only upon itself and the parameter A. One can consider (1.4) as an e-perturbation of the (uncoupled) system (1.3). We refer to (1.4) as being a regularly perturbed WCNN. For reasons that will become clear later, we might call (1.4) the equation that describes the physiology of WCNN and the state variable X = (X1,...,X,,) E M" describes physiological state of the brain. In contrast, the parameter A describes the psychological state of the brain. In the analysis below we assume that the network (1.4) works in a psychologically quasi—static regime, i.e. when A is changed so slowly that it could be assumed to be constant. We will return to this issue in Section 2.6.1. 19 Many processes in the human brain can be described by singularly perturbed, weakly connected dynamical systems of the form {ILtXtI = E(4Xi7)/ia)‘) +€PI(X’Y’A’p’€) 1:1 Tl. (1.5) K, : Gi(XiaYia A) + €Qi(Xa YaAipaE) Here each pair (X,, Y.) E M denotes activity of the i-th neuron. The small parameter )1 denotes a ratio of time scales. Thus {X,-} are ”fast” and {K} are ”slow” variables. Variables X,- and Y,- could also denote the activities of local populations of excitatory and inhibitory neurons, respectively. Then (1.5) could describe dynamics of weakly connected networks of relaxation neuron oscillators such as those depicted in Figure 1.6. We study both dynamical systems here. Our strategy is the following: Suppose we know the behavior of each neuron and, hence, the behavior of the uncoupled systems. What new dynamic behavior can emerge as a result of introducing arbitrary small coupling 5 between them? The behavior of each neuron might be chaotic, and introducing coupling can destroy the chaos or make it more complicated. Even in the simpler case when the dynamics of each neuron is periodic, it is still possible to observe very complicated nonlinear phenomena such as chaos, phase locking or synchronization in the coupled system (Hoppensteadt 1993). Below we study only the case when the dynamics of each neuron is convergent and the attractor is an equilibrium point. Hence, the dynamics of the network of uncoupled neurons is also convergent to a equilibrium point. We will study WCN N in some neighborhood of the equilibrium point and will show that under some conditions its dynamics is very rich. Part I Derivation of Canonical Models 20 Chapter 2 Weakly Connected Neural Networks In this chapter we study local dynamics of a WCNN, described by the dynamical system X,- = F,(X,,A) + 5G,(X1,...,X,,,A,p,s), i = l,...,n, (2.1) near an equilibrium point. First we show that the dynamics is not interesting from neuro—computational point of view unless the equilibrium corresponds to a bifurcation point. In biological terms such neurons are said to be near a threshold. Using a Center Manifold reduction we prove the Fundamental Theorem of WCNN Theory, which says that neurons should be close to thresholds in order to participate in brain dynamics. Then we show that a WCNN near a bifurcation point can be significantly simplified and reduced to a canonical model. We derive the canonical models for multiple saddle- node, pitchfork and Andronov-Hopf bifurcations and for multiple cusp singularities. 2.1 Hyperbolic Equilibrium Since we are interested in local structure, we may assume that M is a Euclidean space Rm for some m > 0. We may also assume that A and R are some Banach spaces. 22 To avoid awkward formulas, we will use the following vector notation X = (X1,...,X,.,)T E M" F(X,A) = (15‘1(X1,A),...,F,,(X,,,A))T 2M” x A —> M" C(X,A,p,e) (G1(X,A,p,e),...,G’n(X,A,p,e)) : M" x A x 7?, x R—-> M”. Then we can rewrite (2.1) more concisely as X = F(X,A) + eG(X,A,p,e). (2.2) Note that F (X ,A) has diagonal structure. Without loss of generality we also may assume that when 5 = 0 the equilibrium point is the origin X = 0 for A = 0, i.e. F(O, 0) = 0. Let L,- be the Jacobian m x m-matrix of first partial derivatives of the function F,(X,',0) with respect to X,- at the origin 0F,- Li : DXtFi(0a0) : (0Xk(010)) ' '3 k,j=l,...,m When L,- has no eigenvalues with zero real part, the equilibrium point is said to be hyperbolic. It is easy to see that the Jacobian matrix L = DXF(0,0) of the right hand-side of (2.1) for e = 0 has the form L1 0 o 0 L2 0 = z (2.3) 0 0 L. The main purpose of this section is to prove the following 23 Theorem 2 If the dynamics of each neuron is near a hyperbolic equilibrium, then the weakly connected network (2.2) of such neurons, the uncoupled network X5 = F,’(X,',/\), t: 1,. . . ,n (2.4) and the linear system X=LX mm have topologically equivalent local flow structures. Hence, the local dynamics of (2.2) is the direct product of the single neuron dynamics and, therefore, the entire neural network is, in this sense, no more complex than a single neuron, which has locally linear dynamics. Therefore, a hyperbolic WCNN is not interesting as a brain model. Proof. Since each L,- is hyperbolic, then so is L. In particular, it is nonsingular. The Implicit Function Theorem guarantees existence of a unique set of smooth func- tion X = h(e, A,p) : IR X A x R —> Rm" defined in a neighborhood of the origin such that h(0,0,p) = 0 and for all sufficiently small 5, A and bounded p. This means that near the equilibrium point X = 0 of the unperturbed (uncoupled) system (2.4) there is a unique equilibrium point X of the perturbed (coupled) system (2.2). Note that we do not require that parameter p E R be small. This is possible because we assume that C(X, A,p,e) is a smooth function, which is multiplied by 5. Hence, for small 5 it suflices to require only that p be bounded. By continuity properties of the spectrum of linear operators no eigenvalues can cross the imaginary axis provided the parameters are sufficiently small. Hence, the equilibrium 2? is also hyperbolic with eigenspaces and invariant manifolds of the same 24 dimensions as those of the unperturbed (uncoupled) system (2.4). In particular, if one of the equilibrium points is stable, then so is the other. Local topological equivalence of the flows of (2.4), (2.2) and the linear system (2.5) follows from Hartman-Grobman Theorem (Guckenheimer and Holmes 1983). CI Note that the above analysis is also valid when M is a Banach manifold. In that case, instead of Jacobian matrices we consider Fréchet derivatives, but along with hyperbolicity we must impose some technical conditions (for instance, the Fréchet derivatives must have bounded inverse), which are always met in finite dimensional case. Corollary 3 The only equilibrium points that require further discussions are non- hyperbolic ones. 2.2 N on-Hyperbolic Equilibrium We consider the unperturbed (uncoupled) system (2.4) and compare its behavior with the behavior of perturbed (coupled) system (2.2). We seek such changes that endow (2.2) with non-trivial neuro—computational abilities. In the previous section it was shown that a necessary condition for a weakly con- nected neural network (2.2) to exhibit local nonlinear behavior is non-hyperbolicity. Next we consider (2.2) “near” a non-hyperbolic equilibrium, as explained later on. Suppose that the origin X = 0 is a non-hyperbolic equilibrium point of (2.4) for A = 0. This means that the Jacobian L has eigenvalues with zero real part. Due to the diagonal structure of L, this is possible only if one or more of the Jacobians L, have eigenvalues with zero real part. Depending upon the type and number of them there might be various cases: 25 e Some of the Jacobian matrices L; have many eigenvalues with zero real part. Comprehensive analysis of this case is difficult and has not been done yet. e Some of the Jacobian matrices have a pure imaginary pair of eigenvalues (i.e. the neurons are near an Andronov-Hopf bifurcation point). This case has at— tracted much attention in recent years due to its connection with synchroniza- tion phenomena (Ermentrout and Kopell 1990; Hoppensteadt and Izhikevich 1995; Kopell 1986). Comprehensive analysis of a system of two such neurons can be found in (Aronson eta] 1990). We study this case in Section 2.5. e Some of the Jacobian matrices have only one simple zero eigenvalue, whereas the others have no eigenvalues with zero real part. The last case appears to be the simplest one. Thus, the natural way to start studying weakly connected neural network dynamics is to explore this case. We concentrate our efforts on this case, although the theorem that we prove in the next section is applicable to any non—hyperbolic weakly connected system. 2.3 The Center Manifold Reduction Without loss of generality we may reorder the system and so assume that only the first k equations in (2.1) have a non-hyperbolic Jacobian matrix L,. We will study the most interesting case when all nonzero eigenvalues of L,- have negative real parts. Let us represent the phase space M E Rm of i—th neuron (i = 1,. . . , k) as a direct sum M=E®E, where the center subspace Ef is spanned by the eigenvectors of L,- that correspond to eigenvalues with zero real parts and the stable subspace Ef is spanned by the other 26 (generalized) eigenvectors. We will use the notation EC = Ef x >< Ef. Theorem 4 Suppose that each of the first k Jacobian matrices L,- = DX,F,-(0,0) is non-hyperbolic. Then the WCNN (2.1) is locally governed by a dynamical system of the form i,- = f,-(:r,-, A) + egi(.r, A,p,e), i = 1,. . . ,k, (2.6) where :13,- E Ef and J,=D,.,f,-(0,0)=L i=1,...,k. ilEC , In particular, J,- have all eigenvalues with zero real parts. More precisely, there is a function Z : EC x A X R x IR —> M" such that any local solutions X(t) of (2.1) tend erponentially to Z(.r.(t), A, p, e), where :r(t) E EC is some solution of (2.6) determined by X. Proof. Proof of the theorem uses the Center Manifold Theorem and the fact that the Center Manifold has a convenient weakly connected form. Our treatment of the Center Manifold Reduction is based on that of Iooss and Adelmeyer 1992, Section 1.2. Let 7rf : M —) Ef denote a projection operator such that ker NE 2 Ef. 27 Let :13,- = qu, 6 BE. We use the notation a: = (1:1, . . . ,crk)T 6 EC. In order to apply the Center Manifold Theorem, we must consider the auxiliary system X = F.(X,-, A) + eG,(X, A, p,e) II" II COG )\ p 5' at the equilibrium point (X, A,e) :2 (0,0,0). Its center subspace is ECxAxRsz {(ar.x\.p.€) l m E EC;A€A;p€R;€€ R}- Applying the Center Manifold Theorem (Iooss and Adelmeyer 1992), we obtain the function H=(H1,...,Hn) I ECXAXRXR —) Eisx...XEZXMn—k with H(0,0,p,0):0 and D,H(0,0, p,0) = o, (2.7) such that for A and e sufficiently small and for bounded p E R the manifolds M(A,p,e) = {1: + H(:1:,A,p,e) I .r 6 EC} are locally attractive and invariant with respect to (2.1). Furthermore, :r on M(A, p, e) is governed by a dynamical system of the form 13,-: 7T? (We + lit-(iv. Ante). A) + €Gi(fv + H(:v. A.p.€). A.p,€)) (2-8) for i = 1,. . . , k. Since M(A,p, e) is locally attractive, all local solutions of (2.1) tend exponentially to those of (2.8). Therefore, the function Z mentioned in the theorem has the form Z(1:,A,p,e) = :1: + H(a:,A,p,e) 28 for ac 6 EC. Let us show that H has a weakly connected form. For this, notice that M(A, p, 0) is the Center Manifold for the uncoupled (e = 0) system X,- = F,(X,~,A) i=1,...,.n Hence, the function H(x, A,p, 0) has uncoupled form . __ V,(:r,-,A) fori=1,...,k, Hz($,AapaO)_{ VXA) for izk+1,,”,n, where V;- are some functions. Recall that we assume that all data are as smooth as it is necessary for our computations. Therefore, H depends smoothly on e, and for e # 0 we can rewrite it in the weakly connected form _ V,(;r,~,A)+eW,-(:r,A,p,e) fori=1,...,k, H(:r,A,p,e)—{ I/;-(A)+5Wi(a:,A,p,e) fori=k+1,...,n, where W,- are some functions. The functions F, in (2.8) can also be rewritten in the weakly connected form F,(:1:.- + Vila. A) + (SI/Vilma A,pfl). A) = me, + v.(:c,-, A),A) + smut/1.5) for some functions 13}. If we denote for i = 1,. . . , k f,(.r,-, A) = 7r,-CF,-(:L‘,- + V}(:r,~, A), A) and g,-(:r, A,p, e) : 7rf (F;(.’L‘, A,p,e) + G.(:r + H(:r, A,p,e), A,p,e)) then equations in (2.8) can be written as (2.6). Now note that J,- 2 DAL-(0,0) = D,,7rfF,-(:c,- + W($i,0).0)|.,=o = 7.51),, F,(0,0) - (E + D.,t/,-(0,0)), 29 where E is the unit matrix. From (2.7) it follows that Bald-(0,0) = 0. Recall that Data-(0,0) = L.-. Hence J,’ = Tic-:L,‘ = LflT-C = L i l 'lEf' Remark 5 The theorem can be restated concisely as follows: The center manifold for a weakly connected system has weakly connected form. Remark 6 A generalization of the theorem for the case when the J acobian matrices may have eigenvalues with positive real parts is straightforward. The center manifold is not attractive in this case. Most important cases when the Jacobian matrix L,- is non—hyperbolic correspond to bifurcation of the i~th neuron dynamics. Activity of such a neuron is sensitive to external influences, and biologists say that such a neuron is near threshold. The fact that only the first It neurons participate non-trivially in (2.6) motivates the following result: Corollary 7 (The Fundamental Theorem of Weakly Connected Neural Network Theory.) In order to make a non-trivial contribution to brain dynamics, a neuron must be near threshold. The Fundamental Theorem is not totally unexpected for neurobiologists. It cor- roborates their belief that in the human brain, which is undoubtly an extremely complex system, all neurons must be near thresholds or, otherwise, the brain could not be able to cope with its tasks. 30 2.4 Canonical Models In this section we consider a weakly connected network of non—hyperbolic neurons described by a dynamical system of the form (2.6) ii:fi($ia)‘)+€gi($a’\ipi€)a i=1,...,n, where x,- 6 IR. is a scalar and for all i. We study (2.6) for p = p(e) and A = A(e) such that p(0) 2 p0 and A03) : 0 ‘1' EA] '1' 52A2 "1' C(53), /\1, A2 E A, (2.10) i.e. A(e) is e-close to the bifurcation value A : 0. This is not a restriction on p and A since we allow p0, A1 and A2 to assume any values. Definition 8 We say that A satisfies the adaptation condition if D.\fi(090)’\1 + 911010» P010) : 0 (211) for all i. The adaptation condition (2.11) is equivalent to the conditions fi(03’\)+€gi(0103p90) :O(€2)a i=1,...,n, which can be explained in ordinary language as: The internal parameter A counter- balances (up to order 5) the steady state input from the entire network onto each I181] I'OIl . 31 Notice that the representation (2.10) is not necessary for the adaptation condition to be satisfied. Thus A(e) can be any function (differentiable in 5), although it is easier to work with smooth functions of the form (2.10). The adaptation condition can also be restated in terms of the original weakly connected system (2.1). In this case it suffices to demand that A counterbalances the input from the network along with EC direction. 2.4.1 Multiple Saddle-Node Bifurcation A dynamical system if = fi(iriaA)7 113i 6 R is near saddle-node bifurcation point .13.- : 0 for A = 0 if avfi(010) 0’ 3.1., p. —. 40,0) aé 0 (2'12) and 01.23010) ¢ 0» (2-13) although in the theorem below we do not require condition (2.13). If all equations in the uncoupled system (1.3) are near bifurcation points of the same type, then the bifurcation is called multiple. Theorem 9 If the WCNN {2.6) is near a multiple saddle-node bifurcation and for the external input 10(5) = po + 5101+ 0(52), pom E R (‘2-14) the internal parameter A satisfies the adaptation condition (2.11), then the invertible change of variables .r,- = spyli, (2.15) 32 and introduction of the ”slow” time 7' 2 et transforms (2.6) to a3:=r,+b,x;+x?+Zc,-jxj+0(e), i=1,...,n, (2.16) j¢i where ' = d/dr and we have erased ~. One can think of {2.16) as being the canonical model for multiple saddle-node bifurcations in WCNNs. Proof. Using (2.10) and (2.14) we can write the initial part of the Taylor series of (2.6) in the form 13,-: ea, + 52d,- + 5 Z 3013+ par? + h.o.t., (2.17) 1:1 where ai=D1frA1+gi EIR 0 d1: DAfi ' A2 +(Dif1)'(/\1,A1)+ D.\g."/\1+ nga'p1+ E913 _ M U - 8551' for i 75 j, and aft 891‘ 311— Dng ' A1 + 0—1:; where all derivatives are evaluated at (.r, A,p,e) = (0, 0,p0,0). The constants p,- 3£ 0 were defined in (2.12). Notice that the adaptation condition implies a,- : 0. If we denote 7". = it'd.» _ —l Cij — 19181ij . bi = Cit and use rescaling (2.15), then (2.17) transforms into (2.16). CI 33 2.4.2 Multiple Cusp Singularity A dynamical system .73" = fi(.’17,',/\), 1,6 IR is a near cusp singularity 5r..- 2 0 for A = 0 if i (070) = 232:? i(030) : 09 8.1:. ' (Ii = %f1(0.0) 75 0 (2.18) Theorem 10 If the WCNN {2.6) is near a multiple cusp singularity, if the external input is p(€) = M) + fipg + 0(5), 100.10% 6 R (2-19) and if the internal parameter A satisfies the adaptation condition {2.11), then the invertible change of variables 1‘; = «Emil—iii (2.20) and introduction of the “slow” time 7' = st transforms (9.6} to .13:- = r, + biCL‘,’ + mm? + Zen-.13 + O(\/E), a, = :tl, i = 1,...,n, (2.21) 19“ where ’ = d/dr and we have erased ~. One can think of {2.21) as being the canonical model for multiple cusp singularities in VVCNNs. Proof. The initial portion of the Taylor series of (2.6) has the form in = 5\/c:,(l,' + 5 Z SiJ‘LL‘j + (11'1"? + h.o.t., (2.22) 1:1 where Sij and q,- were defined earlier, and d. = ngi(0.0,po.0) ' p . .1. 2 34 If we denote Cij = lqilisulqz'l"%, bi = cm ; Ti = Wild.» 0,- 2 sign q,- and rescale using (2.20),then (2.22) transforms into (2.21). D 2.4.3 Multiple Pitchfork Bifurcation In Theorem 10 we demanded that the deviation of p(e) from p0 be of order \/E (Formula (2.19)). If we require that p(€) have the form as in (2.14), i.e. p% = 0, then r,- = 0 in (2.21), and (2.21) has the form :1):- 2 bar,- + 0,1? + ECO-1',- + (Oh/E), o.- = :1:1, i = 1,. . .,n. (2.23) i?“ One can think of (2.23) as being the canonical model for a multiple pitchfork bifurca- tion in WCNNs because (2.23) is exactly what one receives if one considers (2.6) with functions f,- and 9,- having Z2 symmetry, i.e. invariant under the reflection at,- —> —.r,-. In this case the adaptation condition (2.11) is satisfied automatically. In canonical models (2.21) and (2.23) the choice 0,- = +1 (0,- = —1) corresponds to a subcritical (respectively supercritical) bifurcations in the i-th neuron dynamics. 2.5 Multiple Andronov-Hopf Bifurcation The assumption that connections between neurons are weak arose first as an auxiliary assumption for networks of oscillators near Andronov-Hopf bifurcations. In our work it is a primary assumption and networks of weakly connected oscillators are just one of many possible scenarious to be studied. We start discussion of this case below, and we continue it in Chapters 6 and 10. 35 A dynamical system ii : fi($ia A) is near Andronov-Hopf bifurcation if the Jacobian matrix L.- = Dr,f,-(0,0) has a simple pair of pure imaginary eigenvalues. Using the Center Manifold Reduction for WCNN (Theorem 4) we may assume without loss of generality that L,- is a 2 x 2- matrix. Let iiQ, be the eigenvalues of L, where 91' = V det Li. Let v,- E C2 and v,- E (C2 denote the (column) eigenvectors of L, corresponding to eigenvalues i9,- and —iQ,-, respectively. Let w, and U").- be dual (row) vectors to v, and 23,-. Let V.- be the matrix whose columns are v,- and 27,-, i.e. Vi = (viii—u). Notice that Vfl has w,- and w,- as its rows. Theorem 11 If the WCNN (2.1) is near a multiple Andronov—Hopf bifurcation, then there is an invertible change of variables x-(t) = VEV( ””2“” ) + 0(a) (2 24) ,, , ) , . fatal-(r where T 2 at is a ’slow’ time, which transforms the WCNN to z£=b.z.-+d.z.-|z.-|2+ Z Cijzj+0(\/E), (2.25) 9.29] where ’ = d/dr, b,,d,-,z,- E (C, and the (synaptic) coefficients Ci,- 6 (C are given by Ci,- 2 w,- ' Drjgi -v,-, (2.26) 36 System (2.25) is the canonical model for a WCNN near multiple a Andronov—Hopf bifurcation. Proof. Consider the uncoupled (e = 0) system ii = fi($i, 0)- The change of variables transforms the system to i’i = iQiZi + h(3i1§i)a where h accounts for all nonlinear terms in .22 and 2,. Next, we use the well-known fact from Normal Form Theory (Arnold 1984, Guckenheimer and Holmes 1983) that there is a near identity change of variables 3:; Z 2,’ + p,’(Z,', 531‘), (2.29) which transforms (2.28) to P}; 2 10,2, + d,::,-|:,-|2 + O(IZ,'|5) for some d,- E (C, where we have erased N. Now suppose 5 ¢ 0 and (2.10) holds. Then we apply the composition of (2.27) and (2.29) to (2.6) and receive 71 2}; = ill-z,- + diziIZglz + E 2(Cij2j + egjfj) + O(lz,|5,elz|2,€2|z|). i=1 Introducing the ’slow’ time T 2 st and changing variables aim = we? (r) in (2.30) 37 transforms the system to n -n n- - , 2 1_.L-_L 1 Z, = dgzilzil + E (6 ‘ Tc,,-z, + 6 1:1 _Q_ #Teifij) + 06/5), (2-31) where we have erased ~. After averaging all terms that have the factor el'i‘T for 6 75 0 vanish, and we obtain (2.25), where b,- = c,-,-. It is easy to see that (2.24) is a composition of (2.27), (2.29) and (2.30). D Alternative proofs of this theorem can be found, for example, in Ermentrout and Kopell (1992) or in Hoppensteadt and Izhikevich (1995). Notice the remarkable resemblance of (2.23) to (2.25). The latter has the same form as the former except that all variables are complex. 2.5.1 Equality of Frequencies and Attention It is customary to call 9,- the natural frequency of the i-th neuron. Since /\(5) = 0 + 5A1+ 0(52) each oscillator 13,: f,(.r,-, /\(e)) + €g,(0, ..a‘,, .., 0, /\(€), p, E) for e 75 O has natural frequency wf(e) = Q.- + aw.- + 0(52), which is e-close to 0,. Therefore, when we say that a pair of oscillators have equal frequencies, we actually mean s-close frequencies. One direct consequence of Theorem 11 is the following Corollary 12 All neural oscillators can be divided into groups, or pools, according to their natural frequencies. Oscillators from different pools have different natural frequencies and interactions between them are negligible {see Figure 2.1). @ @612) ------- functionally insignificant connections functionally significant connections Figure 2.1: Synaptic connections between neurons having different natural frequencies are functionally insignificant. Therefore, the network can be divided into subnetworks (pools) of oscillators having equal or e-close natural frequencies Proof. From (2.25) it follows that the i-th neuron dynamics depends on the j-th neuron dynamics only if Q,- = 0,. Indeed, if Q, — Q,- = w;(0) — wf(0) 75 0, then the term c,J-z,- in (2.31) vanishes after the averaging. D Thus, oscillators from different pools work independently from each other even when they have nonzero synaptic contacts Cij, i.e. one neuron can ”feel” another one only when they have equal natural frequencies. It is reasonable to speculate that the brain has a mechanism to regulate the natural frequencies 9,- of its neurons so that some of them can be entrained into different pools at different times simply by adjusting 9,. This might be related to such phenomena as attention and dominanta (Hoppensteadt 1991, Krukov 1991, Kazanovich and Borisyuk 1994). Remark 13 There are interactions between neurons even when 0,- 75 (2,, but these interactions have smaller order and are hidden in the term (9(f) in (2.25). They are noticeable only on time scales of order 0(1/\/E) (for “slow” time T, or 0(1/e\/E) for normal time t) and are negligible on shorter time scales. For example, if all oscillators have different natural frequencies, then we must 39 study weakly connected dynamical systems of the form 2:- = biz,- + d,-z,~|z,-|2 + «En-(21,. . ., 2”, fl, T), 2, E (C (2.32) for some functions pg. If all Re b,- < 0, then the uncoupled (e = 0) system has an asymptotically stable hyperbolic equilibrium 21 = = 2,, = 0 and dynamics of coupled (e 75 0) system is trivial. When Re b,- > 0 for at least two different i’s and corresponding Re d,- < 0, then (2.32) is a weakly connected network of limit-cycle oscillators. Such networks are studied elsewhere. Suppose the whole network is divided into two or more subnetworks (pools) of oscillators each having equal frequencies. Then it is natural to study dynamics of one such pool first, and then study how these pools interact. In order to study the first problem we may assume that 91 = = 9,, = Q, i.e. the whole network is one such pool. We use this assumption in Chapters 6 and 10. In order to study the interactions between the pools we have to consider a dy- namical system of the form X: = F,(.X’,) + \/e:G,(.X1, . ..,Xk, \/E,r), i = 1,...,k, where X,- = (2,1, . . . ,z,ml) describes activity of the i-th pool and k is the number of pools. Such a system is weakly connected, it coincides with 2.1), and, hence, can be studied by the bifurcation methods developed in this work. 2.6 Discussion Consider the canonical models. In the neural network literature it is customary to call the matrix C = (c,,-) the synaptic (or connection) matrix. It is believed that C describes memorization of information by the brain. We will show that the same is true for the canonical models: They have interesting neuro-computational properties. 40 In particular, they can perform pattern recognition and recall tasks by association —— a basic property of the human brain. Parameters b,- are called bifurcation parameters because they depend on A. Each parameter r,- depends on A and p and has the meaning of rescaled external input on the i-th neuron. It is convenient to think of po 6 R in the expresion p(€) = po + 6m + (9(52) as a parameter representing an environment in which the WCN N (2.1) does some- thing, whereas p1 E 7?. denotes an input pattern from this environment. For example, p0 may parameterize a degree of illumination and p1 represents the shape of an object to be recognized. Notice that the synaptic coefficients Ci,- computed in the proof of Theorem 4 depend on the environment p0, but not on the input p1. We can also ascribe meaning to each term in A(5) 2 0 + eA1+ 0(52), A16 A. The first term (which we postulated to be 0, but it could have been any A0 6 A) means that (2.1) is near a non-hyperbolic equilibrium point, which corresponds to one of the multiple bifurcations considered above. We showed that this was a necessary condition for a WCNN to exhibit any non-linear behavior. The second term (which is 5A1) is of order 5. This explains the notion ”near”. So, the ”near” means to be e-close to the bifurcation value A0 = 0. The fact that coefficient A1 6 A satisfies (2.11) means that the network (2.1) is adapted to the particular environment pg 6 7?. in which it works. 41 2.6.1 Adaptation Condition and Psychology Suppose the adaptation condition (2.11) is violated. Consider, for example, the mul- tiple saddle-node bifurcation (the other multiple bifurcations and singularities may be considered similarly). The rescaling x,- —> 5% 11?,- t —-> e‘it transforms (2.17) to :ic;=a,-+p,-;r?+(’)(\/E), i=1,...,n (2.33) with a,- # 0 and p,- # 0. The behavior of this system is predetermined by the environment p0 and the internal parameter A (a,- depends upon them). The dynamics of (2.33) is hyperbolic and, hence, locally linear. System (2.33) does not have memory and cannot perform the recognition task because it is insensitive to the input pattern p1. It can react only to p0 and its reaction is locally trivial. We see that the adaptation condition (2.11) is important. Without it, recognition is impossible. That is exactly what we expected from our experience. Indeed, when we come into a dark room from bright sunshine, we cannot see anything until we adapt to the new environment. The human brain is an extremely flexible system. It has many subsystems with different time scales that help it to adapt to any new environment. Their functions are not known completely yet. The fact that we did not make any assumptions about the Banach spaces ’R and A gives us freedom in interpretations of our results. For example, we do not have to specify where exactly the adaptation takes place — in the cortex, in the retina, or in the pupil of the eye. It does take place. And there is a mechanism responsible for that. 42 Let us formalize this observation. Recall that we called X E M n and A E A phys— iological and psychological variables, respectively. Thus, (2.1) describes physiology of the brain in contrast to the dynamical system A = H(A,X,po), A e A, X e M”, po 6 R, (2.34) that describes psychology of the brain. The division into physiology and psychology is convenient. In (2.1) we assumed that A and p are constants. Hence, the physiological dynamics is a mere response of the brain to the input p from receptors. The response depends upon the internal state A of the brain as a parameter. For different choices of A the response could be (and frequently is) different, because dynamics of the canonical models depends upon A, for example, through constants b,- and r,-. What do we know about (2.34)? First of all, the characteristic time of (2.34) must be much slower than that of (2.1), so that we may assume A to be constant in (2.1). We showed that in order to perform any interesting task, the system (2.1) must be ' near a non-hyperbolic equilibrium point, i.e. A must be in an e-neighborhood of the . origin. After rescaling A —-> 5A, H ——> 5H, we may assume that A = (9(1) in (2.34), i.e. we identified it with A1 from (2.10). We know how important for (2.1) the adaptation condition (2.11) is. Hence, it is natural to assume that in the (possibly infinite-dimensional) Banach space A the linear manifold ~ Afpo) = {A1 E AI D.\fi(0,0) ' A1 + gi(0907p090) = 0} is a global attracting set. So, A-dynamics governed by (2.34) can be divided on two parts: Changes ap- proaching to A(po), which might be called adaptation dynamics, and dynamics along 43 ~ A(P0) While A is approaching A(po) any recognition is impossible, and the neural network (2.1) can react only to the environment p0. After the adaptation is completed and A is in an e-neighborhood of A(po), the adaptation condition (2.11) is satisfied and the neural network is able to perform a pattern recognition task. The internal parameter A continues to stay e-close to A(po). By moving along A(p0), A cannot destroy the ability of the neural network (2.1) to recognize something, but it can affect how (2.1) does it. The internal parameter A has great impact on the way the brain perceives the world. The WCNN (2.1) together with (2.34) could have features that psychologists might call attention, emotions, feelings, etc. To the best of our knowledge, the WCNN (2.1) together with (2.34) has not been studied yet. Chapter 3 Singularly Perturbed WCNNs 3.1 Basic Definitions Following are some definitions that are used here. Definition 14 A Relaxation Neuron (RN) is a singularly perturbed dynamical sys- tem of the form uX’=F(X,Y,A) (31) Y’ = G(X,Y,A), ' where ’ = d/dr and 0 < u << 1 is a small (dimensionless) parameter reflecting a ratio of time scales; the vector X E Rh denotes fast and Y E Rm (relatively) slow variables; A E A is a parameter. We assume that the functions F : Rk x Rm x A —) Rk and G : IR" X Rm x A —> Rm are as smooth as it is necessary for our computations. Example 1 IfX denotes activity of ”fast” ion channels (e.g. Na+, Ca++, etc) and Y denotes activity of ”slow” ion channels (e.g. K+) in a model neuron, then (3.1) could describe a mechanism of generation of action potentials by a neuron. The typical examples of such dynamical systems are Hodgkin-Huxley equations (Hodgkin and Huxley 1954) and Fitzhugh—Nagumo equations (Fitzhugh 1969) but for u ~ 1. Example 2 If X and Y denote the activities of local populations of excitatory and inhibitory neurons, respectively, then (3.1) could describe dynamics of relaxation 44 45 neuron oscillator. An example of such a neural oscillator is given by Wilson and Cowan (1973). Introducing the ”fast” time t = r/u, we can rewrite (3.1) in the form { X = F(X,Y,A) (3.2) Y = you, Y, A), where dot denotes d/dt. When u is small, we can assume that Y :2: 0 and consider the reduced fast system X=HXKM, am where Y and A are treated as parameters. Definition 15 We say that the singularly perturbed dynamical system (3.1) is at quasi-static bifurcation point (X*,Y*) for A = A* and ,u << 1 if the corresponding reduced fast system (3.3) is at a bifurcation point X* when (Y, A) = (Y*, A*). In this article we study quasi-static saddle-node, pitchfork and Andronov-Hopf bifurcations. Remark 16 The quasi-static bifurcations of (3.1) when X and Y are scalars could be mistaken for a Bogdanov—Takens bifurcations of (3.2) at the equilibrium point (X*,Y*) for (A,p) = (A*,0). Suppose (3.3) is at saddle-node bifurcation. Then Fx 2 0 and the Jacobian matrix of (3.2) for u = 0 is 0F)» 0 0 ’ which for Fy # 0 corresponds to the Jacobian matrix for Bogdanov—Takens bifurca- tion. Nevertheless, there is a difference between them: In the Bogdanov-Takens bi— furcation, perturbations of the equations for X’ and Y’ have the same order, whereas 46 in quasi-static bifurcations the perturbations have essentially different orders of mag- nitude. They differ by factor )1 << 1. We will see later that local dynamics of (3.1) depends more on F than on G. Next we consider networks of such neurons. Let (X,, Y.) E Rk x Rm denote the activity of the i-th RN, i = 1,... ,n. We use the notations X 2 (X1, . . . ,Xn)T E Rh" and Y: (Y1,...,Y,,)T E Rm". Definition 17 A weakly connected network of relaxation neurons is a dynamical system of the form { ”xx-.- amt-A)+ePi(X)Y,/\a#»€) i=1 n (3.4) K, : Gi(Xi9 K» A) + €Qi(X7 Ya ’\i #15) where e is a small parameter, and the functions P,- and Q,- represent connections from the whole network to the i—th relaxation neuron (RN). Note the diagonal structure of (3.4) when 8 = 0 and there are no connections between RN’s. The weakly connected network (3.4) can be considered as an e-perturbation of the uncoupled (e = 0) system { ,UXiI = Fiininv”) i=1 n. (35) Y.’ = G.(X.-, 14.)) We assume that each RN in (3.5) has an equilibrium point (X,-*, Y,*) for some common value A = A* E A (recall that A is a multidimensional parameter space). Hence the system (3.5) has the equilibrium point (XI, . . . ,X:, Y1", . . . , Y;). 3.2 Motivational Examples We study behavior of (3.4) in some neighborhood of the equilibrium point (X *, Y*). In particular, we are interested in how the equilibrium loses its stability. As soon 47 Figure 3.1: Possible intersections of nullclines of the relaxation neuron (3.1). as the phase point (X(t),Y(t)) leaves a 0(1)-neighborhood of the equilibrium, we cannot say anything about subsequent behavior of (3.4). Thus, our analysis is local, not global. Still, this analysis is useful, much in the spirit of thresholds of epidemics, explosion modes in chemical kinetics, extinction of chain branched reactions, phase changes in physics, etc. Example 3 Suppose X,- and Y, in (3.5) are one-dimensional variables and the null- clines F;(X,-, Y,-, A*) = 0 and G;(X,-, Y,, A*) = 0 intersect transversally as depicted in Figure 3.1a. Then, each RN has an asymptotically stable equilibrium (Xf‘, Yf), and hence, the uncoupled system (3.5) as a whole has an asymptotically stable equilib- rium (X*,Y*). The weakly connected system (3.4), which is an e—perturbation of (3.5), also has an asymptotically stable equilibrium which is in some neighborhood of (X*,Y*). Theorem 18, presented in Section 3.3, and the Fundamental Theorem of WCNN Theory (Section 2.3) ensure that (3.4) does not acquire any non-linear features that make its dynamics more interesting than that of (3.5). Example 4 Suppose the nullclines F,(X,-, Y,~, A*) = 0 and G,(X,-, Y), A*) = 0 intersect non-transversally as it is depicted in Figure 3.1b. Obviously, under perturbations, the non-hyperbolic equilibrium (X{, Y-*) may disappear or transform into a pair of ! equilibria. Thus, it is reasonable to expect that the weakly connected system (3.4) 48 Figure 3.2: An excitable system. There are initial conditions for which the system (3.1) generates an action potential, or spike (dotted line). might have some non-linear properties that (3.5) does not have. This case can also be reduced to the one studied in the previous chapter. These examples show that sometimes the existence of two time scales (slow t and fast 7’ = t / ,u) is irrelevant when the network of RN’s can be reduced to a network of non-relaxation neurons (see Section 3.3). Example 5 (Excitable Systems). When the nullclines intersect as in Figure 3.2,the equilibrium point is globally attractive. Nevertheless, there are initial conditions, which are relatively close to the equilibrium, such that the dynamics of the RN lead to large changes in the state variables X and Y before the RN activity eventually returns to the equilibrium. This amplified response (dotted line in Figure 3.2) is called an action potential or spike. Such systems are called excitable (Alexander et.al. 1990). One can think of excitable systems as being systems that are near a threshold or phase transition. This property can be observed when F; 32706”, K”, A”) z 0, (3.6) 49 Y1) G—O Yb 0:0 a b Figure 3.3: Intersections of nullclines which do not correspond to an excitable system. a. A relaxation oscillator with non-zero amplitude. b. A relaxation oscillator with zero amplitude. i.e. when (XI, Y-*) is near a point for which 6F,- _ 0. 0X,- If X,- is a vector, then condition (3.6) should be replaced by the condition that the Jacobian matrix Dx.Fi(X.-*, Y1", ,\*) has eigenvalues with zero (or close to zero) real parts. Such matrices are called non- hyperbolic. If the closeness is compatible with the strength of connections 5, then weakly connected system (3.4) may have interesting non-linear properties. The E—pertur- bations from the other neurons can force a RN to generate the action potential, or to remain silent. The excitable system is one of the many relaxation systems under consideration. For example, the RN whose nullclines intersect as depicted in Figure 3.3 are not 50 excitable. Nevertheless, our theory is applicable to them too. In both cases condi- tion (3.6) is satisfied and a network of such neurons can have interesting nonlinear properties. We start studying networks of such RN’s in Section 3.4. Below we show that if condition (3.6) is violated for all RN’s, then the network of such RN ’8 behaves similar to a network of non-relaxation neurons, which we studied in previous chapter. 3.3 Reduction to Regular Perturbation Problem One can consider (3.4) as a singular perturbation of the unperturbed (u = 0) weakly connected system { 0 : Fi(Xia)/H’\) + €Pi(X9 Y,A,/.t,€) w = G.-(X.-,Y.-, A) + saucy, m5) .=1,...,n. (3'7) The reduced system (3.7) is a quasi-static approximation to (3.4). Then the questions arise: When do (3.4) and (3.7) have similar dynamical properties; Can (3.7) be further simplified? Partial answers are contained in the following Theorem 18 Suppose that (3.4) has an equilibrium point (X*,Y*,A*) for e = 0. Suppose that each Jacobian matrix DX.F,-(Xf,Yi*,A*) (3.8) has all eigenvalues with negative real parts. Then for (X, Y,A,)u,e) sufficiently close to (X*,Y*,A*,0,0) the singularly perturbed weakly connected system (3.4) is approx- imated by the regularly perturbed weakly connected system Y}! = g,(Y,-, A,u) + eq,(Y, A,p,e), i = 1,... ,n, (3.9) for some functions g,- and q). In particular, if the equilibrium point of (3.9) for small 5, p and A near A* is asymptotically {un)stable, then so is the equilibrium point of (3.4)- 51 Proof of the theorem is based on Implicit Function arguments and singular per- turbation techniques. We do not present the proof here because Theorem 18 is a corollary of Theorem 19, which we prove in the next section. There we also explain the notion approximated. In the rest of this chapter we study (3.4) for the case when its reduction to (3.9) is impossible, i.e. when some (or all) of the Jacobian matrices (3.8) have eigenvalues with zero real parts. This occurs when the corresponding RN’s are near thresholds and are sensitive to external perturbations. 3.4 Center Manifold Reduction Consider (3.4) near the equilibrium point (X*,Y*,A*) for e = 0. Without loss of generality we may assume that (X*, Y*, A*) = (0, 0, 0). Theorem 19 Suppose that each of the first n1 Jacobian matrices L, = DxtF,(0',0,0) has some eigenvalues with zero real parts and all other eigenvalues with negative real parts. Let Ef (i = 1,. . .,n1) be the eigensubspace spanned by the (generalized) eigenvectors corresponding to the eigenvalues with zero real parts, and the other n—nl Jacobian matrices have all eigenvalues with negative real parts. Then (3.4) is locally approximated by the lower dimensional dynamical system of the form ux,’ = f,(:r,, Y,,A,,u) + 5p,(:r, Y,A,u,e), i = 1,. . .,n1, Y,’ = g,(:r,,Y,,A,,u) + €q,(.r,Y,A,p,€), i = 1,. . .,n1, (3.10) Y}, = gj(Yja’\aH) + qucvayaAsuvgla j: "1+1a- ”in, where .r, E Ef and J, = 1),, f,(0,0,0,0) = L ilEf’ 2:1,...,Tl1. 52 In particular, J, have all eigenvalues with zero real parts. More precisely, there is a. function Z : EC x Rm" x A x R x IR —> Rk" >< Rm" such that all local solutions (X(t),Y(t)) of {3.4) tend exponentially to Zita), Y(t), A, p, e), where (a:(t),Y(t)) is some solution of {3.10). Proof. If we rewrite (3.4) as =1,...,n. {Xi=F( X{,K,A)+€Pg(X,Y) Yi : ”(G (X19 YHA) + €Q,(X,Y)) where - : d/dt and t : r/u is the ”fast” time, then the result follows directly from Theorem 2.3. CI Corollary 20 Theorem 18 follows from Theorem 19 when n1 2 0. In this case dim EC 2 0 and (3.10) has the same form as (3.9.) Remark 21 Without loss of generality we assume in the following that all Jacobian matrices L,, i = 1, . . . , n1, have all eigenvalues with zero real parts. This means that we consider the weakly connected system (3.4) to be restricted to its center manifold. 3.5 Canonical Models In this section we study the local dynamics of the singularly perturbed weakly con- nected system (3.4) { ,LtXI“F(X,,1/,, ,H)+5Pi(XaY,)\,/435) X A . Y5: G( ..K./\,u)+€Qs(X.Y./\,u.€) ”I’M" (3“) for the case when each Jacobian matrix Li : DX.‘Fi(07 0,030) 53 has one simple zero eigenvalue. By Theorem 19 and Remark 21 we may assume that each X, 6 IR, i.e. it is a one dimensional variable and a Li:—Fi a 3 7 = a . 0X,- (0 0 0 0) 0 (312) i.e. we restricted (3.4) to its center manifold. Fix Y = 0 and e = 0 and consider the reduced fast system ”Xi, : Fi(‘¥i70a)‘$iu) (3'13) at the equilibrium point (X, A) = (0,0). When (3.12) holds and a bifurcation occurs in (3.13), then we say that the RN { “Xi, = F(XiaYia)‘)H) K, : G(X,‘,Y;',)\,/1), (314) is at quasi-static bifurcation. We study here the case when all equations in (3.14) undergo bifurcations of the same type simultaneously. Such bifurcations are called multiple. We derive canoni- cal models for the simplest and most interesting cases; namely multiple quasi-static saddle-node, pitchfork and Andronov-Hopf (see Section 3.6) bifurcations. Our analysis of the weakly connected system (3.4) is local in the sense that we study its dynamics in an e—neighborhood of the equilibrium point (X, Y,A,,u,e) = (0,0,0,0, 0). In particular, we study the dependence of solutions of (3.4) on parame- ters A,p,e by supposing that /\(5) = 0 + 5A1 + C(52), A16 A u(e) = 0 + 5m + 52m + C(53), [11,112 E R. (3'15) The analysis below does not depend crucially on A1. In contrast to this, the values of #1 and [12 are important. We derive canonical models when #1 = 0. The case in 75 0 is discussed in Section 3.5.3. 54 We also assume that the following transversality condition Dy,Fi(0,0,0,O)-DX,G,(0,0,O,0) #0, i: 1,...,n, (3.16) is satisfied for all i. If this condition is satisfied, we say that the relationship between X, and Y, is non—degenerate. Before proceeding to derivations of the canonical models we prove the following result: Lemma 22 There is a mapping Y : R —> Rm” such that the initial portion of the Taylor expansion of (3.4,) at (X, Y) = (0, Y(e)) has the form )2, = A,Y, + (ix-2 + ax? + e 2;, C,,X,- + 0(XY, sY, Y2, X3, 5X2, 52X) K = [1(5R,‘ + BiXi) + #O(Y,X2,€4X,€2), (3.17) fori : 1,. . . ,n. In particular, the equations for X, do not have terms 5,82, . . .. Proof. From the transversality condition (3.16) it follows that A, : Dy,F,(0,0,0,0) 75 0, i = 1,. . . ,n. (3.18) Applying the Implicit Function Theorem to the system of algebraic equations 0 = F,(0, 1'}, A(e),u(e)) + €P,(0, Y, A(e),u(s),e), i = 1,. . . ,n gives a unique smooth function We) = Y(A(e).u(e).e) such that an F,(0,Y,(e),A(e),p(e))+eP,(0,Y(e),A(e),u(e),e)EO, i=1,...,n (3.19) for all sufficiently small 5. Let (X, Y) 6 IR." x Rm" be local coordinates at (0,Y(e)). Equation (3.19) guarantees that the Taylor expansions of equations for X: in (3.4) 55 at (0,Y(e)) do not have terms of order 5,52, . .. not multiplied by state variables. Then, the initial portion of the Taylor expansion of (3.4) is defined in (3.17), where A, : Rm ——> IR are defined in (3.18), and B, = Dx,G,(0,0,0,0) E Rm, 6 CUZEX—j P,(0,0,0, 0,0) 6 R (3.20) for i 75 j, and C“ = 5ng (DYiFi(0209030)%92 + DAF£(0,0,0,0)A1 + 5%Fi(03 09 0,0))111 + Pi(0,0,0,0,0)) E R for i = j. Also d, = % 5%F,(0,0,0,0) 6 IR, 6, = %5%E(0.0,0,0) e n and 12,-: 1),, G,(0,0,0, 01%? + D,\G,-(0,0,0,0)/\1 + gist-(mammal + Q,(0,0,0,0,0) C] We study here the case #(6) = 52/12 + 0(53) i.e. u, = 0. The case u, at 0 presents no problem and is discussed in Section 3.5.3. 3.5.1 Multiple Quasi-Static Saddle-Node Bifurcations Each equation in the reduced fast system (3.13) is at saddle-node bifurcation if 6 —E373 :3 8X,- (0000) 0 56 DAFi(0909030) # 0 (3-21) and d-—1—8—2—F-(0000)¢0 (322) ’_ 2 8X3 ‘ ’ ’ ’ ' for all i. The theorem presented below is valid even when (3.21) is violated. Theorem 23 Suppose that the singularly perturbed weakly connected system (3.4} is at a multiple quasi-static saddle-node bifurcation point, that the transversality condi- tion (3.16) is satisfied, and that ”(5) = 52M + (W53), #2 7i 0- Then the change of variables 11?, = E-ldi2f,’ E R y. ___ —a"“d.-A.-Y.- en i=1,...,n (3.23) reduces (3.4) locally to _ .. .2 n '= { 1‘:- — *g: + 171' + ijl C1113] + 0(7) 2:1,” .,n, (324) y.'- = (“(3% — Ti) + 0(5) where x,,y, E R are scalar variables and a, 3i 0. We refer to {3.24) as being the canonical model for the multiple quasi-static saddle-node bifurcation in a weakly con- nected network of RN’s. Proof. If we use (3.23) and rescale the time 7' 2 at, then (3.17) transforms to (3.24), where at = _,U2AiBi 7‘.“ = diAiRi(AiBi)_1 (3.25) CU = (I,C,jd;1 for all i and j. Note that A,B, 75 0 due to transversality condition (3.16). Hence a, 74 0 and r, is finite. D 57 Remark 24 Note that while Y, 6 IR'”, the canonical variable y, 6 IR is a scalar. Therefore, the change of variables (3.23) cannot be invertible if m > 1 and the canonical model describes the dynamics of (3.17) projected into a linear subspace. Thus, we might lose some dynamics taking place along the kernel of the projection. Remark 25 Using the translations Si‘,=.’II,—7‘, ..._ . 2 n ... i=1,...,n y: —9: ‘7‘; “23:16:37? we can rewrite (3.24) in the form (where we erase ~) { x:- = —y.- + (2r, + calm + 1’? + 22¢; Cifl’i + 0(5) i=1,...,n, (3.26) yf = a,x, + 0(5) which is sometimes more convenient. For, example, it is easy to see that (3.26) has a unique equilibrium (x, y) = (0,0) up to terms of order (9(5), which is asymptotically stable when r, ——> —00. Remark 26 With the obvious change of variables, the canonical model (3.24) can be written as 71 I 31,, - c,,y,'~ — (3102 + aiyi = 2003/;- + 0(5) i = 1,... ,n- (3-27) 1?“ If all a, > 0, then the left—hand side of each equation in (3.27) is the classical model for studying singular Andronov-Hopf bifurcations (Baer and Erneux 1986,1992; Eckhaus 1983). Remark 27 We see that in the canonical model (3.24) only connections between ”fast” variables X, are significant. The other three types of connections have order s or 52 and hence are negligible. Indeed, (3.24) can be rewritten in the form { i: —y, + 3?? + 2;; Cij-Tj + 5 Z}; dijyj + 0(5) y:- = a,(x, ‘- T‘i) + 8 Zj=18v$j + 52 zf=1fijyj+ 0(5) i=1,...,n, for some d,,-, 6,], f,,- 6 IR. We discuss possible applications of this fact in Section 3.7.1. 58 3.5.2 Multiple Quasi-Static Pitchfork Bifurcations Suppose that the conditions 6X, 1 l , , (9X? ' l i , and e: 6 6X? : , l 3 ( i ) are satisfied for all i. Then the uncoupled system (3.13), which describes fast dynamics of (3.4), is near multiple cusp singularity. Using Lemma 22 one can see that the reduced fast weakly connected system ”X3 = F,(X,,Y,,A,p) + sP,(X, Y, A,p,e) (3.29) At has a family of equilibrium points (X, Y, A,p,e) = (0, Y(e), A(e),u(e),e) for all suffi- ciently small 5. If we additionally demand that 82 0A6X 1?,(0,0,0,0) at 0 (3.30) for all i, then (3.29) at (X,Y) = (0,Y(e)) has a multiple pitchfork bifurcation. The result presented below is valid even when (3.30) is violated. Theorem 28 Suppose that singularly perturbed weakly connected system (3.4) is at a multiple quasi—static pitchfork bifurcation point, transversality condition (3.16) is satisfied, and 11(5) = 62/12 + 0(53), #2 ¢ 0'. then the change of variables .17, : ‘/(6,‘|€_1X,‘ ER i—l (3 31 9i = ‘VleiIE-iiAil/i‘ 61R — ,... i ) reduces (3.4) to $2: ‘yi+0i$?+z”= c,-x-+O(\/E) . {yizai$i+0(\/E) 11 J J 2:1,...,n, (3'32) 59 where a, ¢ 0 and a, =signe, = 21:1 (6, was defined in (3.28)). One can think of {3.32) as being the canonical model for the multiple quasi-static pitchfork bifurcation in weakly connected networks of RN’s when in addition {3.30) is satisfied. Proof. If we use (3.31) and rescale time T = at, then (3.17) transforms to (3.32), where 02‘ = —H2A£Bi Co = Vlez‘ICijx/Iejl" for all i and j. Note that A,B, ¢ 0 due to (3.16). Hence a, 7b 0. D Remark 29 With an obvious change of variables the canonical model (3.32) can be rewritten as n y,'-I — c,,y,'- - o,(y,'.)3 + a,y, = Z Cay;- + (Oh/E) i = 1,. . . ,n. (3.33) i?“ If all a, > 0 and o, = —1, then the left-hand side of each equation in (3.33) is a van der Pol’s oscillator in Lienard representation. Notice that the oscillators are connected through the derivatives y;- (”fast” vari- ables), not y, (”slow” variables) as is usually assumed (Grasman 1987, Section 3.1). Remark 30 An unfolding of the cusp singularity should include the terms x? in the equations for .131. This is equivalent to introducing constant terms in the equations for y,: i’ : ‘92‘ ‘1' ”£3313 ‘1' 22:1 Cijxi + Olfi) _ 1 n 313: a,(x, - Ti) + (Oh/E) _ V ’ When 0, = —1 and a, > 0, this is a network of connected Bonhoeffer—Van der Pol oscillators. Remark 31 If 41(5) 2 0 + (9(5"), k > 2, 60 then (3.4) can also be reduced to the canonical models (3.24) and (3.32), respectively. But in these cases a, = (flak—2), i = 1,...,n, i.e. each oscillator in the canonical models is a relaxation oscillator. 3.5.3 Discussion of the case p = 0(8) Next, we discuss the case “(5) = 0 + eu1+ 0(52). #1 > 0 with the additional assumption that in the weakly connected system (3.4) each vari— able Y, is one—dimensional. According to Lemma 22, system (3.4) can be expanded in the Taylor series shown in (3.17). Rescaling x, = e 2X, y, = 8-1“}, i=1,...,n transforms (3.17) to 2,, : A.-y.- + dim? + data-x? + fang/1+ 23;, Cijl'j) + 0(5) 1. __ 1 n y,’ = #lBfli + x/EWIRi + 9292' + hill) + 0(5), — ,..., (3.34) which is a perturbation of the uncoupled system 5173', = Aiyi + (lift? . __ { yi, : #18103“ i — 1,. . . ,n (3.35) near the equilibrium point (x, y) = (0, 0). Theorem 32 If A,B, > 0 (3.36) 61 for all i, then the weakly connected system (3.34), the uncoupled system (3.35) and the linear uncoupled system i (C, _ 0 Ai)(xi) Z—l. n dr yi — H13.“ 0 y.- ’ — "H, are topologically conjugate. Proof. Inequality (3.36) and p1 > 0 imply that (x,,y,) = (0,0) is a hyperbolic saddle point for this system. The result follows from Implicit Function and Hartman- Grobman Theorems. Cl Remark 33 When ,n,/1,8, > 0, the weakly connected system (3.4) does not have interesting local nonlinear neuro—computational properties since it is essentially linear and uncoupled in that case. If all A,B, < 0, then each Jacobian matrix 0 A, [1181' 0 has a pair of pure imaginary eigenvalues. In this case the WCNN (3.34) is at a multiple Andronov-Hopf bifurcation point. We studied this case in Section 2.5. Finally, we note that it is not clear which facts derived in this subsection remain valid when we drop the assumption that each Y, is a scalar. 3.6 Multiple Quasi-Static Andronov-Hopf Bifur- cations In this section we study the case when in the singularly perturbed weakly connected system (3.4) each Jacobian matrix Li : DX.Fi(0703030) : ( all an) 0:3 W4 62 has a pair of purely imaginary non-zero eigenvalues iiQ,. This corresponds to a multiple quasi-static non-singular Andronov-Hopf bifurcation of (3.4). . Theorem 34 Suppose that {3.4) is at multiple quasi-static Andronov-Hopf bifurca- tion point, then the invertible change of variables 1 l em‘TZ, Xi = fi( diffiflt (lg—2.9: ) ( e—ZOQH’E. ) + 0(5) 2:1,. .. ,n, at? “:2 Yi=5vi reduces {3.4) to Z: = (“2' + Aivi)zi + bizilzi|2 + n, . Ci'Z' + (Oh/E) 2 20.32. J ’ i=1,...,,n (3.37) f: (MR; + Silzil + Ti'Ui + O(\/E)) where z, E C, v, E IRm and a,,b,,c,, E C; A, : IR’” —> C; R,, S, 6 R“; T, : IRm —> IRm and d, = p/e. We do not present a proof here. Derivation of (3.37) coincides (after obvious modifications) with the derivation of the canonical model for multiple Andronov- Hopf bifurcations in weakly connected networks of non-relaxation neurons and can be found in Section 2.5. Notice again that in the canonical model (3.37) (as well as in (2.25)) only those RN’s interact that have equal natural frequencies 0,. If it, 75 Q, for some i and j, then the i-th and j-th RN’s do not ”feel” each other even when c,,- 7g 0 and c,, 7t 0. The i-th RN can turn on and off its connections with other RN’s simply by adjusting its natural frequency 0,. This feature might be related to such a phenomenon as attention. 63 3.7 Conclusion We have analyzed local dynamics of the singularly perturbed weakly connected system (3.4) at an equilibrium point and showed that in many cases it is governed locally by well-studied systems like regularly perturbed weakly connected systems (3.9) (Theo- rem 18). In other interesting cases the dynamics of (3.4) are governed by the canonical models (3.24), (3.32) or (3.37). To the best of our knowledge these models have not been analyzed yet. We think they have interesting computational properties with possible applications to neurocomputers. We present some basic analysis of (3.24) and (3.32) in Chapter 8. 3.7.1 Synaptic Organizations of the Brain We saw (Remark 27) that in the canonical models only connections between ”fast” variables X, are significant and the other three types of connections (”fast” —> ”slow”, ”slow”——>”fast”, ”slow”—+”slow”) are negligible. Even this simple fact has some im- portant biological implications. Suppose (3.1) describes a mechanism of generation of action potentials by a neuron (as in Example 1). Then the significance of ”fast”—>”fast” connections means that the synaptic transmission between neurons is triggered by ”fast” ion channels (in fact, by Na+ and Ca++; The former are responsible for depolarization, the latter open synaptic vesicles). Since the synaptic transmission mechanism is well studied now (Shepherd 1983), this observation does not carry much new information. Suppose (3.1) describes the activity of the relaxation neural oscillator as in Exam— ple 2. Then ”fast”—>”fast” connections are synaptic connections between excitatory neurons. Thus, one can conclude that information is transmitted from one part of the brain to another one through excitatory neurons, while the inhibitory neurons 64 serve only local purposes. And indeed, copious neurophisiological data (Rakic 1976; Shepherd 1976) suggest that excitatory neurons usually have long axons capable of forming distant synaptic contacts, while inhibitory neurons are local-circuit neurons having short axons (or without axons at all). They provide reciprocal inhibition. This division into relay- and inter-neurons has puzzled biologists for decades (see Ra- kic 1976). Our analysis shows that even if the inhibitory neurons had long axons, their impact onto other parts of the brain would be negligible. Hence, the long-axon inhibitory neurons are functionally insignificant. The analysis above attempts to describe why a brain might have the anatomical structure it does. But we are still far away from satisfactory explanations of this problem. We continue our study of possible synaptic organizations of the brain in Chapter 10. Chapter 4 Weakly Connected Maps In previous chapters we studied dynamics of weakly connected networks governed by a system of ordinary differential equations (ODE). It is also feasible to consider weakly connected networks of difference equations, or mappings, of the form X, H F,(X,',)\) +5G,(.X,/\,p,€), 2: l,..,n, 5 <<1, (4.1) where variables X, E Rm, parameters A E A, p E R and functions F, and C, have the same meaning as in previous chapters. Difference equations of the form (4.1) arise when one studies Poincare maps of flows. A weakly connected mapping (4.1) can also arise as a time-T map of the flows. Indeed, consider a periodically forced weakly connected network of the form x,- = F,(X,,A, P,(t)) + eC,(X,A,p,Q(t),e), where P,(t) and Q(t) are T-periodic functions. Such systems can describe thalamo— cortical interactions (see Figure 4.1). In this case P,(t) and Q(t) denote inputs from the thalamus to the i-th cortical column X,. Knowing X(0) one can find X(T), which is some function of X(O), A, p and 5. It is easy to check that the function has a weakly connected form and can be written as (4.1). 65 66 --7 \ / Cortex columns sari Jlf‘ Weakly Periodic connected Oscillator signal network Figure 4.1: Thalamo—cortical interactions Recall that our strategy is to compare dynamic behavior of the uncoupled (e = 0) system X, H F,(X,,/\), i=1,..,n (4.2) and coupled (e 74 0) system (4.1). Obviously, the uncoupled system (4.2) is not interesting as a model of the brain. We are looking for such regimes and parameter values which endow the coupled system (4.1) with “interesting” neuro-computational properties. As in previous chapters, we study dynamics of (4.2) near a fixed point X* — (Xf, ..,)(::)T for some X” E A. Thus, we have x = F,(X;",A*) ‘8 for all i. Without loss of generality we may assume X* = O for x\* - 0. Notice that the time-T map having a fixed point corresponds to a continuous time dynamical system having a limit cycle. Analysis of weakly connected mappings is parallel to analysis of weakly connected systems of ODE, which we performed in Chapter 2. As one can expect, local dynam- ics near a fixed point is not interesting when the point is hyperbolic (Section 4.1). 67 Therefore, the only points that deserve our attention are non-hyperbolic. In this chap- ter we study dynamics of (4.1) in some neighborhood of a non-hyperbolic fixed point corresponding to saddle-node and flip bifurcations (Sections 4.2.1 and 4.2.2). First we derive canonical models and then we reveal the relationship between them and the canonical models for continuous time weakly connected neural networks (Section 4.3). 4.1 Hyperbolic Fixed Points A fixed point X: = 0 is said to be hyperbolic if the Jacobian matrix L, = DxtF,(0,0) does not have eigenvalues of unit modulus. Notice that the Jacobian matrix for the uncoupled system (4.2) near the fixed point X* = (Xf, .., X3“)T has the form and the fixed point X* is hyperbolic if and only if each X: is hyperbolic. Theorem 35 If the dynamics of each neuron is near a hyperbolic fixed point, then the weakly connected network (4.1) of such neurons, the uncoupled network (4.2) and the linear mapping X»—>LX have topologically equivalent local orbit structures. In this case the local behavior of the weakly connected system (4.1) is essentially linear and uncoupled. 68 Proof of the theorem uses the Implicit Function and Hartman-Grobman Theo- rems and coincides (with the obvious modifications) with the proof of the analogous theorem for ODE (Theorem 2 in Chapter 2). 4.2 Non-hyperbolic Fixed Points It follows from the previous section that the only fixed points requiring further dis- cussion are non-hyperbolic ones. An important case when such fixed points occur corresponds to bifurcations in dynamics of (4.2). In this section we study the sim- plest bifurcations of those: saddle-node and flip types. For each, the Jacobian matrix L, has only one eigenvalue on the unit circle: +1 for saddle—node and —1 for the flip bifurcation. We assume that the other eigenvalues are inside the unit circle. The Center Manifold Theorem for the maps (analogous to the one for flows used in Section 2.3) guarantees that there is a locally attractive invariant manifold in Rm” on which the dynamics of (4.1) is governed by the lower-dimensional system of the form .17, H f,(.l',', A) + eg,(;r, A,p,tf), (4.3) where each 1?, E R is a scalar and f,(0.0) = 0, gran) = i1. As before, we assume that /\ = Me) = 0 + 5A1 + C(52) and p = p(€) = P0 + 5101+ 0(52) for some A1 6 A and p0,p1 E R. flx) . n I ' I Z I - I . . I . I ; I - I 2 I - I a b c Figure 4.2: Bifurcations of a mapping. a. Saddle-node bifurcation for c,p, < 0. b. Saddle-node bifurcation for c,p, > 0. c. Flip bifurcation. 4.2.1 Multiple Saddle-Node Bifurcations Consider the weakly connected system (4.3) near a multiple saddle-node bifurcation point 32* = 0 for X“ = 0. The initial portion of the Taylor series of the right—hand side of (4.3) at the origin for small 5 has the form :r, + pa? + €(c, + Z c,j:L'J-) + h.o.t., i=1 where 1 82 Pi = ga—ngiloaol 7'5 0: 8 Co“ = 5;:9r(0,0,po,0) and Ci = DAf-iloaolAl +gi(0309Po,0)- Recall that we called c, = 0 the adaptation condition for the i-th neuron dynamics. Suppose c, 54 O for all i. Then the rescaling .7: : flit transforms the weakly connected mapping to 532‘ H 57.‘ + \/c:(Ci + Pt???) + 0(5) 70 which is essentially uncoupled (although non—linear). Indeed, the i-th neuron’s dy- namics is determined by the constants c, and p,. Depending upon the sign of their product the dynamics can have two qualitatively different local phase portraits: o If c,p.- < 0, then there is a pair of fixed points. One of them is stable and the other one is not (see Figure 4.2a). o If c,p, > 0, then there are no fixed points. For any initial conditions the dy- namics eventually leaves some neighborhood of the origin (see Figure 4.2b). Local dynamics of the i-th neuron is not affected by the dynamics of the other neurons, provided the other neuron activities are bounded. We see that the neuron activity depends only on the environment p0 and internal state A1 (since c, depends upon them) and does not depend on the other neuron activities. If the adaptation condition (c, = 0) is satisfied for all i, then after the rescaling :1: = sit/p, the weakly connected system (4.3) transforms to 517, H if, + 5(7‘1‘4" j? + 2: 5,157,) + 0(52), (4.4) i=1 for 6,,- = c,jpj/p, and some constants r, 6 IR. We call (4.4) the canonical model for a multiple saddle—node bifurcation in weakly connected maps. 4.2.2 Multiple Flip Bifurcations A weakly connected mapping (4.3) is near flip a bifurcation point 513* = 0 for X" = 0 if f.(0,0) = 0, “3%“an = —1 (see Figure 4.2c). The flip bifurcation does not have an analogue for one-dimensional flows. Thus, it is natural to expect some new qualitative features. The most promi- nent feature of the multiple flip bifurcation is that the adaptation condition is not 71 necessary for the local dynamics of the weakly connected mappings to exhibit “inter- esting” properties. Theorem 36 If the weakly connected mapping (4.3) is near a multiple flip bifurcation point, then there is an invertible change of variables which transforms (4.3) to :r, r——> :10,- + \/5(r,:1:, :l: 1::-3 + Z Cami) + 0(5). (4.5) 1:1 We call (4.5) the canonical model for multiple flip bifurcations in weakly connected mappings. Proof. Let us denote the k-th iteration of the mapping (4.3) by :1)”. Thus, we have xi“ = flail-cab) + €9i($k,>\,p.€)- The initial portion of the Taylor series of the right-hand side at the origin is given by —rf +p.-(.r ”,)+q.-(If )+(EC:+ZC.-,rf) )..+hot. j=l Let us find how 33”” depends on 13k .For this we have to evaluate the composition at?” = fi(fi(cvf. A) + 694$". MM), A) + egi(f(:vk. A) + 89(31’“. MM). Mm) f,—25(c p517? + Z c,j:r. 1H) 2(q,- + p,- 2:r:)( f)3 + h.o.t. j=l Now let :13, = \/5:'r':,-/\/2|q, + p§|, 1*. = —2c.p, and c,,- = —%é..-\/Iqj + pfl/Iq. + pEI. then we obtain 56””: i:f‘+\/5(r,:1::c +a,(d: D3+Zc,j:i:f) +0(5), j=1 where 0,- : —sign(q, + pf) = :l:1. Finally notice that this result does not depend on the adaptation condition c, = O. U 72 4.3 Connection With ODE There is an intimate relationship between the canonical models for weakly connected maps and ODE. The former have a form :13:C+1 = 1.5+ rh,(:1:k)+ 0(r2), 7' << 1, k for some functions h,. If one considers the iteration a: as a value of a smooth function at the moment kr, then he can rewrite the equation above as :r,((k +1)r) = :r,(kr) + rh,(:1:(kr)) + 0(r2) or 1:,(t + 7') — .r,(t) = (tiff-(ll) + 0(T). where t 2 hr. The equation above is Euler’s discretization of the ODE dilf, W = h,(l’). In this sense, we see that the canonical model for multiple saddle-node bifurcation 1‘, |—> (I), + 5 (7‘, + 1?? + Z Cami) + 0(62) 1:1 is the discretization of that for ODE d512, n gt— : 7‘, + 1'? + ZCijl'j '1' 0(5)- i=1 Similarly, the canonical model for a multiple flip bifurcation :r, +—> 1:,- + \/E (13m, 4: 1'? + 2 Carl) + (9(5) i=1 is the discretization of da‘, n E : 7’,.’L‘, :l: £13,3‘l' Z Cijl'j + O(\/E)1 i=1 73 which is the canonical model for multiple pitchfork bifurcations. This relationship between canonical models allows us to concentrate our efforts on studying the canonical models for multiple saddle-node and pitchfork bifurcations in ODE, which we perform in subsequent chapters. Part II Analysis of Canonical Models 74 Chapter 5 Multiple Saddle-Node Bifurcation In this chapter we continue to study the weakly connected neural network 23;: f.(:c.-. x\) + 59416, MW) (5.1) near a multiple saddle—node bifurcation. In particular, we explore an important case where such a bifurcation occurs. 5.1 Saddle-Node on a Limit Cycle Consider a dynamical system :1: = fan), (5.2) where :r E [—l,1]/({—l} = {1}) has a meaning of phase variable and the function f is periodic in :2. Suppose (5.2) is near a saddle-node bifurcation point :1: = 0 for /\ = 0, i.e. f(0,0) = 0 and a—if(0,0) = 0 but 132 P = igngloaol 3'5 0- 75 76 c<0 c=0 c>0 Figure 5.1: Phase portraits for various c Also suppose that f(:I:, 0) > 0 for all at 79 0. This implies p > 0. Consider the leading terms of the Taylor series of f for small :1: and A = 5A1 + 0(52) f(:r, A) 2 5c + pr” + h.o.t., where c = D,\f(0, 0)).1. (5.3) Posible phase portraits of such system for various c are depicted in Figure 5.1. The saddle-node bifurcation occurs when c = 0. We see that the local behavior near the non-hyperbolic equilibrium :1: = 0 has global ramifications: For c < 0 there are two equilibria - a stable node (filled circle) and a saddle (open circle). For any initial condition not at the saddle the dynamics approaches the node. When we consider saddle-node bifurcations, the saddle and the node are close to each other. This means that there could be some perturbations which move :r(t) to the right hand side, beyond the saddle. In this case the activity makes one rotation (an excursion, an action potential or spike) and eventually approaches the node. Such a dynamical system is the typical (and possibly the simplest) example of an excitatory system. It resembles the excitatory properties of real neurons. For c = 0 the saddle and node coalesce, and for c > 0 there is no equilibrium, in which case the dynamics of (5.2) is periodic. Such behavior can be observed in general dynamical systems when there is an asymptotically stable limit cycle with a 77 incl? / K Figure 5.3: An intersection of nullclines in a relaxation system which exhibits saddle- node bifurcation on a limit cycle non-hyperbolic equilibrium on it. In two-dimensional case there is also an equilibrium inside the limit cycle (see Figure 5.2). Saddle-node bifurcation on a limit cycle can also arise in relaxation systems with the nullclines intersected as in Figure 5.3 Let us return to the dynamical system (5.2). As we see, local bifurcations of a non- hyperbolic equilibrium :1: = 0 can have a global effect. In order to find out whether (5.2) has periodic activity or not it is suffices to determine the sign of c defined in (5.3). If c < 0, then there are two equilibria 5|c| :l: ———+O(5) P and the dynamics converges. If e > 0, then there are no equilibria (local or global) and dynamics is periodic. The remarkable feature of the saddle-node bifurcation on a limit cycle is that even though the dynamics becomes periodic, :z:(t) spends most time in a neighborhood of the origin; that is, where the non-hyperbolic equilibrium used to be. We prove this fact in Lemma 37 below and use it throughout this chapter. In 78 particular, we can determine how various neurons interact simply by analyzing local behavior near a saddle-node bifurcation. 5.2 The VCON There is an invertible change of variables a: = A(qb,5) (Ermentrout and Kopell 1986) which transforms the dynamical system (5.2) to a simpler system ¢' = (1 -COS¢)P+(1+COS¢)C+O(\/E), . (5-4) which is an example of a Voltage Controlled Oscillator Neuron (VCON) (HOppen- steadt 1983). In the VCON, ’ = d/dr where r = \/5t is a “slow” time variable. Notice that VCON is at saddle-node bifurcation when c = 0. For c < 0 it has two equilibria and for c > 0 it has none. The phase portrait of a VCON coincides with the one depicted in Figure 5.1. One can think of VCON as the canonical model for the saddle-node bifurcation on a limit cycle. The Ermentrout-Kopell’s change of variables a: = A( {75, then A(d>,5) is extended to a one-to—one smooth map satisfying some technical conditions. VCON models such as (5.4) have attracted attention in part due to the fact that their dynamics is described in terms of phase variables. Thus, the hard mathematical problem of converting physical variables to phase variables is eliminated by choos- ing the phase—like variables in the first place. VCON models are also appealing to electrical engineers since they can be constructed as electrical circuits. 79 There are many phenomena which can be studied using VCON models (Hoppen- steadt 1986). The most intriguing of them is the problem of synchronization and attention. We study this problem in Section 2.5. In this chapter we use a VCON model to illustrate various dynamic behaviors. 5.3 Preliminary Analysis In this section we are interested in behavior of (5.2) for c > 0. As we have already said, its dynamics is periodic, but what is the period? Since the period of oscillations in VCON (5.4) for c 75 0 in terms of the “slow” time is 0(1), the period of oscillations in (5.2) in terms of the normal time t = r/\/5 is (Oh/Lg)- Actually it is 321%? but we do not need this expression. Next, how much time does :r.(t.) spend away from some small neighborhood of the origin, say away from [— {/5, +\“/5]? To determine this it suffices to consider the Ermentrout-Kopell transformation :1: = \/5 tan 9. 2 Notice that its inverse is the transformation :1: (b = 2 arctan — \/E’ maps [—{/5, +\“/5] one-to-one onto [—7r + 2 4 5, 7r — 23%. From (5.4) it follows that when a5 3 7r or —7r. Thus, it takes 2% C + 0%) units of slow time T for d to cover the distance from 7r — 24/5 to 71' + 24/5. In the terms of normal timet = r/\/5 it takes 33/? Now recall that d) 6 (7r — 2{/5, 7r + 235) corresponds to [1:] > {/5. Thus, we proved the following result: 80 Lemma 37 During one oscillation {action potential) :1:(t) spends in the {/5-neighborhood of the origin. Since 1/6/5 is less than 1/\/5 one can conclude that the activity :1:(t) spends most of the time in the neighborhood of the origin. 5.4 Case c 7E 0 Now consider the weakly connected neural network (5.1). Suppose that in the uncou- pled (5 = 0) system 313,: f,($,,)\), Z: 1,..,n (5.5) each equation describes dynamics having a saddle-node bifurcation on a limit cycle. Consider the parameter c, defined by Ci : DAfi(030)Al +91“): 03 Panl- Theorem 38 o If c,- > 0, then the i-th neuron oscillates with the period O(—\}—E). The activities of the other neurons do not change the qualitative behavior. 0 If c,- < 0, then the i-th neuron is silent. More precisely, its activity :1:,(t) stays in 5-neighborhood of the origin and the activities of the other neurons do not aflect it {see Figure 5.4). 81 C. <0 9, No global oscillations. 36,5048“) Global Oscillations. Period 0(lf) Cannot tell. Figure 5.4: Dynamic behavior for various c, Proof. Suppose all the variables :1:,- are small, say l:1:,| g {/5. Then the dynamics of the i-th neuron is governed by it, 2 EC, + pp??? + 5 Z 60'1", + h.o.t. (5.6) In this case we say that the network is in Stage I. If at least one of :1:,- is large, say |:1:,-| > {/5, then the i-th neuron dynamics is governed by the system i‘, 2 EC, + 1),;1'? + €§,($) + h.o.t. (5.7) for some function a. We call this Stage II. Since the number of neurons n is finite, the network spends 0(3):) in Stage II and, hence, the rest of the time (namely (Xi) ) in Stage I. Suppose c, > 0, i.e. the i—th neuron in the uncoupled network (5.5) oscillates. Can weak connections between the neurons stop the oscillations or distort substantially the period? The answer is NO. Indeed, the oscillations can be stopped only when :1:,- is small enough. Suppose the network is in Stage II. Its dynamics is governed by (5.7). The value of the function g,- could be negative and greater than c,- in absolute value. In this case 113,: —O(€) 82 but it could happen only during a time period of total length 0(71fil' Hence :1:,(t) could be pulled back for no more than (9(5- 1&7?) = 0(53/4), but the dynamics in Stage I pushes :1:,(t) forward more than 0(5 - i) = O(\/5). Distance (9(53/4) is negligible in comparison with 0(\/5) for 5 —-> 0. The case c,- < 0 can be analyzed using the same arguments with obvious modifi- cations. D It is not correct to assume that the network is uncoupled for c, 75 0. Indeed, if say c1 > 0 and c2 > 0, then the first and second neurons oscillate and one can observe such phenomena as synchronization or entrainment. Unfortunately, to study these phenomena one needs to consider the network on a very large time scale. If, say, c3 < 0, then the system still can be wobbling in a neighborhood of —\/ 5c,/ p, due to the influences from the other neurons, but the oscillations have very small amplitude and, again, are negligible for an observer. The theorem above covers the case c, 75 0. What if c, = 0 for some, but not all i? In this case it is possible to show that activity can be convergent or periodic. For example, consider a network of VCONs of the form (bl = 1 — cos gbl + 5cm sin2 452 gbg =1— cos¢2 + 5c; where c2 > 0, and, hence, gbz oscillates. It is easy to see that in the case C12 < 0 the variable qbl stays in a neighborhood of the origin, but in the case C12 > 0 it oscillates. 5.5 Adaptation Condition Is Satisfied Now suppose c, = 0 for all i. Recall that in this case we say that the adaptation condition is satisfied. In Chapter 2 we showed that the local dynamics of (5.2) is 83 governed by the canonical model 113:: 7‘, + 5,513, + 13+ 2: c,,-:1:,-. (5.8) i=1 How much can we tell about global dynamics of (5.2) studying this canonical model? Obviously, all attractors of the canonical model are also local attractors of the original weakly connected neural network and vice—versa, since (5.8) is obtained from (5.2) (essentially) by rescaling. Can the weakly connected neural network have an attractor which the canonical model does not have? The answer is YES, but as one expects, the attractor cannot be local. Following is an example of a weakly connected neural network with non—trivial global behavior which the canonical model cannot have. Consider a weakly connected network of VCONS of the form { qbl 21— cos (151 — 52(1+ cosq51)+ 5V( 0. It is easy to check that such system is near a multiple saddle—node bifurcation for $1 : ab = 0. The canonical model for this system is 17: = —1 +173, i = 1,2. It is uncoupled because V’ (0) = 0. Obviously, the only attractor is a stable node :1: = (—1,—1). Nevertheless, it is possible to show that (5.9) has a family of global limit cycles for all sufficiently large a. One such limit cycle is depicted in Figure 5.5. Thus, the analysis of the canonical model does not necessarily reveal global dy- namics of the original weakly connected neural network. 84 Figure 5.5: Co—existence of a local attractor and a global limit cycle Chapter 6 Multiple Andronov—Hopf Bifurcation Recall that a WCN N near a multiple Andronov-Hopf bifurcation point is governed (see Theorem 11) by the dynamical system of the form 2:- = b,Z,+d,z,|Z,|2+ZC,ij, (6.1) 3553' where b,, c,,, d,, z,- E C. In this chapter we study general properties of this canonical model. 6.1 Complex Synaptic Coefficients c,,- It is sometimes convenient to rewrite (6.1) in polar coordinates: Let 2,- = new", b,- = p,- + iw,, d,- = 01, + i5, and c,, = Ic,j|ewv, then (6.1) is equivalent to the system r." = pm- + our? + 2}}, |c.,lr, 005% + that — (A) 4%, = wi + 51'7"? 1 n - +7, Zj¢5|6ijl7‘j51n(¢j + 11% - (bi)- (6.2) We see that lc,,| represents the strength of synaptic connections between the j-th and i-th neurons while 4),,- = Arg c,,- encodes phase information of the synaptic connec- tions, which we call the network’s natural phase differences. 85 86 Indeed, consider a network consisting of two identical neurons having synaptic connections only in one direction, for example, from 22 to 21. Suppose also that Im d, E 5, = 0, i.e. the frequency does not depend on the amplitude. Such a network is governed by the dynamical system 7‘1, = 107‘1 + 017“? + l012l7‘2 COS($2 + $12 — $1) $1, = w + fil012l7‘2 sin(q§2 + $12 — $1) r2’ = pr2 + arg’ $2, = w. The unique stable solution of this system satisfies 452(7) + $12 — $1(T) E 0 mod 27r, i.e. the oscillators have constant phase difference wlg. This motivates our definition of the natural phase difference since it occurs “naturally” in a pair of neurons connected in one direction. Notice that when $12 < 0, an observer sees that one of the oscillators (in this case 21) oscillates with some time delay. Obviously, it would be wrong to prescribe this to spike propagation or synaptic transmission delays. As we will see later, the coefficient 4112 may take on many values depending on the synaptic organization of the network, but not the speed of transmission of spikes through axons or dendrites. If the neurons are connected in both direction (i.e. cm 75 O and c2] # 0), then the phase difference between them generically differs from Arg cm or Arg 621. N ever- theless, even for a network of n interconnected neurons we can prove the following result Lemma 39 If c,,- # 0, then there are values of the parameters p1, . . . ,pn such that qb,(r) — cf),(r) mod 27r —1 41,], (6.3) i.e. the i-th and j-th oscillators have constant phase difference 11),, 87 Proof. Fixj and let p,- = 1,p,- = —% for i aé j. After rescaling (z,- —> dz,- for i at j) the system (6.1) transforms to 2:" = (1 + in')Zj + diZlejV + 0(5), 2:" = ilcijzj — 22') + 0(1), i751 Applying singular perturbation methods (see Hoppensteadt (1993)) to the second equation we see that z,(r) = c,jz,-(7') + (9((5, e‘f'). After a short initial transient period (e.g., 7' > 0(Idlog 6)|) this equation gives (6.3) for 6 —> 0. D It follows from this proof that the choice of parameters corresponds to the case when the amplitude of j—th neuron is much bigger than that of the other neurons. One can consider the j-th oscillator as the leading one that synchronizes the whole network. A similar phenomenon was studied by Kazanovich and Borisyuk (1994). The representation (6.2) of the canonical model is interesting since one can obtain valuable information about behavior of coupled oscillators simply by looking at (6.2), without any further mathematical analysis. Indeed, it is easy to see that the impact of one oscillator on the amplitude of another one is maximal when their phases are synchronized so that (6.3) holds, since COS(¢1+ $0“ — 95:) = cos(z,b,j — $0) = 1 reaches its maximal value then. If the two oscillators are completely out of phase, i.e. if ¢,(T) — 45(7) mod 27r = $153: g, then cos(¢,- + 1b,, — ¢,) = 0 and the influence of j-th neuron on the amplitude r, of i-th neuron is negligible even when |c,,-| is very large! 88 It is also easy to see that the larger is an oscillator amplitude Tj, the larger its impact on the other oscillators. Conversely, if i-th neuron has very small amplitude r,, then it is susceptible to the influences of the other oscillators because of the term 1 " , '7: Z lc.-,|r, Sln(¢j + $25 - (A), ‘ #i which can grow as r, —+ 0. We next study some local properties of (6.1), in particular, the stability of the origin 21=---=z,,=0. 6.2 Oscillator Death and Self-ignition Note that the canonical model (6.1) always has an equilibrium point 21 = - - - = 2,, = 0 for any choice of parameters. If all p, are negative numbers with sufficiently large absolute values, then the equilibrium point is stable. In this section we study how the equilibrium can lose its stability. Using the canon- ical model we also illustrate two well-known phenomena: Oscillator death (or quench- ing, or Bar-Eli effect) and coupling-induced spontaneous activity (or self-ignition). We consider (6.1) for the most interesting case when Red, < 0 for all i. This corresponds to supercritical Andronov-Hopf bifurcation, i.e. to a birth of stable limit cycle. We start from the observation that in the canonical model each oscillator is gov— erned by a dynamical system of the form 2’ = (p + iw)z + dzlzlz. (6.4) Obviously, if p S 0, then the equilibrium 2 = 0 is stable. As p increases through p = 0 (6.4) undergoes Andronov-Hopf bifurcation. As a result, the equilibrium losses its stability and (6.4) has stable limit cycle of radius O(,/;3) for p > 0. 89 We can characterize the qualitative differences in dynamic behavior of (6.4) for p S 0 and p > 0 as follows: 0 When p S 0, dynamical system (6.4) describes intrinsically passive element incapable of sustaining periodic activity. 0 When p > 0, dynamical system (6.4) describes intrinsically active oscillator, or pacemaker. In the uncoupled network (C = 0) z,'= (p+iw)z,+d,z,|z,|2, i=1,...,n (6.5) of such oscillators the equilibrium point 21 : = 2,, = 0 is stable for p S 0 and unstable for p > 0 We ask what happens when we consider the canonical model (6.1) with nonzero matrix C = (c,,~)? A partial answer is given in the following result. Lemma 40 Let 0 denote the largest real part of all eigenvalues of connection matrix C = (c,,-). Consider the network of identical oscillators governed by Z,’ = (p + ‘15))2, + d,z,-|z,|2 + 2 $ij , 2:1, . . . ,n. (6.6) i=1 The equilibrium point 21 = = 3,, = 0 is stable if p < —a. It is unstable if p > —a. Proof. The full system has the form ...! _ ' . .7. c..2 7‘ ...... l4: _ (p+lw)z’+d‘”"“" +Z,=,c.,~.» i=1,...,n. (6.7) 21'- = (P — iw)§, + d,§,|z,|2 + 2L1 5:551» 90 It is easy to see that the origin 21 = 21 = = 2,, = 2,, = O is always an equilibrium point of (6.7). The (2n) x (2n) Jacobian matrix J at the origin has the form J___ ( (p+iw0)E+C (p_iw11)E+C), where 0 and E are the n x n zero and identity matrices, respectively. Suppose A1, . . . , An are eigenvalues of C counted with their multiplicity and sup- pose that v1, . . . , vn are corresponding (generalized) eigenvectors. Direct computation shows that J has 2n eigenvalues p+iw+A1, p—iw+A1,..., p+iw+A,,, p—iw+A,, and 2n corresponding eigenvectors (3W.)’---w(”o”)’(.°.)v where 0 denotes a vector of zeros. Stability of (6.7) is determined by the eigenvalues of J with maximal real part. These eigenvalues are of the form p+iw+A, p—iw-l—A, where A are eigenvalues of C. Let er :Re A, then the origin is stable if p+a0. In the rest of this section we use Lemma 40 to illustrate two interesting effects. 0 If a < 0, then the network (6.6) is stable even when 0 0, then the network (6.6) can exhibit spontaneous activity even when —a < p < 0, i.e. when each oscillator is intrinsically passive, coupling can induce synchronous activity in the network. This effect, which can be called self-ignition (Rapaport 1952), is discussed in details by Kowalski et al. (1992). Remark 41 IfC has only one eigenvalue with marimal real part, then (6.6) under- goes an Andronov-Hopf bifurcation as p increases through p = —a. Note that in this case the coordinates of the limit cycle depend upon the matrix C; more precisely, upon the eigenvector that corresponds to the ”leading” eigenvalue of C. Thus, to understand the dynamics of the canonical model (6.6), one should understand possible structures of the connection matrix C = (c,,-). We will do this in Chapter 10. 6.3 Synchronization and Convergence In this section we reveal the conditions under which the canonical model (6.1) can operate as an MA-type NN (see Section 1.4). First, we assume that all d,- are real and negative. Without loss of generality we may take d,- = —1. Thus, we study dynamical system of the form 2:: = (p, + w,)z, — 2,|2,|2 + Z c,,-zj. (6.8) 3'95 92 We take advantage of the fact that the system is invariant under rotations to prove the following theorem. Theorem 42 (COHEN—GROSSBERG CONVERGENCE THEOREM FOR OSCILLATORY NEURAL NETWORKS) If in the canonical model {6.8) all neurons have equal center frequencies to, = = can 2 w and the matrix of synaptic connections C = (c,,-) is self-adjoint, i.e. Cij = 5,13 then the neural network dynamics converges to a limit cycle. 0n the limit cycle all neurons have constant phase shifts, which corresponds to synchronization of the network activity. Proof. In the rotating coordinate system u, = e—imz,(r) (6.8) becomes 71 u,’ = p,u, — u,|u,|2 + 26,311,, i = 1,. . . ,n. (6.9) i=1 Note that the mapping U : C2" —> 1R given by 71 _ _ 1 " _ U(u1,..,u,,,u1,..,u,,) : — Z (piluz'l2 - glitz)” + Z Cijuiuj) 1:1 i=1 is a global Liapunov function for (6.9). Indeed, it is continuous, bounded below (because it behaves like §|u|4 for large u), satisfies , av _, av __EE;1 1 ——CU,, and, hence, dU " 8U 8U " a; = Z (a—“" + 55“?) = ‘2: '“(f S 0' i=1 ‘ ' i=1 Notice that 53—? = 0 precisely when u1’ = 2 un’ = 0, i.e. at the equilibrium point of (6.9). Let u* E C" be such a point. Then, while the solution it of (6.9) converges 93 to u*, the solution of (6.8) converges to the limit cycle 2 = eiwu*. Obviously, any pair of oscillators have constant phase difference on this limit cycle. D It should be noted that dynamics of (6.8) can converge to different limit cycles depending upon the initial conditions and the choice of the parameters p1, . . . ,p,,. For fixed parameters there could be many such limit cycles corresponding to different memorized images (Baird 1986, Li and Hopfield 1989). The theorem states that the canonical model (6.8) is an MA-type NN model (Hop- field 1982, Grossberg 1988), but instead of equilibrium points, the network dynamics converge to limit cycles, as was postulated by Baird (1986). Whether this new feature renders the canonical model any advantages over the classical Hopfield model or not is still an open question. We continue our studying of the canonical model in Chapter 10. Chapter 7 Multiple Cusp Singularity We analyze the canonical model (2.21) for 5 —> 0. Thus, we study the system :17:- = r, + b,:1:,- + 0,3? + Z c,,-:1:,~, (7.1) j=l where r,, r,, b,, c,,- are real variables and a, 2 21:1. Before proceeding further, we explain the meaning of the data in (7.1). Each :1:, depends on X, and is a scalar that describes in some sense activity of the i—th neuron, and the vector :1: = (:121,...,:1:,,)T 6 IR” describes a physiological state of the network; The parameter r, 6 IR is an external input from receptors to the i—th neuron. It depends on A and p. Each b, E R is an internal parameter, which also depends upon A and p. The vector (b1, .., b,,)T E R" is a multidimensional bifurcation parameter; C = (c,,) E Rn“ is a matrix of synaptic connections between neurons. There is strong neurobiological evidence that synapses are responsible for associative memorization and recall (Shepherd 1983). We will see this realized in the canonical model (7.1) but from a rigorous mathematical point of view. It should be stressed that (7.1) is an interesting dynamical system in itself without any connection to the WCNN theory. It exhibits useful behavior from a computational point of view and deserves to be studied per se. The simplicity of (7.1) (in comparison with (2.1)) is misleading. It is very difficult 94 95 to study it for an arbitrary choice of the parameters. We will study it using bifurcation theory. Nevertheless, we can answer interesting questions about (7.1) only by making some assumptions about the parameters {r,}, {(2,} and C. In Section 7.1.2 we assume that the input from receptors is very strong, i.e. all r, have large absolute values. In Section 7.1.3 we consider extreme choices for b,. Section 7.3 is devoted to the study of (7.1) under the classical assumption that the matrix of synaptic connections C is symmetric. This restriction arises naturally when one considers Hebbian learning rules, which are discussed in Section 7.4. In Section 7.5 we study bifurcations in the canonical model (7.1) when the external input r, = 0 for all i. In this case (7.1) coincides with the canonical model for multiple pitchfork bifurcations in the WCNNs. Section 7.6 is devoted to studying the canonical model when only one or two images are memorized. In Sections 7.7 and 7.8 we illustrate the phenomena of bistability of perception and decision making in analogy with problems in psychology. We start studying (7.1) by asking the following question: What is the behavior of its solutions far from the origin 1‘ = 0, i.e. outside some ball BO(R) C R" with large radius R? 7 .1 Extreme Values of Parameters 7 .1 .1 Global behavior Let BO(R) C IR” denote a ball with center at the origin and radius B. By the term ”global behavior” we mean a flow structure of a dynamical system outside the ball Bo( R) for sufficiently large B. A dynamical system is bounded if there is R > 0 such that BO(R) attracts all trajectories, i.e. for any initial condition :1:(0) there exists to such that :1:(t) E BO(R) 96 for all t Z to. Obviously, all attractors of such a system lie inside BO(R). Theorem 43 A necessary and sufl‘icient condition for {7.1) to be bounded is that 01 = = 0,, = —1, i.e. (7.1) must be c122: r,+b,$,—$?+Zc,ja:j. (7.2) i=1 This corresponds to a multiple supercritical cusp singularity. Proof. We are interested in the flow structure of (7.1) outside some ball BO(R) with sufficiently large radius R. Let 5 = R‘2 be a small parameter. After the rescaling :r, —> \/53:,, t ——> 5‘1t, (7.1) can be rewritten as n I:- = 0,1223 + 5(b,:1:, + Z Cijivj + £73), 1:1 which must be studied outside the unit ball 80(1). Note that this is an 5-perturbation of the uncoupled system :L"- = 0,:1‘33 1 g i S n. (7.3) 1? Obviously, the unit ball 80(1) attracts all trajectories of (7.1) if and only if all 0, < 0. Any 5-perturbation of (7.3) has the same property provided 5 is small enough. D So, the flow structure of (7.1) outside a ball with sufficiently large radius looks like that of (7.3), as depicted in Figure 7.1a for the case 01 = 02 = —1 and in Figure 7.1b when 01, is positive. To be bounded is a desirable property in applications. Any initial condition :1:(0) lies in a domain of attraction of some attractor that lies somewhere inside Bo(R). Hence, for any :1:(0) we have at least a hope to find the asymptotic dynamics. From now on we will consider only (7.2) as the canonical model of the WCNN near a multiple cusp singularity point, i.e. we study the supercritical cusp singularity. 97 5132 D a b Figure 7.1: Global flow structures of the canonical models. a. System (7.1) for 01 = 02 = —1 is bounded. b. System (7.1) for 01 = +1, 02 = —1 is not bounded. 7 .1.2 Strong Input From Receptors What is the behavior of the canonical model (7.2) when the external input from receptors is very strong, i.e. when the parameter R = min IT‘,I 3 is very large? 2 . . 1~ §.~ Let 5 = R'5 and rescale variables by setting: :1:,- = 521‘,, r, = Br, and 7' 2 5t. Then (7.2) can be rewritten as (after dropping ~) 513;: 7‘, —Cc?+€(b,$,+ZC,J‘IJ‘). (7.4) i=1 System (7.4) is 5—perturbation of the uncoupled system 1:: = r, — at? (7.5) 1 1 It is obvious that (7.5) has only one equilibrium point :1: = (r13, . . .,r,:§)T for any external input r1, . . . ,r,,. The Jacobian at that point is g —3rf’ 0 - - ' 0 _ i L = 0 3""? . 0 (7.6) g 0 0 —3r3 Note that according to the rescaling all |r,| 2 1, hence all diagonal elements in L are negative. Thus, the equilibrium point is a stable node. 98 All of the equilibrium points considered above are hyperbolic. Any 5—pertur- bations of (7.5) do not change the qualitative picture provided 5 is small enough. So, the phase portrait of (7.2) for strong external inputs is qualitatively the same as that of (7.5). Thus, when the input from receptors is strong (in comparison with the synaptic connections c,,- or the internal parameters b,), then the network dynamics approach the unique equilibrium. At this equilibrium each neuron is either depolarized (excited) or hyperpolarized (inhibited) depending on the sign of the input r,. 7 .1.3 Extreme Psychological Condition It is convenient to assume that the set of parameters {b,} describes the psychological state of the WCNN because it affects the way that the network reacts to external inputs. One might speculate that when t3 = min lbil is very large, the network is working in an ”extreme psychological condition”. We use the same method of analysis of (7.2) as in previous sections. Let 5 = 3“ be a small parameter. By rescaling (:1:,- = \/5:i:,, b, = 5b,, 7’ 2 5t) we can rewrite (7.2) as n r: = b,:1:,- — x? + 5(2 c,,~.r,~ + \/5r,-). (7.7) 1:1 The weakly connected system (7.7) is an 5-perturbation of the uncoupled system (I): = 5,23, - 1,13. (7.8) In (7.8) each equation has either one or three equilibrium points depending upon 99 ——..—.—' —._. -—-—-v.¢—O—-..¢———- l. J... l | Figure 7.2: Phase portrait of (7.2) working in the extreme psychological regime. a. Allb,>0. b. b, >0, b2<0. the sign of b,. The points are 2:, = 0 for any b, and in addition :17, = :l:\/b—, for b, > 0. The Jacobian of (7.8) is 1.1—3.1:, 0 0 0 b2—3xg 0 0 0 b,,—3:1:,21 Obviously, the equilibrium point :1: = (2:1, . . . ,:1:,,) of (7.8) is a stable node if 0 if b, < 0 ”‘={ \/b_,- ifb,>0. It is a saddle if some (but not all) of these conditions on 1:, are violated, and it is an unstable node if all of the conditions are violated. Note that the presence of saddles and the unstable node is possible only if there exist at least one positive b,. If all b, < 0, then there is only one equilibrium point :17 = (0,. . . ,0)T, which is the stable node (see Figure 7.2). 7 .2 Canonical Models as a GAS-Type NN Systems that have only one asymptotically stable equilibrium point and do not have any other attractors are called globally asymptotically stable systems (Hirsch 1989). Such systems are good candidates for GAS-type NNs (see Section 1.4). We have 100 already seen that (7.2) is globally asymptotically stable when the input r from re- ceptors is strong. (The values of the other parameters, viz. b and C, are irrelevant). This is also true when (7.2) is considered for negative b, with large absolute values. In the previous section we have assumed that all Ib,| are large. The remarkable fact is that for (7.2) to be globally asymptotically stable it is suffices to require that b = (b1, . . . ,b,,)T take only intermediate values comparable to those of entries of C. A more accurate statement is the following: Theorem 44 The dynamical system (7.2) n I 3 (I),- = 7‘, + 5,113, — CC,- + E C,j.”L'j 1'21 is globally asymptotically stable if I b, < — (q, + g 2 ICU + Cjil) (7.9) j¢i foralllgign. Proof. We will prove it using Hirsch’s theorem (Hirsch 1989). Let L be the Jacobian of (7.2) at a point :1: = (1:1, . . . ,:1:,,)T. Hirsch’s theorem claims that if there is a constant —(5 < 0 such that (145,5) S —5(€,E) for all 6 = (61,. . .,§,,)T E R" then (7.2) is globally asymptotically stable. Here (5,17) denote the inner (dot) product of vectors 5 and 17. It is easy to check that (L35) = EUR — 3133K? + Z Cijfiéj i=1 i,j=l 101 i.J=l #1 : ;( (,b +C,,- 3.73?) ),E 'i" 2 12(61'1' +Cjiléi 6] 3:0} +Cn)€i2+ 4ZlCij+cjil( REE—+62) 5951' =i (bi+cii+‘;‘:lcij+cjil) 5,2 < ‘52:? i=1 i=1 j¢i where —6 = max,(b, + c,, + % Enh- |c,, + c,,-|). We used here the inequality 5.5-815%,?» —2 The inequality (7.9) guarantees that —6 < 0 and, hence, all the conditions of Hirsch’s theorem are satisfied. This completes the proof. Cl Note that the inequality (7.9) is much more appealing than the requirement that the absolute values of b, be arbitrary large. Thus, even for ”reasonable” values of the internal parameters b,, the dynamics of (7.2) is globally asymptotically stable. Another remarkable fact is that the external input r E R” does not come into the condition (7.9). What it does affect is the location of the unique attractor of the network. Therefore, the canonical model (7.2) for b, satisfying (7.9) can work as a GAS—type NN. 7 .3 Symmetric Synaptic Connections In our analysis of (7.2) above we have assumed that some of its parameters take their extreme values. The dynamics of (7.2) always converges in these cases, i.e. its attrac- tors are stable equilibrium points. Next we consider (7.2) without the requirement that any of its parameters take extreme, possibly implausible values. 102 Theorem 45 (Cohen-Grossberg) If the matrix of synaptic connections C is sym- metric, then the neural network n. I 3 (E,- = 7‘, + 5,117, — (II,- + E C,jCL‘j i=1 is a gradient system. The symmetry of C is a strong requirement, but it arises naturally if one considers the Hebbian learning rules, which will be discussed in the next section. Such systems are widely studied in the neural network literature (see review by S. Grossberg 1988). Proof. To prove the theorem it suffices to present a function U : R" —> IR satisfying , (9U i 8.71,. It is easy to check that 71 " 1 1 1 {/(1') Z — ZUNI, + ‘Z’bgfl‘? '— all?) — i Z Cijmixj i=1 i,j=l is such a function for (7.2). CI Note that far away from the origin U(:1:) behaves like 12:1 33?, hence U(a:) is bounded below. Being a gradient system imposes many restrictions on the possible dynamics of (7.2). For example, its dynamics cannot be oscillatory or chaotic. Nevertheless, this property is considered to be very useful from computational point of view, which we discuss later. To what type of NNs does the canonical model (7.2) belong? It is clear from previous sections that for one choice of the parameters (7.2) has many attractors and, hence, is a candidate for MA—type NN, whereas for other choices of the parameters it is globally asymptotically stable and, hence, is the GAS—type NN. We will show 103 in the next section that (7.2) stands somewhere between MA and GAS-types and, hence, will be considered as a new NN type. We are interested in the basic principles of the human brain functioning. Hence, we will study only the qualitative behavior of the canonical model and neglect quan- titative features. The main tools in the analysis below come from bifurcation the- ory. Unfortunately, comprehensive analysis of the model for arbitrary r,, b, and C is formidable unless we impose some additional restrictions onto the parameter spaces. Among them there are two we discuss now. First of all, we will study the canonical models when Thus, instead of n bifurcation parameters b1, . . . , b,, we have only one b 6 IR. The second assumption concerns the matrix of synaptic connections C, which is responsible for learning in the NNs. 7 .4 Hebbian Learning Rule For Synaptic Matrix C Little is known about the processes of learning in a human brain. One basic approach to studying learning and recognition in artificial NNs is to assume that the synaptic matrix is constructed according to Hebb’s learning rule (Hebb 1949) 1 "‘ . . Cij = g E flséfgjia 1S 27.7 S n, (710) 8:1 where {3 2 (5f, . . . ,€:)T 6 IR”, 5 = 1,... ,m are key patterns to be memorized; The constants 3, measure ”strength” or ”quality” of the memory about the patterns {3; The number of memorized patterns, m, cannot exceed n. The Hebbian learning rule 104 (7.10) can be rewritten in the more convenient way 0 -—- iZaswf. (7.11) T where means transpose. It is easy to see that the synaptic matrix C constructed according to (7.11) is symmetric. It is also true that any symmetric matrix C can be represented as (7.11) for some, possibly non—unique, choice of the orthogonal vectors 51,. . .,§"‘. Thus, we have proved the following Proposition 46 The matrix of synaptic connections C is symmetric if and only if there is a set of orthogonal patterns £1,”qu such that C is constructed according to the Hebbian learning rule {7.11). Note that the orthogonal vectors {1, . . . , {m are eigenvectors of C. If we normalize them such that Tl. W = (5.5) = 2(5): = n, i=1 then the constants 6, are eigenvalues of C and (7.11) is the spectral decomposition of C. We assume that 61 Z 2 6", > 0. Ifm < n, then there is an n —m dimensional eigenspace kerC C IR" corresponding to the zero eigenvalue. We denote this eigenvalue by 6",“ = 0. To summarize we can say that the Hebbian learning rule for orthogonal patterns gives a way of constructing the matrix of synaptic connections such that each pattern is an eigenvector of the matrix corresponding to positive eigenvalue. In our analysis below we will assume that the learning rule is Hebbian. This as- sumption imposes significant restrictions on possible dynamic behavior of the canon- ical model. For example, it follows from the Section 7.3 that (7.2) is a gradient dynamical system. 105 In order to make all computations without resorting to computer simulations we also assume that |§f| = 1 for all i, i.e. For these purposes we introduce the set 3" = {5 5 IR”, §=(:t1,...,i1)T} c: IR". (7.12) We will also need for our analysis an orthogonal basis for R" that contains vectors only from E". This basis always exists if n = 2" for some integer k > 0. The assump- tion that {3 E _. might look artificial, but it is very important in neurocomputer applications and in digital circuit design. Recall that we are interested in qualitative behavior of the canonical models. All attractors that we will study below are hyperbolic. Hence, if we perturb the parameters r,,...,r,,, b1,...,b,, and C, i.e. if we violate the assumptions made above, the qualitative behavior will be the same provided the perturbations are not very large. 7 .5 Bifurcations for r = 0 We start the bifurcation analysis of the canonical model (7.2) for the special case when there are no receptor inputs, i.e. when r, = = r,, = 0. Thus, we are interested in qualitative behavior of the dynamical system $1: 5.13, — .1'?+ ZC,jSL‘j IS 2 S n. (7.13) i=1 It is easy to check that the canonical model (7.13) describes a WCNN with Z; symmetry :1: —> —:1: near a multiple pitchfork bifurcation point. Symmetry means 106 that for b > 0 each neuron is essentially a bistable element with two states: excitation (:L', = \/b) and inhibition (at, = —\/b). 7 .5.1 Stability of the Origin Note that (7.13) always has a equilibrium point 2:1 = = :1:,, = 0. It follows from Section 7.1.3, that the origin is the only equilibrium point, which is a stable node, for b << —1, or for b satisfying (7.9). We also know that for b >> 1 the canonical model has many attractors (see Figure 7.2). What we do not know is the behavior of the model for intermediate values of b. Thus, we have the following questions: What happens while b is increasing? How does the origin lose its stability? How many and of what type are the new equilibrium points? What is the relationship between them and the synaptic matrix C = (c,,)? These and other questions are studied in this section. Let L be the Jacobian of the right-hand side of (7.13) at the origin. It is easy to see that L=bE+C, where E is the unit matrix. Let 31,. . . ,5", be the (distinct) eigenvalues of C ordered such that R631 2 2 R63... counted with multiplicity. Obviously, L has m eigenvalues with the same eigenvectors as those of C. The matrix L has all eigenvalues with negative real parts, and, hence, the origin is a stable equilibrium point for (7.13) if and only if b < -Ref31 (see Figure 7.3a). 107 1“,. 1 it i , i1 / \ 5 51 1 m 1 ‘ a b c (1 Figure 7.3: Phase portrait of the canonical model (7.13) of weakly connected neural network near multiple pitchfork bifurcation point for different values of the bifurcation parameter b. a. b < —fll. b. —,81 < b < —flg. c. -—62 < b < -—,32 + (61 — ,82)/2. d. “22 +(181_ 529/2 < b- If ,8, is a real eigenvalue with multiplicity one, then (7.13) undergoes a pitchfork bifurcation when b crosses —-fil. For b slightly larger than —fl1 the origin is a saddle surrounded by two sinks (see Figure 7.3b) and those are the only equilibrium points for (7.13). If (61,62) is a pair of complex conjugate eigenvalues with multiplicity one, then we can observe the Andronov-Hopf bifurcation for b = —Refil. For b > —fl1 it is possible to observe the birth of a pair of saddles or an unstable limit cycle every time b crosses —Refl,, where 68 is an eigenvalue with multiplicity one. For the eigenvalues with multiplicity more than one bifurcations could be more complicated. Nevertheless, we will consider some of them later. Recall that each neuron is bistable only for b > 0. For negative b there is only one stable state :13, = 0 and, hence, it is ”passive”. But when the neurons are connected they acquire a new property: bistability for —fi1 < b < 0. This is the property that each neuron alone cannot have. Thus a network of ”passive” elements can exhibit ”active” properties. This is called self-ignition and has already been studied in Section 6.2 for oscillatory neural networks. We encounter this phenomena frequently in our 108 analysis of brain function. 7 .5.2 Stability of the Other Equilibria It is noteworthy that we have not restricted the synaptic matrix C yet. All the bifurcations discussed above take place for any C. In return for this generality, we cannot trace the new equilibrium points and study their stability. Fortunately, we can do it if we assume that the synaptic matrix C is constructed according to the Hebbian learning rule _lm . sr C_n;fl’€(€)’ 212 Zfim>0 and that the memorized images {1, . . . ,{m E E" are orthogonal. For simplicity we assume that all 5, are different. At the end of this section we will discuss the case 51 = = fim. Let :1:, = ysff for i = 1,..,n. Then $2 = yééf = byséf - y3(€f)3 + [3.1/.6?- After (dot) multiplication by (f, we have y; = (b+fi.)ys -—yf.‘. (7.14) This equation has only one equilibrium point y, = 0 for b < —,3, and for b > —6, there are three points y, = 0, y, = :l:\/b + ,83. Hence the original system (7.13) has two new equilibrium points :1: = :l:\/b- + B, {s after b crosses —fl,. Note that the pair of new equilibrium points lies on the line spanned by the memorized pattern {3. Every attractor lying on or near span(§3) is called an attractor corresponding to the pattern {3. When the network activity :1:(t) approaches such an attractor, we say that the NN has recognized the memorized image {3. To our surprise only the pair a = :tx/b + 561 is a pair of stable nodes, whereas 109 the others 5 = :l:\/b + ,3, {3, s 2 2 are pairs of saddles, at least when b is near —fl, (see Figure 7.3c). Let us study the stability of :1: = :l:\/b + 5;, {k for some 1 g k S m. The matrix of linearization L at this point is L = (b— 3(b+ 01.))E + C. It has eigenvalues A. 2 —2(b+fl1)+fi.—fik. 133 s m+1. where fin,“ = 0 corresponds to ker C. Note that Al 2 2 Am > Am“. The maximum eigenvalue A1 = —2(b + 5),.) + 61 — ,8}, is always negative only for k = 1. For k 2 2 the inequality A, < 0 gives us the condition b > 4,. + 31f". (7.15) One could say that in the life of the equilibrium point \/b + 61,5" there are two major events: Birth (when b = —43k) and maturation (b = —/3,,, + (61 — 6m)/2 ) when the point becomes a stable node. For k = 2 see Figure 7.3d and 7.4. It is easy to see that when b = —/3k the eigenvalues of L are A12°'°>Ak—120=Ak EM“ 2 >/\m- So, \/b + 5,, 5" is the saddle such that k — 1 directions corresponding to {1, . . . , 6““ are unstable (see Figure 7.5a). Every time b crosses —Hk + (6, — 50/2, 3 < k there is a pitchfork bifurcation. As a result, the direction corresponding to {3 becomes stable and there appears a new pair of saddles lying in span(£’,{k) (see Figure 7.5b,c and d). To summarize, we can say that for b < —31 the only equilibrium point is the origin, but for b > —fl,,, + (61 — 6m) / 2 there are m pairs of stable nodes corresponding to the memorized images {1, . . . ,{m and many saddles lying in between these nodes. 110 _B2 -B,+(B,-B,)/2 Figure 7.4: Bifurcation diagram. '2 2 2 2 .5 5‘ ,5 £1 (,5 c f 51 /§3 /53 l/3 1,0 3 ‘fi 0 > 4—0—0-04—0—9- a—O—>O<——O—> 4—0—pv.4—O—.. / / 1 01 O / o v v 1 t a b c (1 Figure 7.5: Every equilibrium point :l:\/b + flkfk becomes an attractor after the se- quence of the pitchfork bifurcations. Every time b crosses —fik + (3, — flk)/2, s < k, the 63-direction becomes stable. 111 Recall that we referred to the attractors that do not correspond to any of the memorized images as being spurious memory. Is there any spurious memory in (7.13)? The answer is yes. Fortunately, it happens for large b. Indeed, when b > 0 all eigenvalues of the matrix of linearization at the origin are positive, not only A1, . . . , Am. The new unstable directions correspond to ker C (of course, if m < n). It is easy to check that for 0 < b < @21- all the equilibrium points lying in kerC are saddles (except the origin, which is an unstable node), whereas for b > 42‘— there are 2(n — m) stable nodes among them. In order to avoid spurious memory, one should keep the bifurcation parameter below the critical value 52—1. Actually, it is more reasonable to demand that b be negative. By this means we guarantee that nothing interesting is going on in the directions orthogonal to the all memorized images. But we must be cautious because not all equilibrium points corresponding to memorized patterns are stable for b < 0. Indeed, the stability condition (7.15) for b < 0 can be satisfied only if flk>%. Thus, all memorized images are stable nodes and successful recognition is possible if the weight B", of the weakest image is greater than one third of that of the strongest one. Obviously, we do not have this kind of problem when 31 = = fin, = 6 > 0. For b < —3 the NN is globally asymptotically stable. For b = —6 there is a multiple pitchfork bifurcation with the birth of 2m stable nodes corresponding to the memorized images. For —fi < b < 0 these nodes are the only attractorsl and behavior of the NN (7.13) is very simple in the directions orthogonal to the span (61,. . . ,fim). Thus, the system (7.13) can work as a typical MA-type NN. If the initial condition 1Actually, the correct statement is that these are the only attractors that bifurcated from the origin. Whether there are other attractors or not is still an open question. 112 x(0) is an input from receptors, then the activity x(t) of (7.13) approaches the closest attractor, which corresponds to one of the previously memorized images. It is believed that this simple procedure is a basis for construction of new generations of computers — neurocomputers. Nevertheless, our opinion is that this is too far from the basic principles of how a real brain functions (despite the fact that we know almost nothing about these principles). In the next section we explore another, more realistic, approach. 7 .6 Bifurcations for r 32$ 0 (two memorized images) As we have already mentioned, comprehensive analysis of the canonical model 1::- = r, + bx, — x? + Z c,,-x,- (7.16) 1:1 is formidable. Even for a symmetric synaptic matrix C = (c,,-) it is difficult although there is a Liapunov function in that case. Hence, every piece of information about (7.16) obtained by analytical tools is precious. The next step in studying (7.16) is to assume that the number of memorized images m g 2. The key result in this direction is the Reduction Lemma that enables us to reduce the number of independent variables to 2. 7 .6.1 The Reduction Lemma Suppose that 51,. . .,E” E E” form an orthogonal basis for IR” and that {I and {2 coincide with memorized images, where E is defined in (7.12). Let 1 . 1" . ys—gixaél"n;x:€i 113 be the projection of :1: 6 IR" onto ifs. Obviously, :1: can be represented as the sum :1: = 231,5“. (7.17) A similar decomposition is possible for any input r 6 IR": Let =% 0. There are three equilibrium points when |a| < a*, where ._ 2+1; a—2(3). Hence, (7.21) has nine equilibrium points. Again, there is not any qualitative distinc- tion between the phase portrait of (7.13) depicted in Figure 7.3d and that of (7.21), which we depict in Figure 7.6a for a > 0. We see that |a| < a* is too weak to produce any qualitative changes in the dynamics of the canonical model (7.16) in compari- son with (7.13). Nevertheless, it is easy to see that the domain of attraction of the 116 v 1 {1 l 1 l l l E b v > 1+ 0 fi- .3 —>-O-<-O-—>-O-¢- (1 ‘l I I u I u U. mas—+2... | 11.1%.; ’3’ 'fi“ // l l l g2 g2 {2 a b c Figure 7.6: Phase portrait of the canonical model on the stable invariant plane span(§1,§2). The first image is presented as an input onto the network. a. Input is weak, i.e. |a| < a*. b. For |a| = a* there are fold bifurcations. c. For [a] > a* the canonical model is globally asymptotically stable. equilibrium point corresponding to the presented image {I is much bigger than the attraction domains of the other equilibrium points. By the term attraction domain size we mean here the distance from the attractor to the closest saddle. We use this definition in order to be able to compare domains that have infinite volumes. When the parameter a crosses :ta" one can observe two fold (saddle-node) bifur- cations and one co—dimension—2 bifurcation (see Figure 7.6b). All of them take place simultaneously due to the fact that (7.21) is a direct product of two identical equa- tions. We consider these bifurcations elsewhere when we study the canonical model for WCNNS near multiple fold bifurcation point. If the input 1' = (16‘ is sufficiently strong (i.e. if |a| > (1*), then there is only one equilibrium point, which is a stable node (see Figure 7.6c). The equilibrium point is globally asymptotically stable in this case. We see that the canonical model (7.16) can work as GAS—type NN when the input strength a is strong enough, viz. lal >2(‘l%fl—)§. 117 We performed all the analysis above for the case of one presented and two mem- orized images. 7 .6.3 Recognition: Two Images Are Presented Without loss of generality we may assume in this case that 7‘ = 0161 + 0252 for aha; > 0. If fll = ,82 = fl and all the conditions of the Reduction Lemma are satisfied, then the canonical model (7.16) can be reduced to the two—dimensional system (7.21) u’=s+(b+,6)u—u3 {v’=c+(b+fl)v—v3. We cannot reduce (7.21) to a one-dimensional system because in general 3 74 c. The constant 3 = a1 + a2 has obvious meaning of overall strength of the input from receptors, whereas c = a1 — a2 is the contrast of the input. When c > 0 (c < 0) we say that (1 (62) is dominant. In order to determine the qualitative behavior of (7.21) we have to compare .3 and c with the bifurcation value a*. When both 5 and c are less than a*, the qualitative phase portrait of (7.21) depicted in Figure 7.7a coincides with that of (7.13) depicted in Figure 7.3d provided b + fl > 0. Very interesting behavior arises when the overall input from receptors 5 is strong enough, i.e. when 3 > a*. Then, (7.21) generically has either one or three equilibrium points (see Figure 7.7b). Its behavior is determined by the equation v’:c+(b+fi)v—v3, v6 R. (7.22) Obviously, the dynamics of (7.22) depends crucially not only upon which image is dominant but also upon how dominant it is. If |c| < a*, then there is a co-existence 118 v 161 v 1&1 H —->-OO 4.4— :04— l] u u -—>OC'> >C'><— ;fio4_ H l 1 —>00 47.4— 4.4— H To Te a b Figure 7.7: Phase portrait of the canonical model on the stable invariant plane span(§1,£2). The input is a mixture of two images {I and £2. a. Overall input is weak. b. Strong input and weak contrast. There is a co—existence of two attractors. between these two images (see Figure 7.7b). Both equilibrium points are stable. If |c| > a* then only one image survives, viz. the dominant image. One possibility to explain the co-existence of two attractors corresponding to two different images is that the NN cannot distinguish between them when the contrast [CI is small. One could say that the two-attractor state corresponds to the ”I do not know” answer. We prefer another explanation suggested by the psychological experiment described in the next section. 7 .7 Bistability of Perception In the previous sections we showed that if the conditions of the Reduction Lemma are satisfied and the overall input from receptors is strong (3 > a*), then the canonical model behaves qualitatively like the equation (7.22) v'=c+bv—v3, vER, where c = a1 — a2 is the contrast between two images (2161 and agfz and b is a real parameter (we incorporated fl into b, so b can be positive or negative). 119 We have already mentioned that if the contrast is weak (|c| < a*) then (7.22) has two attractors corresponding to the previously memorized images (I and {2. First of all, note that the co—existence of two attractors contradicts the GAS-type N N paradigm which requires that the NN have only one attractor. We must accept the fact that the brain is a very complicated system having many attractors. Its dynamic behavior depends not only upon the input 1', the synaptic memory C and the psychological state b but also upon a short-term past activity (which sometimes is called a short-term memory (Grossberg 1988)). In our case this is the initial condition a:(0). Obviously, which attractor will be selected by the NN depends upon the initial state. Simultaneous existence of several attractors for the input that is a mixture of images suggests the following hypothesis: The NN perceives the ambiguous input according to the network’s past short-term activity (13(0). The behavior of the artificial NN (7.16) is similar to the behavior of the real human brain in the following psychological experiment (Attneave 1971): The fourth figure from the left in the top row depicted in Figure 7.8 was shown to be perceived with equal probability as the face of a man or the body of a girl. If the figure is included in a sequence, then its perception depends upon the direction in which the sequence is viewed. This phenomena was studied from catastrophe theory point of view (Poston and Stewart 1978, Stewart and Peregoy 1983) and it was shown that there is a one- dimensional section of a cusp catastrophe in the human perception of the figures. The remarkable fact is that the WCNN approximated by (7.22) also exhibits the cusp catastrophe. Suppose 61 and {2 represent girl’s body and man’s face images, respectively. If we fix b > 0 and vary the image contrast c = a1 -a2, then the artificial NN also has the same bistable perception of the presented images alfl and a2{2 (see 120 dv dv "dl' Bf ma irl édvvmanktfilVé girl man girl man irl man it] man man 8 | man girl Figure 7.8: Bistability of perception. bottom row in Figure 7.8). What we have not explained yet is the switching of our attention (say, from girl’s body to man’s face and back) while we observe an ambiguous picture. These oscillations in our perception cannot be explained by the catastrophe theory. We can tackle this problem by embedding the WCNN (2.1) into the A-space, i.e. by allowing the internal parameter A to vary (see Section 2.6.1). This idea was used by Ditzinger and Haken (1989). As we can see, the canonical model (7.16) can work as the MA and GAS-type NNs simultaneously. Indeed, its dynamics crucially depends upon the input 1‘ 6 R" from receptors. If the input is strong enough and there is no ambiguity, then (7.16) has only one attractor and, hence, works as the GAS-type NN. If the input is weak or ambiguous, then (7.16) can have many attractors and, hence, can work as the MA-type neural network. We think that the real brain might use similar principles. Consider, for example, the olfactory system (Baird 1986, Erdi et.al 1993, Li and Hopfield 1989, Skarda and Freeman 1987). It is believed that each inhaled odor has its own attractor — a stable limit cycle. The analysis of the canonical model (7.16) suggests that when an animal inhales a mixture of the odors, the appropriate limit cycles become stable so that 121 there is a one-to-one correspondence between the inhaled odors and the attractors. Similar results were obtained by studying another NN (Izhikevich and Malinetskii 1993), but the attractors there were chaotic. 7 .8 Quasi-Static Variation of Parameter b In the two preceding sections we studied the behavior of the canonical model (7.16) for fixed b. We varied the contrast c and saw that there were two attractors when the contrast was weak. The N N recognized one of the two presented images according to the initial conditions, not to dominance of one of them over the other. Of course, the attraction domain of the stronger image was bigger than that of the weaker one, but the network could not determine which image was dominant. One possibility to do it is to collect statistics over many trials for random initial conditions. There is another possibility of determining which image is dominant. We have to fix the contrast c and vary the bifurcation parameter b very slowly so that we can neglect the transient processes and assume that :L'(t) is arbitrarily close to attractor. Such variation of the parameter is called quasi-static. Recall that for b sufficiently small the canonical model (7.16) is globally asymp- totically stable, i.e. it has only one attractor a stable node. For large b, system (7.16) has many attractors. Suppose we start from small b. Then for any initial condition :1:(0) the activity :1:(t) approaches the unique attractor, and after some transient process :r(t) is in a small neighborhood of the attractor. Let us increase b quasi-statically. The activity :r(t) remains in the small neighborhood provided the attractor is hyperbolic. 122 Suppose the input is a combination of two previously memorized images £1 and {2. Suppose also that all conditions of the Reduction Lemma are satisfied, ,81 = 62 = fl and s = a1 + a2 are large enough. Then the qualitative behavior of the canonical model (7.16) is governed by the dynamical system (7.22). v'=c+bv—v3, vER, Suppose c = 0, i.e. the input images have equal strength. As we expected, (7.22) has only one attractor v = 0 for small b. Here small means b < —fl. If we increase b quasi—statically, the activity v(t) is always in a neighborhood of the origin provided b<—fl When b = —,B, the N N must choose one of the stable branches of the pitchfork bifurcation diagram depicted in Figure 7.9a. The nonzero contrast c is a perturbation (or an imperfection, see Golubitsky and Shaeffer (1979)) of the pitchfork bifurcation (see Figure 7.9b and c). No matter how small the contrast c is, the NN correctly chooses the corresponding branch provided the quasi-static increasing of b is slow enough. The case when the first image (I is dominant is depicted in Figure 7.9b. The stroboscopic presentation of the phenomenon is depicted in Figure 7.9d. One can speculate that when the internal parameter b crosses a bifurcation value, the N N ”thinks”. Choosing one of the stable branches could be called the ”decision making”. Prolonged staying near the unstable branch could be called the ”I don’t know” state. Thus, we speculate that some types of non-hyperbolic behavior exhibited by the canonical model near bifurcation points are intimately connected to such vague psychological processes as recognition and thinking. 123 stable Image l unstable stable stable 3 b c g l g 1 g l ‘ ,+.‘_ »z+ "I. ’ , o . ' ‘ .‘ o I, ‘ £2 “‘- g 2 I” \“ g 2 d Figure 7.9: Bifurcation diagrams for quasi-static variation of parameter b. a. The contrast c = 0. b. The first image is dominant. c. The second image is dominant. e d. Stroboscopic presentation of the phenomenon for b < —,B, b = —fl and b > -—/3, respectively. Chapter 8 Quasi-Static Bifurcations In this chapter we analyze the canonical models (3.24) and (3.32) for singularly per- turbed WCNNS in the special case a1=-~=an=a>0. An appropriate change of coordinates and taking the limit as e —> 0 transform the canonical models to { ::,- fgyi + Tiffz‘ + 1.2+ zy=1ciimi _1, ,n (8.1) and "'=_i iii.3 n 1'". ' {ylt—xy-l-TJ? TI+ZJ=1CJ$J 2:1,”_,n, (82) respectively. Here r,- E R describes input to the i-th relaxation neuron. We use the notation X = (.'1:1,...,:r.n)T E R"; Y = (y1,. . . ,yn)T E R" and C = (Cij) 6 RM". 8.1 Stability of the Equilibrium Note that the canonical models (8.1) and (8.2) always have a unique equilibrium point, namely, the origin (X,Y) = (0,0) 6 IR” x R”. lln this section we study the stability of the origin, as determined by the Jacobian matrix L=(REC ‘01”), (8.3) 124 125 where E is the unit n x n-matrix, C = (Cij), and 7'1 0 0 0 7‘2 O R: ; ; -. I . (8.4) 0 0 r,1 A matrix is called hyperbolic if all its eigenvalues have non-zero real parts. It is stable if all its eigenvalues have negative real parts. An eigenvalue with the largest (most positive) real part is the leading (dominant) eigenvalue. Theorem 48 The Jacobian matrix L E RZnXZn defined in {8.3) has the following properties: (a) L is non-singular. (b) L is stable if and only if R + C is stable. (c) L is hyperbolic if and only if R —l- C is hyperbolic. (d) IfR+C has a zero eigenvalue, then L has a pair ofpure imaginary eigenvalues {6) [fit + C has a pair of purely imaginary eigenvalues, then L has two pairs of pure imaginary eigenvalues. Proof. Suppose Lv 2 av for some ,a E C and a non-zero vector v 6 C2". We use notation where v1, v2 6 C”. Then (L—uE)v= ( R+%—ME :fi;)(::) _ (R+C—uE)v1—v2 _ 0 _ vl—pvg _ 0 ' 126 Thus, we have (R+C—pE)v1—v2 =0 vl—pv220 ' (8.5) (a) If p = 0, then the second equation implies v1 = 0. The first equation implies v2 = 0, therefore v = 0. This means that L cannot have zero eigenvalue, hence it is always non-singular. (b) From part (a) we know that a ¢ 0. Then v2 = [1'le and the eigenvector v of ._( ”‘ ) _ M , (12+ C — (a +u’1)E)v1 = 0 L has the form The first equation in (8.5) gives for some non-zero v1 6 R". Hence the matrix R+C—(u+p_l)E (8.6) is singular. Its eigenvalues are Ai—(p+a"l), i=1,...,k, where A1, . . . , Ah are those of R+C. Since it is singular, at least one of the eigenvalues of (8.6) should be zero. Hence a is a solution of one of the equations /\,-=p+p-l, i=1,...,k. (8.7) It is easy to check that if Re A,- < 0, then Rep < 0 and vice—versa. (c) Condition (8.7) reveals the relationship between eigenvalues of L and R + C. If L is non-hyperbolic, then from part(a) it follows that p is pure imaginary. Hence /\ is pure imaginary or zero. Conversely, if A is zero or pure imaginary, then a is pure imaginary. (d) and (e) follow from (8.7). D 127 Corollary 49 The equilibrium point loses its stability through a (possibly multiple) Andronov-Hopf bifurcation. We study the Andronov-Hopf bifurcation in Section 8.3. In particular, we are interested in when the bifurcation is subcritical or supercritical and how this depends on the input. Next, we analyze some neurophysiological consequences of Theorem 48. 8.2 Dale’s Principle and Synchronization One application of (3.4) is in modeling weakly connected networks of relaxation neural oscillators. In this case :1:,- and y,- denote rescaled activities of local populations of excitatory and inhibitory neurons, respectively. Each coefficient cij describes the strength of synaptic connections from .72]- to 33,-. We say that 1']- is depolarized when :1:,- > 0. Notice that if Cij > O (Cij < 0), then depolarization of a3]- facilitates (impedes) that of 33,-. We call such synapses excitatory (inhibitory). Copious neurophysiological data suggest that excitatory neurons have only excitatory synapses. This observation is usually referred to as Dale’s Principle (Dale 1935; Shepherd 1983). In our case it implies that ng Z 0 for all i and j (where Ci, 2 0 corresponds to absence of a synapse from 3:]- to iii). In this section we show that Dale’s principle imposes some restriction on local dynamics of the canonical models when the origin loses stability. In particular, we prove that the neural oscillators can synchronize. Theorem 50 Suppose the synaptic matrix C 2 (CU) satisfies Dale ’3 principle {cij Z 0 for alli and j). Then generically 128 o The equilibrium of (8.1) and {8.2) loses stability via an Andronov-Hopf bifur- cation. 0 The network ’3 local activity is in-phase synchronized, i.e. any two neural oscil- lators have nearly zero phase diflerence. Proof. Let and consider matrix A defined by A = R + C — pE. Dale’s principle ensures that A has non-negative entries. The Perron-Frobenius Theo- rem (Gantmacher 1959) applied to A shows that the leading eigenvalue A of A is real and non-negative and that the corresponding eigenvector u has only non-negative entries. Typically, A has multiplicity one and u has positive entries. The leading eigenvalue of R + C is A + p, which is real and also has multiplicity one. Theorem 48 guarantees that when the equilibrium loses stability, the Jacobian matrix L has only one pair of pure imaginary eigenvalues ii. Thus, the multiple Andronov-Hopf bifurcation is not typical in this sense. From the proof of Theorem 48 it follows that the corresponding eigenvectors of L (a), where u was defined above. Local dynamics near the equilibrium is described by ( :58 ) = ( in ) 2(t) + ( +2, ) at) + higher-order terms, (8.8) Where 2(t) E C is small. The activity of each neural oscillator has the form :1:,(t) __ _ Rez(t) . ( yi(t) ) _ 2U: ( Imz(t) ) + hlghef-Ol‘der terms, have the form 129 We can express the activity of the i-th oscillator through that of the j-th oscillator by ( :8; ) = Z—: ( :8; ) + higher-order terms, (8.9) where Ug/Uj > 0 because u has positive entries. Hence the i-th and j-th oscillators have zero phase difference (up to some order). D If the Andronov-Hopf bifurcation is supercritical, i.e. there is a birth of a stable limit cycle, then in-phase synchronization is asymptotic. More precisely, all local solutions have the form (8.8), where z(t) is small and periodic and the higher order terms remain sufficiently small as t ——> 00. If the Andronov-Hopf bifurcation is subcritical ( i.e. there is a death of an unstable limit cycle), then 2(t) in (8.8) grows as t ——> co and in-phase synchronization is only local. The higher-order terms in (8.8) can grow with time and after a while they can be significant. Remark 51 If Dale’s principle is not satisfied, then: 0 Multiple Andronov-Hopf bifurcation with exactly two pairs of pure imaginary eigenvalues is also generic. This follows from Theorem 48, part (e). 0 Either in-phase or anti-phase synchronization is possible, i.e. the phase differ- ence between any two oscillators could be nearly 0 or 7r. This follows from (8.9) because u,- / u j is a scalar and could be positive or negative. Remark 52 There have been many studies of existence and stability of in-phase and anti-phase solutions in linearly coupled relaxation oscillators. See, for example Belair and Holmes (1984), Storti and Rand (1986), Somers and Kopell (1993), Kopell and Somers (1995) and Mirollo and Strogatz (1990). Our results complement and extend 130 those of these authors since they perform global analysis of two coupled oscillators, while we perform local analysis of n coupled oscillators. Remark 53 An important difference between weakly connected networks of relax- ation and non-relaxation oscillators (Hoppensteadt and Izhikevich 1995b) is that in the former the phase differences are usually either 0 or 7r, but in the latter they may assume arbitrary values. 8.3 Further Analysis of the Andronov-Hopf Bifur- cation In this section we study the Andronov—Hopf bifurcation in canonical models when R + C has a simple zero eigenvalue. Our major goal is to determine when it is subcritical or supercritical and how this depends on the matrix C and the inputs T1,...,T'n. We begin with an analysis of the canonical model (8.2) 0i=i1, i=1,...,n I __ . . . 3 2:" . {13,-— —yi "l' rim: + 01$,“ "l‘ i=1 C2313] I __ yr — 5171' because it is simpler than that of (8.1). Let v1 = (v11, . . . , v1")T E R” be the normalized eigenvector of R + C correspond- ing to the zero eigenvalue. Let wl = (1011,. . .,w1n) E R” be dual to v1, i.e. n 101221: E wuvu =1 i=1 and wl is orthogonal to the other (generalized) eigenvectors of R + C. Theorem 54 If the parameter a defined by a 3 n ZZinlivTi (8'10) i=1 131 is positive (negative), then the Andronov-Hopf bifurcation in {8.2) is subcritical (su- percritical). Proof of the theorem is given in Section 8.4. In neural network studies it is frequently assumed that synapses are modified according to Hebbian learning rule. This implies that the synaptic matrix C is sym- metric. It is also reasonable to consider a network of approximately similar oscillators. They can have different quantitative features, but their qualitative behavior should be comparable. These two observations motivate the following result Corollary 55 Suppose that 1. The synaptic matrix C = (ng) is symmetric, and 2. All oscillators have the same type, i.e. 0'1="'=0'n=0'. Ifo : +1 {0 = —1), then the Andronov-HOpf bifurcation in (8.2) is always subcritical (supercritical). Proof. lf C is symmetric, then so is R + C. Since every symmetric matrix has orthogonal eigenvectors, we have v1 = wI. Therefore (8.10) can be rewritten as n 3 E : 4 3:1 and its sign is determined by a. C1 Remark 56 We can relax assumption 1 in the corollary simply by requiring that v1 be orthogonal to the other (generalized) eigenvectors. 132 Analysis of the canonical model (8.1) i=1,...,n { x: = —yi + T‘ifvi + 13.2 + 221:1 Cij‘rj I yi=$i is more complicated than that of (8.2). Since we are interested mostly in the case when the synaptic matrix C is symmetric, we assume at the very start that it is. Theorem 57 The Andronov-Hopf bifurcation in (8.1) for a symmetric matrix C = (cij) is always subcritical. Proof of the theorem is given in Section 8.4. 8.4 Proofs of Theorems 54 and 57 The canonical models (8.1) and (8.2) can be written concisely in the form Z'=F(Z), (8.11) where Z = (X,Y) E R2", F : R2" —> R2" and F(0) = 0. The Jacobian matrix L 2 BF at the equilibrium is given by (8.3). From Theorem 48 it follows that the equilibrium Z = 0 loses stability via an Andronov-Hopf bifurcation when R + C has one simple zero eigenvalue and the other eigenvalues lie in the left half-plane. Let v1 be the eigenvector of R + C corresponding to the zero eigenvalue. Then L has a pair of pure imaginary eigenvalues :l:i with the corresponding eigenvectors ( £21m ) . (8.12) To determine the type of bifurcation that occurs we restrict (8.11) to the center manifold, which is tangent to the center subspace: Ec=spanlu>ri={<(>x+<:>mw) 133 where a: and y can be treated as coordinates on EC. On the manifold (8.11) has the normal form '1": _y+f($9y) { y’ = Iv+g(:v,y), (8'14) where f and 9 denote the non-linear terms in x and y. Then the Andronov-Hopf bifurcation is subcritical (supercritical) if the parameter a: ll—6(f;cxx+fxyy+gxry+gyyy+fxy(fxx+fyy) (815) —ga:y(gx:c + gyy) "' fxxgra: + fyygyy) is positive (negative). For derivation of (8.15) see Guckenheimer and Holmes (1983). Note that during the center manifold reduction it suffices to compute f and 9 only up to third order terms in a: and y. Our treatment of the center manifold reduction is based on that of Iooss and Adelmeyer (1992). To perform the reduction we must introduce some objects: Let E8 C R2" be the stable subspace spanned by the (generalized) eigenvectors of L corresponding to the eigenvalues of L having negative real parts. Thus, we have the splitting R2" = EC EB E3. Let 7rC : 1R2" ——> EC and 7r, : R2” —+ E3 be projectors such that ker 7rC = E3 and ker if, 2 EC. If wl is a dual vector to v1, then Tl'c is given by we = ( ”1““ ), (8.16) vlwl where vlwl denotes the n x n-matrix defined by the tensor product v1 (8) wl. Note also that 7rc and 7r, commute with L. The Center Manifold Theorem ensures that there is a mapping ‘1! : EC —+ E3 with \Il(0) = 0 and D‘IJ(0) = 0 134 such that the manifold M defined by M = {v+\Il(v) | v6 EC} is invariant and locally attractive. The reduced system has the form v' = 7rcF(v + ‘Il(v)) (8.17) and (8.14) is just (8.17) written in local coordinates on EC. The initial portion of Taylor’s expansion of the function \Il(v), which defines the center manifold, can be determined from the equation D\§|?(v)vI = 7r,F(v + \Il(v)), (8.18) where v’ is defined in (8.17). We do not need (8.18) for proving Theorem 54 because the canonical model (8.2) does not have quadratic terms in :r and y. But we use (8.18) in proof of Theorem 57. 8.4.1 Proof of Theorem 54 v=(%)s+(£)a mm) Since 7rC and L commute, 7rC\Il = 0 and 7er = v for v E EC, we have From (8.13) we see that 7rCL(v + \IJ(v)) = L7rC(v + \Il(v)) 2 Lu. (8.20) Therefore the right-hand side of (8.17) is L ( v1 )3: + L ( 0 ) y + 7Tc( (01(v11r)3,...,on(v1n.r)3) + h.o.t. ) 0 ’01 0 _ 0 —v1 v1r3 221:1 wuoivi- + h.o.t. —(..)w+( o lml o - 135 Multiplying by (w1,0) and (0,w1) gives I {x’ - —y +x3Z?=lo,-w1gvf‘,-+h.o.t. y = 2:. The parameter a defined in (8.15) is n 3 E : 3 a = — Ug’wlgvli. 4 . i=1 8.4.2 Proof of Theorem 57 If the connection matrix C is symmetric, then so is R + C. Therefore, R + C has n orthogonal eigenvectors v1, . . . , v" forming a basis for IR". As before, the vector v1 corresponds to the zero eigenvalue of R + C and v2, . . . ,vn correspond to the other eigenvalues A2, . . .,An, which are negative. Using the proof of Theorem 48 we can define the stable subspace of L by s __ 02 0 Un 0 E spell o llvzlwl 0 Had} n T _vv. 71.3:(z31u2kk). 22:2 Ukv; To determine the parameter a we must find the quadratic terms of \Il(v). Let ((3)..(gWt;)..(Z)..(,)..... where and 71 TI Pi = E Pikvk and (Ii: E (Iikvk k=2 k=2 for i = 1,2,3. Since :13’ = —y + h.o.t. y’ = a: + h.o.t., 136 the left-hand side of (8.18) is 2P3 - 2Pl ) 2 2 P2 ) Dlll ' = - h. .t. 8.21 (v)v :ry( 2% _ 2q1 + (at y ) Q2 + o ( ) Since 7r3 commutes with L, and 7r3v = 0, we have “314(1) + ‘I’(v)) = L7r3(v + \Il(v)) = L‘I’(v) = 1:2 Zk=2 )‘kzkplk — <11 ) 1 n n 8.22 +a:y ( 218:2 “2::sz “ C12 ) + yz ( Zkz2 M12193}: — (I3 ) + h.o.t. ( ) Thus, the right—hand side of (8.18) is 2 n T‘ . 2. “1(8) + 7r, ( ("161): ) = L\Il(v)+:172( EH ”k %3.=1 ”’“Ul' ) (8.23) Combining like-terms in (8.21),(8.22) and (8.23) and considering projections on each vk, we obtain the system ( sz ) ___ ( Akplk — c11k ) + 22;, vkivf,‘ (12k Plk 0 2(P3k ) _2(P1k ) = ( Akpzk—Cm (13k (11}: P21: _ ( P21: ) = Aszk — C131: C121: P31: ’ which is be solved for p,k,q,-k, i = 1,2,3; k = 2,. . . ,n. Below we will use only the values of plk and p3k, which are given by 2A n Pik = —P3k = ___L E: vkivfi- (8.24) 1 137 Now let us determine the reduced system (8.17). Its left—hand side is given by (8.19). Using (8.20), we can find its right-hand side: it is 01 0 L( 0 )x+L(vl)y+ (viii? + 22:2 vied-732p“: + (Bypzk + y2P3k) + h.o.t.) _ C 0 _ =(fl).+(-;l)y+ v1 221:1 v1,- (vf,-:r2 + 2v1ia: 22:2 vk;(:r2p1k + .Typgk + y2p3k)) + h.o.t. ) 0 . 2 i Multiplying by (vir,0) and (0, vlT) gives :r’ = -y + 932 2le vi.- + 933 EL 22:2 2vfivkip1k+ :cy2 23:1 22:2 2vfivkip3k + h.o.t. y' = 2:. It follows from (8.15) that TI. 11 Z vf, vki(3plk + P3k)- 1:1 k=2 a :: J>|hd Using (8.24) we see that since each Ak < 0. Cl Chapter 9 Non-Hyperbolic Neural Networks Let us return to the question how the canonical models might perform pattern recog- nition tasks. First we outline the main idea, then we present rigorous mathematical considerations. Consider the canonical models for multiple subcritical pitchfork bifurcation 51::- = r,:c,-+a:?+Zc,-jrrj, (9.1) i=1 for multiple subcritical Andronov-Hopf bifurcation z; = TiZ,‘ + dizilzilz + Z CiJ'ZJ', Re d,‘ > 0, (9.2) i=1 for multiple quasi-static saddle—node bifurcation .1: —yi + 77$; + £17? + 231:1 Ciflj y'- z :0 , (9.3) and pitchfork bifurcation 171' = “311+ Tami + 13;” + 221:1 Cij-Tj y’- = :r- . (9.4) Notice that these models have an equilibrium — the origin, for any choice of parame- ters. Stability of the origin is determined by the Jacobian matrix R + C, where 7‘1 0 0 R: 0 7:2 0 , 0 0 r,, 139 and C = (Cij) is the synaptic matrix. Both R and C are complex-valued for (9.2) and real valued for the other canonical models. The origin is stable if all eigenvalues of R + C have negative real parts and is unstable otherwise. The origin loses its stability via subcritical pitchfork bifurcation (for (9.1)) or Andronov-Hopf bifurcation (for the other models). In some neighborhood of the bi- furcation point the direction along the eigenvector corresponding to leading eigenvalue of R + C becomes unstable and the activity vector moves along this direction. After a while it leaves a small neighborhood of the origin and an observer notices some macroscopic changes in dynamics of the canonical models (see Figure 1.8). Thus the local event — loss of stability by the origin, produces a global effect. This is the key- idea of the non-hyperbolic N N approach, which we described in Section 1.4. Below we explain in detail the idea outlined above. 9.1 Problem 1 Given an input vector r" = (rf, . . . ,rfi) G R" we can construct diagonal matrix R" by r1 0 0 R” = . 0 rlj 0 0 0 r,,: We use the following notation: Ak E R denotes the leading eigenvalue of the matrix Rk + C, i.e. the eigenvalue with the largest real part. The vector uk 6 1R" denotes an eigenvector of R" + C corresponding to Ak, if the leading eigenvalue is unique and simple, i.e. it has multiplicity one. Suppose we are given a set of input vectors {r1, . . . ,r'”} C R" and a set of key patterns {v1, . . . , vm} C R" to be memorized. Consider the following problem 140 PROBLEM I. Find a matrix C E Rn“ such that for all matrices R” + C, k = 1, ..,m the leading eigenvalues A)c = 0 are simple and u" = v". Suppose that given rl,..,rm and v1,..,v"‘ there is such a matrix C. Then the canonical models can perform the pattern recognition tasks in the sense described next: We say that the k-th input r” from external receptors is given if the parameters r1, . . . , r,, in the canonical models are given by rizrf-l-p, i=1,...,n, (9.5) where p E R is a scalar bifurcation parameter. Then, for p < 0 the equilibrium point of the canonical models (the origin) is stable; for p = 0 there is a bifurcation; and for p > 0 the equilibrium is unstable. For the canonical model (9.1) the equilibrium loses stability through subcritical pitchfork bifurcation (see Section 7.5). For small positive p the canonical model dynamics approach the center manifold, which is tangent to the center subspace E 6 IR" defined by E = span {uk} , where u’c is the eigenvector of Rk + C corresponding to the leading eigenvalue Ak. k 1: According to Problem 1, the vector u coincides with the memorized pattern v . Thus, when the k-th input pattern is given, the activity of the canonical models is close to the linear subspace E which is determined by the memorized pattern v". A rough sketch of local dynamics is depicted in Figure 9.1 In the canonical models (9.2), (9.3) and (9.4) the origin loses stability through Andronov-Hopf bifurcations (see Remark 41 and Section 8.3). The center manifold 141 \.. o———OO""‘O OHOt—O rl r2 r “ v2 0 /O\ V’ 0/0 0 0 Figure 9.1: Depending upon the input (r1 or 2) the center manifold is tangent to r the corresponding memorized vectors (v1 or v2) is tangent to the center subspace defined by E=.p..{(z;;),(;.)}. (9.6) If the Andronov—Hopf bifurcation is supercritical for all inputs rk, then the new-born stable limit cycles lie close to the corresponding center subspaces defined by (9.6). Thus, for each input rk there is an attractor — stable limit cycle, which has location prescribed by the memorized pattern v". This was observed in the olfactory bulb (see Section 1.4.1). The only difference is that the limit cycles in the canonical models have small radii. Finally, we note that the supercritical Andronov-Hopf bifurcations can be observed in the canonical model (8.2) for o,- = —1, i = 1,. . . ,n. Therefore, (8.2) can function like a GAS-type neural network, but there are two steady states: an unstable equilibrium (the origin) and a stable limit cycle. If the Andronov—Hopf bifurcations is subcritical for all inputs rk, then the canonical models are like non-hyperbolic neural networks. Indeed, after the bifurcation, which is local, the dynamics leave some neighborhood of the equilibrium point along the direction determined by one of the memorized pattern v1, . . . , vm. For example, suppose the network recognizes v1, and that |vi| >> |v%|. Then 142 the first neural oscillator oscillates with the amplitude much larger than that of the second one. If the Andronov-Hopf bifurcation is supercritical, then the attractor has the same property. If the bifurcation is subcritical, then these oscillations are observed locally and might persist globally. In both cases, an experimenter on the olfactory bulb discovers that there is a spatial pattern of oscillations: various sites of the olfactory bulb oscillate with the same frequency but with different amplitudes (Skarda and Freeman 1987). In that case the results predicted by our analysis of the canonical models agree with the neurophysiological experiments. 9.2 Problems 2 and 3 To the best of our knowledge, Problem 1 is still unresolved. We do not know any general method that allows one to construct such synaptic matrix C, although for some special cases, such as m = 1, the construction is trivial. Below we present alternative problems that can be easier to resolve. PROBLEM 2. Find matrix a C such that each Ak is simple and real and uk = vk for all 1:. Since we do not require that Ah 2 0, the bifurcations occur for p = —Ak, where p is defined in (9.5). PROBLEM 3. Find matrix a C such that the leading eigenvalues A)c of R,“ + k C, k = 1, .., m, are real and simple and the corresponding eigenvectors u are pairwise orthogonal. Note that we do not require that u” : vk here. The requirement that all u,c be orthogonal means that the response of the neural network on various inputs is as different as possible, even when the inputs to be memorized are similar (like a cat and a dog). Chapter 10 Synaptic Organizations of the Brain Neurophysiological studies of various brain structures (see Rakic 1976 and Shepherd 1976) show that there is a pattern of local synaptic circuitry in many parts of the brain: Local populations of excitatory and inhibitory neurons have extensive and strong synaptic connections between each other so that action potentials generated by the former excite the latter, which in turn, reciprocally inhibit the former (see Figure 10.1). They can be motoneurons and Renshaw interneurons in spinal cord; Mitral and granule cells in olfactory bulb; Pyramidal cells and thalamic interneurons in corticothalamic system, etc. Such pairs of interacting excitatory and inhibitory populations of neurons can also be found in cerebellum, hippocampus, olfactory cortex and neocortex (Shepherd 1976). This is one of the basic mechanisms for the generation of periodic activity in the brain. Such a pair is called a neural oscillator. The neural oscillators within one brain structure can be connected into a network because the excitatory (and sometimes inhibitory) neurons can have synaptic contacts with other, distant, neurons. For example, in the olfactory bulb the mitral cells have contacts with other mitral cells (see Figure 10.2), whereas the granule cells apparently do not make any distant contacts, they do not even have axons. Their only purpose 143 144 .., Inhibitory ‘ ; Neuron Figure 10.1: Schematic representation of the neural oscillator. It consists of excitatory (white) and inhibitory (shaded) populations of neurons. For simplicity only one neuron from each population is pictured. White arrows denote excitatory synaptic connections, black arrows denote inhibitory synaptic connections Olfactory Bulb Figure 10.2: The neural oscillators (dotted boxes) are connected into a network. The mitral cell makes contacts with other mitral cells and may have contacts with other granule cells 145 is to provide a reciprocal dendro-dendritic inhibition for the mitral cells. Sometimes inhibitory neurons can also have long axons; for example, the periglomerular cells in the olfactory bulb. Though on the local level all neural oscillators appear to be similar, the type of connections between them may differ. In this case we say that the networks of such oscillators have different synaptic organization. For instance, in Figure 10.2 the contacts between mitral cells and distant granule cells (dashed line) might or might not exist. These cases correspond to various synaptic organizations and, hence, to various dynamical properties of the network. The notion of synaptic organization is closely related to the notion of anatomy of the brain. Thus, in this chapter we study relationships between anatomy and functions of the brain. For example, we show that some synaptic organizations allow the network to memorize time delays, or phase deviation information, whereas the others do not allow such a possibility. For the sake of clarity we always depict only two neural oscillators and the synaptic connections only in one direction, as in Figure 10.2. It is implicitly assumed that the network consists of many neural oscillators and the synaptic connections of the same type exist between any two oscillators and in all directions. We have already used Dale’s principle (Dale 1935; Shepherd 1983) in Chapter 8. Recall that it says: The excitatory neurons may have only excitatory synaptic connections with other neurons and inhibitory neurons may have only inhibitory synaptic connections. This principle imposes some restrictions on possible synaptic organization of the networks and we study how it affects the dynamical properties of the networks. There is neurophysiological data about the importance of oscillations and chaos in the brain (Eckhorn et a1. 1988, Gray 1994 and Skarda and Freeman 1987), and there 146 have been many studies of the role of oscillations in the processing of information by the brain. Most of them devoted to study of synchronization phenomenon (Aronson et al. 1990, Hoppensteadt 1989, Kazanovich and Borisyuk 1994, von der Malsburg and Buhmann 1992 and Schuster and Wagner 1990). Pattern memorization and recognition by oscillatory neural networks have been studied, for example, by Baird (1986), Erdi et al. (1993) and Li and Hopfield (1989). To the best of our knowledge, there have been no attempts to connect dynamical properties, such as the possibility to learn a pattern, with the synaptic organization of a network. We think that such attempts could reveal the relationship between function and structure of the real brain. 10.1 Neural Oscillators A neural oscillator is described by a dynamical system of the form { it = f(:r,y,/\) Q =g(~’vay,/\), where x, y E R are the activity of excitatory and inhibitory neurons, respectively. A typical example of the neural oscillator is Wilson and Cowan model (1972,1973). In this chapter we assume that the activities :1: and y are one-dimensional variables. This is a technical assumption made to simplify our computations and to allow us to use Dale’s principle. We call the oscillator neural to emphasize its connections with neuroscience. Thus, using Dale’s principle we can gain some information about f and g, viz. (9f _ > _ 8x ‘0 and By S0’ for all x, y and A. The first inequality means that x excites y. The second inequality means that y inhibits x. 147 Let (x:, y.) E R2 denote the activity of i-th neural oscillator for i = 1, . . . , n. A weakly connected network of neural oscillators is a dynamical system of the form { (15,“: fi(xivyi9 A) +5pi($layla' ° 'a'rnay'nae) (101) yi = gi(xiiyi7)‘) + €q:(:c1,y1, - ° - ,xn’y‘fhe) satisfying for all x,y, A and i 753' Dale’s principle, namely 98”, 8133‘ _ (1%; 8113]" 3p: aqi >0, —S0 and —S0, 10.2 " 0w 3311 ( ) where p,, q,- : Rh“ —> IR are functions that represent synaptic connections from the whole network onto the i-th neural oscillator. One can think of (10.1) with (10.2) as being a generalization of Wilson’s-Cowan’s model of an oscillatory neural network (Wilson and Cowan 1972,1973). Remark 58 We do not demand that the connections within an oscillator are weak. For our analysis we need weakness of connections between neural oscillators. In order to apply the WCN N theory that we developed in Chapter 2 we have to impose the additional assumption that each neural oscillator is not a pacemaker, i.e. that each equation in the uncoupled (e = 0) system 33: = fi($i,y£,/\) - . =1,..., , 10.3 { yi : gi($iayii’\) 3 n ( ) has a stable equilibrium point, say (0, 0) for A = 0. So we have f.-(0,0,0) - 0 0 0 0 gz(, ) 0. , Throughout this chapter we denote the J acobian matrix for i-th neural oscillator L: (11 a2 :8(fiagi): (ii—i (ii—if i a3 a4 ,- C($i,yi) 3%:- Qfll , 3y: by 148 strong connections ........... weak connections —{> excitatory synapse ——> inhibitory synapse Figure 10.3: A network of two neural oscillators. Open boxes are local populations of excitatory neurons, shaded circles are local populations of inhibitory neurons. The real numbers a1, a2, a3, (14 are entries of the Jacobian matrix of i-th neural oscillator. The real numbers .31, .92, s3, s4 denote the strength of synaptic connections where all derivatives are evaluated at the equilibrium point (0,0). The matrices L1, . . . , Ln could differ because we do not assume that the neural oscillators are iden- tical. Sometimes we denote entries of the Jacobian matrices by a,1,a,2,a,3,ag4 to stress that they belong to L. The matrices 3 .‘ a .‘ Sn 2 ( SI 82 ) : 3(Pi,q.-) : 35; 3% (10-4) '53 8“ ij 0(xj,yj) 37% 55;" denote rescaled synaptic connections from j—th to i-th neural oscillator (see Figure 10.3) because the actual synaptic connections have order 5 and look like ESQ-j. Note that according to Dale’s principle (10.2) the synaptic matrices have signature + _ + _ . 10.1.1 Multiple Andronov-Hopf Bifurcation Recall that the Fundamental Theorem of WCNN Theory requires that (10.1) be near a multiple bifurcation, or otherwise its dynamics is not interesting from neuro- computational point of view. It is reasonable to require that the bifurcation be Andronov-Hopf. 149 The equilibrium point (0, 0) corresponds to an Andronov-Hopf bifurcation if tr L.- = a1+ a4 = 0 and det L,- = a1a4 — am; > 0. Recall that we associate with each neural oscillator its natural frequency 9.- : vdet L.- = x/anaa — ai2ai3. As usual, we assume that A = Me) = 0 + 5A1+ 0(52) (10.5) for some A1 E A. Thus, all neural oscillators are e-close to the Andronov-Hopf bifurcation and their natural frequencies are 9i + Ewi for some w; E R. Recall that according to Corollary 12 we consider networks of oscillators having equal natural frequencies, i.e. Q,- = Q,- = (2. Recall that the WCNN (10.1) near multiple Andronov—Hopf bifurcation is gov- erned by the canonical model (see Theorem 11) 2:" = bizi + dizi Zi'2 + Z 61'ij i?“ We restrict our attention to the case when all coefficients (1, are real and negative. This implies that the frequency of oscillators does not depend upon their amplitudes. The condition d,- < 0 implies that the Andronov-Hopf bifurcation for each oscillator is supercritical. This corresponds to a birth of a stable limit cycle. Without loss of generality we can take (I, = —1. Thus, we study the dynamical system 2;, 2 (pg + int-)2,- — ZiIZglz + 2 Ciij (10.6) #i for 2 = 1, ..,n. 150 Synaptic Coefficients In this chapter we need the exact relationship between complex-valued synaptic coef— ficients Cij and actual synaptic connections ng. This is important for neurobiological interpretations of the theory developed below. It will allow us to interpret all results obtained by studying (10.6) in terms of the original WCNN (10.1), i.e. in terms of excitatory and inhibitory populations of neurons and interactions between them. Lemma 59 The relationship between cij and ng is given by l in ia l Cij : §(1+fi4 , —-fii)i5;j( a Ii“) . (10.7) i (12 Proof. The eigenvectors of L,- are 1 1 vi:(a|in) and fii:(a—i9)' 0.2 i 02 2° The dual vectors are _ 1 .a4 .a2) w" 2(1+‘n’ "Q .- and w-=1(1— iE ia2) i 2 Q, Q i. The result follows from expressions (2. 26) and (10. 4).CJ 10.1.2 Type A and B Neural Oscillators Recall that according to Dale’s principle (12 < 0 and a3 > 0. Hence, each Jacobian matrix L generically either has signature (1:1) .. (.13.)- In first case we say that the neural oscillator is of type A, in the second case type B. 151 y y /0x \ x L/Sx 3x a b //x? Figure 10.4: Differences in dynamic behavior of type A and B neural oscillators. See text Consider the differences between these types. Suppose the neural oscillator is at the equilibrium point (0, 0) and we apply a short impulse to its excitatory neuron so that its new activity becomes 0 + 63:, where rim is small. Since the equilibrium (0, 0) is stable and the Andronov-Hopf bifurcation is non-degenerate, the activity vector (a:(t),y(t)) approaches (0,0). If the neural oscillator is of type A, the relaxation to (0, 0) has the following form: The excitatory neuron increases its activity a: further increasing the inhibitory neuron activity y. After a while, y reciprocally inhibits :1: and both neuron activities decrease (see Figure 10.4a). If the neural oscillator is of type B, then there is no such initial amplified response of excitatory neuron activity, but there is an increasing of inhibitory neuron activity (see Figure 10.4b). We see that in type A neural oscillators it is possible for excitatory and inhibitory neurons to reach the peak of activity approximately simultaneously, whereas for type B neural oscillators it is not. The differences between type A and B neural oscillators were not essential for our mathematical analysis so far. Nevertheless, they are crucial in the next section and in Section 10.6 where we show that type A neural oscillators have some interesting properties: In some sense they are ”smarter”. 152 10.2 Dale’s Principle and Connectivity There are some neurophysiological implications of formula (10.7). We first observe that if the WCNN (10.1) is uncoupled then so is the canonical model (10.6). Indeed, if p.- = q.- = 0 for all i, then 92.: £13; a=(%’%)=(00) 5;;- 53: 0 0 and, hence, from (10.7) Cij = 0 for all i and j. It turns out that the converse is not true. There could be nonzero connection functions p,- and q,- such that the synaptic coefficient Cij = 0. This means that although two neural oscillators can be physically connected (ng 75 O), the synaptic connections between them are not effective because the canonical model is uncoupled (c,,- = O). The existence of such a phenomenon follows from the next theorem. Theorem 60 If the i-th and the j-th neural oscillators are of type A, then there are nonzero synaptic configurations 5},- between the i-th and j-th neural oscillators such that Ci, 2 0. Such configurations can be found that satisfy Dale ’5 principle. If the neural oscillators are of type B, then such synaptic configurations always violate Dale ’8 principle. Proof. Consider formula ( 7.10) for c,-J-. After rearrangement we see that it is equivalent to Cij = ’0181 + ’0282 + ”U383 + U484, (10.8) where v 1 +.ai4 : — 1—, 1 2 2o 2 aj4 — 0M .aj4ai4 + Q 2aJ-2 I 20130 , 'L’z = 153 n'vz "Q A"? v, MV3 “‘9 -21 V1 (1 b Figure 10.5: Complex numbers v1, -v2, v3, --v.; as vectors on the complex plane. For simplicity we depict —v2 with zero real part. a. Type A neural oscillator. b. Type B neural oscillator _ .012 ’03 — —l—2fl, v _ an iaj4ai2 4 _ __ _ 2(1j2 2aj2§l are four complex numbers. Consider v1,v2,v3 and 124 as vectors in R2. Obviously, they are linearly dependent. Hence the equation 0 = vlsl + v2.92 + U383 + v4s4, (10.9) has nontrivial solutions. In order to satisfy Dale’s principle 51 and 33 must be non-negative and 32 and s4 non-positive. Thus we must find solutions for 0 = "0181+(—1)2)|82|+ U333 + (—U4)|84| (10.10) with non—negative coefficients sl, l82l, 33 and |s4|. If the neural oscillators are of type B, then all of the complex numbers v1, —v2, v3 and —v4 have positive imaginary parts (see Figure 10.5b). All their non-trivial linear combinations with non-negative coefficients also have positive imaginary parts. Thus equation (10.9) and Dale’s principle cannot be satisfied simultaneously. If the neural oscillators are of type A, then '01, —v2, v3, —v4 have imaginary parts with different signs (see Figure 10.5a), and it is easy to see that (10.10) can be satisfied. Indeed, one can take a linear combination of 121 and —v4 so that the imaginary part )3 CE 3 '3; £1 e/e o\o e’xp Figure 10.6: Synaptic configurations that can exhibit the phenomenon described in Theorem 60. Open boxes depict excitatory neurons and shaded circles depict inhibitory neurons. A vertical pair of excitatory and inhibitory neurons is one neural oscillator (projection on vertical axis) of vlsl + (—v4)|s4| is zero and add v3 or -—v2 or both to kill the real part (projection on horizontal axis). Thus, equation (10.9) and Dale’s principle can be satisfied simultaneously. Cl Theorem 60 could be a revelation for neuroscientists. Indeed, it is probably diffi- cult to accept that the existence of synaptic connections between neurons from two different neural oscillators does not necessarily imply that the oscillators interact. We have already seen this for oscillators that have different natural frequencies fl (see Corollary 12). In Theorem 60 the neural oscillators could be identical; they can even act as if they were synchronized, but the synaptic connections between them are not responsible for that. If we somehow suppress one of them, the other one will not ”feel” it. Its amplitude and phase will be unchanged. It should be noted that the effect described above is a synergetic phenomenon, and can be observed on the level of interacting neural oscillators, but possibly not on the level of interacting neurons. A direct consequence of the preceding proof is the following Corollary 61 The phenomenon described in Theorem 60 can be observed only in networks with the synaptic configurations depicted in Figure 10.6. Proof. From Figure 10.5 it follows that (10.10) has non—trivial solutions with non- negative coefficients only if .91 75 0, s4 # 0 and either 32 ;£ 0 or 33 75 0. The result follows from Figure 10.3. CI 155 In all of the synaptic configurations the inhibitory neurons should be long-axon neurons capable of forming long-distance synaptic contacts (i.e. contacts from one neural oscillator to another one), which is rare in the brain. All of the brain structures we studied are not of the type depicted in Figure 10.6. 10.3 Classification of Synaptic Organizations According to Lemma 39 the values wij =Arg ng decode phase information. Let us determine possible values of the natural phase differences 1b,,- for various synaptic configurations Sij satisfying Dale’s principle. It is easy to do this using (10.8) and the vectors from Figure 10.5. Indeed, if for example, Si,- has the form ..— 810 Sin—(33 0), where 31 and 33 are some positive numbers, then Cij = 0181+ v3.93, and, hence, possible values of Cu are linear combinations of v1 and v3 with positive coefficients. Values of ng may be anywhere between lines spanned through v1 and v3 in Figure 10.5a or b. Thus, d’ij is an angle between Arg v1 and 7r/2 as shown in Figure 10.7. Complete classification of ng for all 3.5 is given in Figure 10.7. We see that type A neural oscillators are able to reproduce the entire range of natural phase differences if they are suitably connected, while type B oscillators cannot. Indeed, all possible values 1b,]- =Arg ng for type B oscillators are between Arg v1 and Arg (—v4). Type B neural oscillators cannot even have natural phase difference W = 0, whereas the type A oscillators can. 156 $3337? 3:??? 34.03??? 3433‘: $8 $3 (‘2‘ V838 OO 0"0 O‘O H H II a b Figure 10.7: Possible values of synaptic connections ng for different synaptic con- figurations Si, satisfying Dale’s principle. For synaptic configurations that are not explicitly depicted the possible values of c,~j may occupy all shaded areas. a. Type A neural oscillator. b. Type B neural oscillator Using the classification in Figure 10.7 we can solve a number of problems. Knowing the phase shift between two neural oscillators we can find possible synaptic configu— rations that can produce the shift; Knowing changes in synapses we can find changes in phase shifts and vice versa, etc. We will use this classification below when we analyze possible synaptic organiza- tions from the point of view of memorization of phase information. 157 10.4 Learning Dynamics Much is not known about learning in the human brain, but our major hypotheses about the learning dynamics appear to be consistent with observations. We assume that 0 Learning results from modifying synaptic connections between neurons (Hebb 1949). 0 Learning is local, i.e. the modification depends upon activities of pre— and post- synaptic neurons and does not depend upon activities of the other neurons. 0 The modification of synapses is slow compared with characteristic times of neu- ron dynamics. 0 If either pre— or post-synaptic neurons or both are silent, then no synaptic changes take place except for exponential decay, which corresponds to forget- ting. These assumptions in terms of the WCN N (10.1) have the following implications: The first hypothesis states that learning is described by modification of the matrices 5,-3- defined in (10.4). Recall that the actual synaptic connections have order 5. We denote them by Mi. Thus 1.03 “)4 ng = ("(1)1 102 ) 265,5. (10.11) ii The second hypothesis says that for fixed i and j the entries of Wij are modified according to equations of the form II zl-z-u 1(wlamia‘rj), 2(w2, 13:“,ij 10.12 1.03 — _3(w3 yia‘r $3) ( ) 104’ — _h4(w4ayiayj ? 3 158 where (x,,y,-) are local coordinates at the origin. We introduce the ”slow” time T 2 at to account for the third hypothesis. We say that a neural oscillator is silent if its activity is at an equilibrium point, i.e. it does not oscillate. Then the fourth hypothesis says that ~ h(w,0,y) = h(w,a:,0) = h(w,0,0) = h(w) = —'yw+6w2+..., for all :r and y, so that h has the form h(w,a:,y) = —7w+0:ry+61w.r +62wy+6w2 +... (10.13) It follows from (10.11) that a synaptic coefficient w is of order 5. From (2.24) we know that the activities as and y are of order fl. After rescaling by w —> es, .7: —> fix, y —+ fly, we obtain the learning rule 8' = —’ys + dry + 0(x/E), (10.14) which we refer to as the Hebbian synaptic modification rule. Note that although we consider general functions h, after the rescaling only two constants 7 and 0 are significant to leading order. They are the rate of memory fading and the rate of synaptic plasticity, respectively. We assume that the fading rate *7 is positive and the same for all synapses. The plasticity rates can differ for different synapses. To distinguish them we write d,,-k for i,j=1,...,n; k=1,2,3,4. In order to understand how learning influences the dynamics of the canonical model (10.6) we must calculate the changes in c;,-. Lemma 62 If all the conditions listed above are satisfied, then I Cij = —’YCij + kij2zi5j + kijsiizj, (10.15) 159 where kiJQ— — %(0 £13 + 0151+ 03(01'3'4 + 0.32) + %L(6gj3 — 9351+ ail-(00“; — 01j2))) 2 kiJ3— "" % (1+ Q :) [0ij1_6ij3 + (0 2+ifl)( 0ij2— 0011)] (10.16) 01’: _aJ: Proof. Using (10.7) we see that ..’_l 1+.ifl _iiz. 81’ 32’ 1. C3] — 2 Q i Q i 33’ 34’ ij aid-210 I. J From (10.11) we have Sijk’ = “wijk 9 k =1923334 for all i and j. So (10.12) and (10.13) imply that Co" = ‘7011' + ix leir- 02mg 1 (10.17) 1 —1—i—“2) ’ ‘ J - h. .t. . ( + ( 93yi33j 04yiyj )...]. ( 111% j+ 0 1 Using (2.24) for :1:,- and y,- we can rewrite (10.17) in terms of z.- and 21- as I 0,,- : —7c.-,- +kU-lefm . 221—22,- +kij223'ZJ (1018) +kij3zi Zj + [€11,461 ‘ TzlzJ + 0(\/—) where km, 19,-2,1953 and 190-4 are some coefficients. We are not interested in km and km because after averaging all terms containing elm . with m 75 0 disappear, and we will have Cij’ = —76ij + kijzzii’j + kij35izj + 0(\/E) Taking the limit 5 —> 0 gives (10.15). It is easy to check that kifl and kijg are given as shown in (10.16). D 160 Remark 63 The fourth hypothesis about the learning dynamics is redundant. In- deed, if we drop it, then the function h(w,:r,y) defined by (10.13) acquires linear terms in a: and y. They add linear terms of the form eim/‘l’z and e‘im/‘)TE to (10.18), which eventually vanish after the averaging. Note that we assumed little about the actual learning dynamics. Nevertheless, the family of possible learning rules (10.15) that satisfy our assumptions is apparently narrow. In the next section we show that to be ”useful” the learning rule (10.15) must satisfy the additional conditions: Im km 2 0 and kij3 = 0. Using this and (10.16) we can determine what restrictions must be imposed on the plasticity rates 6,51, . . . , 0,54, and, hence, onto the possible organization of the network so that it can memorize phase information, which we discuss next. 10.5 Memorization of Phase Information We develop here the concept of memorization of phase differences. By this we un- derstand the following: If during a learning period neuron A excites neuron B such that B generates an action potential with time delay 7', then changes occur so that whenever A generates an action potential then so does B with the same time delay T. Since in the real brain neurons tend to generate the action potentials repeatedly, instead of the time delay we will be interested in phase difference between dynamics of the neurons A and B. So, if during a learning period, two neural oscillators generate action potentials with some phase difference, then after the learning is completed, they can reproduce the same phase difference. Whether memorization of phase differences is important or not is a neurophys- iological question. We suppose here that it is important. Then, we would like to 161 understand what conditions must be imposed on a network’s architecture to ensure it can memorize phase differences. The memorization of phase information in terms of the canonical model (10.6) means the following: Suppose during a learning period the oscillator activities 2,-(7') are given so that the phase differences Arg Zgzj are kept fixed. We call the pattern of the phase differences the image to be memorized. Suppose also that the synaptic coefficients c,,- are allowed to evolve according to the learning rule (10.15). Then we say that the canonical model memorized the image if there is an attractor in the z-space such that when the activity 2(7) is on the attractor, the phase differences between the oscillators coincide with those to be learned. Theorem 64 Suppose the neural oscillators have equal center frequencies (.01 = : can 2 w. Consider the weakly connected network of such oscillators governed by n z,’ = (p,- + iw)z,- — z,-|z,-|2 + Zcijzj, i = 1,...,n, 1:1 together with the learning rule (10.15). The network can memorize phase difierences of at least one image if and only if kifl > 0 and kij3 = 0, (10.19) i.e. the learning rule {10.15) has the form C15,: —'yc,-,- + kgjzizj, i aé j, (10.20) where kij, i,j : 1,. . .,n, are positive real numbers. Proof. Let us introduce the new rotating coordinate system elmzi(r). In the new coordinates the canonical model becomes 2:," =pizi—ZilZiI2-‘I-qu2j, 2=1,...,n. (10.21) i=1 162 First, we prove that (10.19) is a sufficient condition. Our goal is to show that after learning is completed, the dynamical system (10.21) has an attractor such that the phase differences on the attractor coincide with those of the memorized pattern. Let (10.21) be in the learning mode such that the phase differences ¢,(r) — ng-(r) (mod 27r) =Arg zg—Arg Zj (mod 271’) =Arg Zifj are kept fixed. Then, according to the learning rule (10.20) the coefficients Cij approach 53112.- 2,- and, hence, 1b,,- approaches Arg 2.5,, where Cij = Icijlei‘i’v. Note that 212,5 satisfies 1% = film and 1% = thik + wkj (10.22) for any i, j and k. We must show that after learning is completed the neural network can reproduce a pattern of activity having the memorized phase differences 2%,. We assumed that during learning all activities z,- 75 0 so that the phases cf).- of the oscillators are well—defined. It is easy to see that after learning is complete we have ci,;£0fori;£j. Consider (10.21) in polar coordinates { 73' = pm- - 7"? + ELI lCijITj C03(¢j + tbij — (152') (10 23) (15:! = :1: 231:1 ICuITj Sin(¢j + the — 9M ' Let us show that the radial components, determined by , I __ 3 n ' 7, — p,r,- — rz- + Z: [CU-Ir, cos(ql>j + tbij — 915;), i = 1,. . . ,n (10.24) i=1 are bounded. Indeed, let B(0, R) C R" be a ball at the origin with arbitrarily large radius R > 0. Consider the flow of (10.24) outside the ball. After the rescaling r,- —+ Rri, T ——> 1247’, the system (10.24) becomes r,'=—r?+O(R'2), i=1,...,n, which is an 124-perturbation of r,- = —r?, i = 1,...,n. (10.25) 163 For any initial conditions the activity vector of (10.25) is inside a unit ball B (0, 1) after some finite transient. Any perturbations of (10.25) has the same property. Therefore, after the finite transition interval the activity vector of (10.24) is inside B(0, R) for any initial conditions and any values of $1, . . . , (tn. Hence, all attractors of (10.23) lie inside the cylinder B(0, R) x R" C R2". Fix index k. It is easy to check that the hyperplane 492' = 45k + Wk, 2'75 ’9 (10.26) is a global invariant manifold. Indeed, using (10.22) we have 453' + 11sz — 952' = ¢k + ijk + IPij — ¢k - 1P“: = 11’0" — l/Jij = 0 (10.27) for all i and j, and, hence, (151', = 0- From (10.27) we obtain the same invariant manifold for any other choice of k. In order to study the stability of the manifold consider the auxiliary system 1 n ., = _ .. . ' . .. _ . ' 9”“ Ti ; Ic,,|r, 5111(951 + $23 05:), Z 75 ’6, (10.28) where r1, . . . , r,, and (pk are fixed. Since all attractors of (10.23) are inside the cylinder, we may assume that r.- < R for all i. The Jacobian matrix of (10.28), say J = (Jij)i,j;ék, is diagonal—dominant because J- —{ {flail Wei. n U I U “:1: Zmzl lcimlrm i = .7 and n 7' k Jiz‘ + ZJij = "Elcikl < -7‘kmc < 0, #2 where 1 O l I 2' —m1n Ci . Mk R i¢k Ic 164 This means that all eigenvalues of J have negative real parts; Hence, (10.26) is an asymptotically stable equilibrium point for (10.28). Therefore, in the original 2n- dimensional system (10.23) the flow is directed everywhere toward the invariant man- ifold, at least for (r1, . . . , r,,) 75 (0, . . . ,0). Hence, the manifold contains an attractor of ( 10.23). Moreover, it is possible to prove that the complement of its domain of attraction has measure zero, i.e. this is the only attractor for (10.21). Note that on the manifold the phase differences satisfy 451 — ij = l/h‘j- Thus (10.19) is a sufficient condition memorization and recall of phase differences. It should be stressed that the oscillators have constant phase shifts on the mani- fold even when the attractor is not an equilibrium point. For example, if the attractor were chaotic, then one could observe an interesting phenomenon: The oscillator’s am- plitudes r1, . . . , r,, have chaotic activity whereas their phases gbl, . . . , 45,, have constant differences d,,-j. Thus, the synchronization does not necessarily mean that the entire network’s activity is on the limit cycle. Next, we show that conditions (10.19) are necessary. Since the pattern of activity to be memorized and the values of p1, . . . , p" are not specified, it is assumed that the network can learn and reproduce phase shifts of any activity pattern 2* 2 (2‘1", . . . , 2:)T for any choice of pi, . . . ,pn. The phase difference between i-th and j-th oscillators during the learning period is Arg zf—Arg z; (mod 27r) =Arg sz} Hence, the same value must be reproduced after the learning is completed. From Lemma 39 it follows that the network can always reproduce the phase shifts qt),- — qu (mod 27r) 2 114-1- =Arg Cij. Therefore the equality lkij =Arg 2:2; must be satisfied. This is possible, for any 2*, only if (10.19) holds. Hence (10.19) is necessary. D 165 Note the similarity of (10.20) and the Hebbian rule (10.14). The only difference is that in (10.20) the variables c and z are complex-valued. Let us rewrite ( 10.20) in polar coordinates: If c,,- = ICU-lei“! and z,- = new", then lcijl’ = -7|c.-j| + k,,—r.~r,- 008015.- — (153' — the) $23, = Icfjkijrirj Sin(i - 491' — I/h'jl- From the second equation it is clear that Ifng —) (b; - ¢j (mod 271') as we expected on the basis of Lemma 39. Notice that if 1b,,- = gt), — 46,-, then cos(¢,- — d),- — wij) = 1 and first equation coincide with the Hebbian learning rule. Since we know how km and ICU-3 depend upon the original WCNN we can restate the results of Theorem 64 in terms of (10.1). Almost all results discussed in the next section are straightforward consequences of the following Corollary 65 A weakly connected network of neural oscillators can memorize phase differences if and only if the plasticity rates satisfy 9m = 9133, 9m = 9m (10.29) and [Ca 2 (001+ 03002) > O, (1030) where 0,- > 0 is defined in (10.16). The proof follows from application of the condition (10.19) in Theorem 64 to the representation (10.16) in Lemma 62. 10.6 Synaptic Organizations In this section we apply Corollary 10.30 to various synaptic organizations. 166 Corollary 66 The rate of synaptic plasticity is locally determined by the pre-synaptic neurons. Proof. Indeed, the constants 0,51 and 0,53 determine the rate of synaptic plasticity from the same pre—synaptic neuron 33,- onto different post-synaptic neurons at,- and y,, which belong to the same neural oscillator. It follows from condition (10.29) that the plasticity rates must coincide. Similarly the same result is true for d,,-2 and 0,54 D It would be incorrect to think that the values of the actual synaptic connections 5,, depend exclusively upon pre-synaptic neurons. The Corollary merely claims that the pre—synaptic neurons are the only neurons that regulate the rates of plasticity, the ”speed” of modification, but not the modification itself. This corollary is consistent with neurobiology. It is known (Shepherd 1983) that many chemicals pass through the axon of a neuron. Modifications and growth of the axon terminals depend crucially upon these chemicals and, hence, upon the neuron. So, it should be expected that the rate of modification, which is connected with learning, also depends crucially upon the pre—synaptic neuron. Let us study possible synaptic organizations of the network from the point of view of learning of phase differences. Suppose that 0.51 = 0 for some i 76 j. That is, there is no modification of synapses between j-th and i-th excitatory neurons except for fading (atrophy). So, even if a synapse 3.7-1 between x,- and :1:,- existed at the beginning, it would atrophy with time. Thus, without loss of generality we may assume that 0,51 = 0 means that formation and growth of synapses from :1:,- onto 3:.- is impossible. The same consideration can be applied to the 002,003 and 0,54. In figure 10.8 we draw arrows from one neuron to another only when the corresponding plasticity rate is nonzero, i.e. only if synaptic contact between the two neurons is possible. Different choices of the arrows correspond to different synaptic organizations of the neural 167 Cl C] E NEE \3 Vii/EVE IX? IE1 H H a? A Cl O O. b c (1 Figure 10.8: Open boxes are excitatory neurons, shaded circles are inhibitory neurons. If there is an arrow between two neurons, then the synaptic contact is possible. a. The synaptic organizations that cannot memorize phase information. b. The synaptic organization that can either learn or unlearn phase information (but not both). If the network has more than two oscillators, then the Dale’s principle will be violated during the learning. c. The synaptic organization that can learn phase information. (I. The synaptic organization that can both learn and unlearn phase information. network. Corollary 67 The synaptic organizations depicted in Figure 10.8a cannot memorize phase information. Proof. According to the condition (10.29) if one of the plasticity rates is zero, then so should be the other one corresponding to it. Thus, the arrows must be in pairs, i.e. if a neuron has synaptic contacts with some neural oscillator then it must have access to both excitatory and inhibitory neurons of the neural oscillator. Obviously, none of the architectures on Figure 10.8a satisfies this condition. Cl 168 it Figure 10.9: Cij and c_,-.- must be inside the shaded area between —v4 and —v2 Corollary 68 The synaptic organization depicted in Figure 10.8b can memorize phase information only if the network consists of two type A neural oscillators and the phase difference to be memorized is close to 7r. Proof. From the proof of Theorem 64 it follows that Arg c.-J- = —Arg ij, i.e. Cij and c], must lie on two symmetric rays from the origin (see Figure 10.9). In order to satisfy Dale’s principle both beams must be inside the shaded area between —v4 and ——v2 (see classification in our previous paper). This is impossible if the neural oscillators are of type B. If they are of type A, then the phase difference to be memorized should be sufficiently close to 7r (within (Arg v4)—neighborhood). Suppose the network consists of more than two oscillators. Then the phase differ— ence between first and second and between second and third neural oscillators should be close to 7r. Hence the phase difference between first and third oscillators is close to 0. Thus C13 and C31 violate Dale’s principle. D Since networks seldom have only two elements we can conclude that this synaptic organization is not much better than those depicted in Figure 10.8a. We see that the only candidates for the synaptic organizations that can memorize phase differences are those depicted in Figure 10.8c and d. To discuss them we use the condition (10.30). What are the signs of 01-1-1 and 0.7-2? We show that 0ng 2 0 as it was postu— lated by Hebb (1949). Indeed, if two excitatory neurons generate action potentials 169 simultaneously, then the case 001 2 0 corresponds to increasing of strength of ex- citatory synaptic connections between them. For 0i]? we have a slightly different situation. The pre—synaptic neuron yj is inhibitory. It is not clear what changes take place in the synapses from y,- to :1:,- (or y.) if they fire simultaneously. We consider both cases: 00-2 2 0, which corresponds to decreasing of strength of the inhibitory synapse 8m < 0, and 6:72 g 0, which corresponds to increasing of the strength of Sijg. The case 0g; 2 0 is straightforward. Indeed, since 01- > 0 we have 0ng +0,-0.-,-2 > 0 and both synaptic organizations depicted on Figure 10.8c and d can memorize phase differences. The case 0,52 S 0 requires more attention. Obviously, the synaptic organization in Figure 10.8c can memorize phase differences because 0,52 = 0 for it, and hence condition (10.30) is satisfied. The synaptic organization depicted in Figure 10.8d needs special discussion. In this case hi, can be positive or negative depending upon the relative values of 9,51 and 90-2 (see Figure 10.10). Note that when kij < 0 the network memorizes not the presented image but its inverse (photographic negative). Sometimes it is convenient to think of this being that the network unlearns the image. So, the synaptic organization on Figure 10.8d is able not only learn but also unlearn information simply by adjusting the plasticity rates 92'1'1 and 90'2- The special choice of the plasticity constants such that 1% = (9251 + 019112) = 0 is interesting in the following sense: Since the plasticity constants are not zero, there are undoubtly some changes in synaptic coefficients sijk between the neurons. 170 Figure 10.10: For different choices of the plasticity rates 9 the oscillatory neural net- work can learn (k > 0), unlearn (k < 0) or passively forget (k = 0) phase information. Here k = 01 + 092. Negative value of 02 corresponds to increasing of strength of the inhibitory synapses. Positive value — to decreasing of the strength. Nevertheless, the network as a whole does not learn anything because for k,,- = 0 I Cij = —70u as it follows from (10.20). Moreover, the network forgets (loses) information because Cij —> 0 as T increases. Hence, the full-connected synaptic organization can exhibit a broad dynamic repertoire. By applying arguments similar to those used in the proof of Corollary 68, we can see that in both synaptic organizations the neural oscillators must be of type A, or else Dale’s principle will be violated. Moreover, in the architecture on Figure 10.8c the phase difference to be memorized should be between —Arg v1 and Arg v1. Thus we have demonstrated the following result: Corollary 69 The only synaptic organizations that can memorize phase information are those depicted on Figure 10.8c and d for type A neural oscillators. In both cases the excitatory neurons are long-axon neurons capable offorming synaptic connections with distant neurons and the inhibitory neurons might have long {case c) or short (case d) axons. Note that in the synaptic organization depicted in Figure 10.8c only excitatory neurons may have long axons. The inhibitory neurons can make synaptic contacts 171 only between themselves and nearby excitatory neurons within the same neural os- cillator (see Figure 10.3). Copious neurophysiological data (Shepherd 1983, Rakic 1976) suggest that exci- tatory neurons usually have long axons and inhibitory neurons are local-circuit inter- neurons. It is believed that the inter-neurons process information locally whereas the long-axon neurons transmit it to other regions of brain (Rakic 1976). In the model that we studied above there is no local processing of information. Each neural oscillator works in the very primitive regime — oscillation. Nevertheless, even this simple neural network suggests that it is very important to see this natural division into local circuit inter-neurons and long-axon relay neurons. Chapter 11 Discussion 11.1 Canonical Models and Normal Forms One of the most important results presented in this work is the reduction of gen- eral WCNN to canonical models. The canonical models have few non-linear terms, nevertheless, they capture the qualitative behavior of the original WCNN. Another method of simplification of dynamical systems is known as the Normal Form Theory (Arnold 1982, Guckenheimer and Holmes 1983). Recall that the Normal Form Theory considers a dynamical system i‘ = f (a?) at an equilibrium point. Let L be the Jacobian matrix at the equilibrium and let A1, . . . , An be the eigenvalues of L. Then there is a near identity change of variables y=x+Mfl that transforms the original system to a new system of the form 3) = 901) with the following properties: The nonlinear function g has the smallest number of non—linear terms, i.e. it has only the resonant terms corresponding to integer-valued 172 173 relationships between the eigenvalues of L, namely, the relationships of the form As : Zn: mkAka k=l where each mk is a non-negative integer and ka 2 2. The relationship above is called a resonance. Thus, by the change of variables y = a: + h(w) it is possible to “kill” all non—linear terms except the resonant ones. The resulting system is called the normal form. Thus the question: Are the canonical models that we derived in this thesis the normal forms for WCNNS? The answer is NO. Consider a WCNN is; = f,(:r,) + egi(:r), 3:; E R, i = 1,...,n (11.1) near, say, multiple saddle-node bifurcation. Since dfi/dati = 0 for all i, the Jacobian matrix for (11.1) is a zero 72 x n-matrix. Since it has n zero eigenvalues, all terms in f = (fl, . . ., fn) are resonant. Therefore, the normal form for (11.1) coincides with (11.1), whereas the canonical model for multiple saddle-node bifurcation generically differs from (11.1). We see that direct application of the Normal Form Theory to WCNNs is useless. Nevertheless, it must be acknowledged that the Normal Form Theory helped us when we studied multiple Andronov-Hopf bifurcations in Section 2.5. 11 .2 Synaptic Connections Our use of the word “synaptic” is an abuse of language, which has become custom- ary in the neural network literature. There is no reason for the connections between neurons to be exclusively synaptic. One neuron can affect another using other than direct synapse interactions. For instance, a neuron can cause secretion of neuropep- tides and hormones that reach other neurons by passive diffusion or advection. Such 174 connections can play significant roles in processing information by the brain though they are not synaptic. Nevertheless, they can be taken into account by the functions G.- in the WCNN (2.1) and, hence, are accounted for in the synaptic coefficients c,,-. 11.3 . Mathematical Conditions and Biology When we analyze the WCNNs, we impose some mathematical conditions, such as the adaptation condition DAfiAl + gi(09 OapOaO) : 09 (112) or such as Q,- = 51,-, etc. There are many of them in Chapters 2 and 3. Caution should be used in application of these conditions to biological objects. Biologists might say that we do not know (and probably will never know) all the underlying laws which govern the biological systems. Hence we might not know the exact values of the variables needed for checking the mathematical conditions. More- over, due to constant perturbations from the outside world, the values of biological variables fluctuate. Thus, it could be unlikely that an exact condition, say (11.2), is satisfied for any reasonable period of time. Mathematician might add that for generic f.- and g,- (11.2) holds only for A1 from a set of measure zero. Therefore, exact conditions such as (11.2) could be meaningless for biological systems. Recall that all WCNNs have a small parameter 5. In our analysis 15 operates as a basic unit of measure. All the conditions that we impose on the WCN N dynamics must be satisfied up to 0(5): The network should be in some e-neighborhood of a multiple bifurcation point; Natural frequencies 0,- should be e-close to each other; The adaptation condition (11.2) should be satisfied up to order 5, etc. Thus, all conditions that we impose on WCNNs are not “exact”. By allowing fluctuations of 175 order 5 we make the conditions meaningful for biological systems. 11.4 Co-dimensions of the Models Let us count how many conditions should be satisfied so that WCNN (2.1) is governed, for example, by the canonical model (2.16). The number of conditions is called the co-dimension. First, each neuron must be at a saddle-node bifurcation, there n conditions. Second, the adaptation condition (2.11) gives another n conditions. Thus, the co- dimension of (2.16) as a canonical model of (2.1) is 2n. Similar calculations show that the co—dimension of (2.21) is 3n. Thus, there is a natural question: Which of the models is more generic? Suppose we have adapted networks of 100 and 10 neurons near multiple saddle— node bifurcation and cusp singularity, respectively. From the pure mathematical point of view the second network is more generic because there are only 30 conditions to be satisfied (instead of 200 for the first network). From the common sense of a neurobiologist the answer is not so obvious. It is well known that despite the quantitative differences, the neurons are qualitatively similar. If there is an electro—physiological mechanism that forces a neuron to be near threshold and to be adapted, then the same mechanism should be present in the other 99 neurons in the lOO-neuron network. In other words, if two conditions are satisfied for one neuron, then it is physiologically plausible that approximately the same conditions are satisfied for the other neurons. Thus, there are only two non- similar conditions imposed on the co—dimension-2n network, whereas there are three conditions in co—dimension-3n networks. We see that there is an apparent discrepancy between the mathematical and neural network notions of co—dimension. 176 Another aspect of co—dimension is that when we have a dynamical system and a set of bifurcation parameters, it could be difficult (even numerically) to find a bifur- cation point of high co—dimension. But if we know that there are interesting neuro- computational properties near some multiple bifurcation with high co—dimension, then it is an easy task to construct a dynamical system which is close to the multiple bi- furcation. 177 11.5 List of Canonical Models Below is the list of canonical models derived in this thesis. according to the order of appearance in the text. Co—di- Canonical model men- sion 2n 1‘:- = 7‘,‘ + 0,185 + 13.2-1- Z ngl‘j 3n 1':- : r;+b,-.r,- i$?+265j£j 2n —1 z,’- = bizi + dizilzil2 + 20'ij 2n :13:- = “‘31:” + ril‘i + 1312+ Zen-xi y; z (1513; 3n 2:1- : —y.' + rixi i x? + Zen-1:,- y: : air, 2n 2‘ = Alvizi +biz1|21|2+20ijzj v; = d.-(R.- + 5.1.2.1" + Tm) 2n xgH$i+€(ri+xi2+Zcijxj) n. 1251—) r5+e(r.'1?.‘ ix?+ZCijrj) Conditions Saddle-node bifurcation, adaptation condition Cusp singularity, adaptation condition Andronov-Hopf bifurcation or quasi-static pitchfork bifurcation, equality of frequencies Quasi-static saddle-node bifurcation, p 2 0(62) Quasi-static pitchfork bifurcation, p = 0(52) Quasi-static Andronov-Hopf bifurcation, equality of frequencies Saddle-node bifurcation, adaptation condition Flip bifurcation, They are presented Reference Theorem 9 Theorem 10 Theorem 11 and Section 3.5.3 Theorem 23 Theorem 28 Theorem 34 Section 4.2.1 Theorem 36 Derivation and analysis of the canonical models is considered to be the major result of the thesis. BIBLIOGRAPHY BIBLIOGRAPHY J.C. Alexander, E.J. Doedel and HG Othmer (1990), On the Resonance Structure in a Forced Excitable System, SIAM J.Appl.Math., 50:1373-1418 V.I. Arnold (1982), Geometrical Methods in the Theory of Ordinary Differential Equations, Springer-Verlag, NY, (russian original, Moscow 1977) DO. Aronson, G.B. Ermentrout and N. Kopell (1990), Amplitude Response of Coupled Oscillators, Physica D 41:403—449 F. Attneave (1971), Multistability in Perception, Scientific American 225:63—71 S.M. Baer and T. Erneux (1992), Singular Hopf Bifurcation to Relaxation Oscil- lations II, SIAM .I.Appl.Math., 52:1651—1664 S.M. Baer and T. Erneux (1986), Singular Hopf Bifurcation to Relaxation Oscil— lations, SIAM J.Appl.Math., 46:721—739 B. Baird (1986), Nonlinear dynamics ofpattern formation and pattern recognition in the rabbit olfactory bulb, Physica D 22:150—175 K. Bar-Eli (1985), On the stability of coupled chemical oscillators, Physica D 14:242—252 J. Belair and P.Holmes (1984), On linearly coupled relaxation oscillations, Quar- terly of Appl.Math 42:193—219 H.H. Dale (1935), Pharmacology and the nerve endings, Proc Roy Soc Med 28:319- 332 T. Ditzinger and G. Haken (1989), Oscillations in the Perception of Ambiguous Patterns, Biological Cybernetics 61:279-287 W. Eckhaus (1983), Relaxation Oscillations Including a Standard Chase of French Ducks, Lecture Notes in Math.,985:432—449 178 179 R. Eckhorn, R. Bauer, W. Jordan, M. Brosch, W. Kruse, M. Munk, H.J. Reitboeck (1988), Coherent Oscillations: A mechanism of feature link in the visual cortex? Biol Cybern 60:121-130 P. Erdi, T. Grobler, G. Barna and K.Kaski (1993), Dynamics of the Olfactory bulb: bifurcations, learning, and memory, Biological Cybernetics 69:57—66 B. Ermentrout and N. Kopell (1994), Learning of phase lags in coupled neural oscillators, Neural Computation 6:225—241 G.B. Ermentrout and N. Kopell (1991), Multiple pulse interactions and averaging in systems of coupled neural oscillators, J.Math. Biol. 29:195—217 G.B. Ermentrout and N. Kopell (1990), Oscillator Death in Systems of Coupled Neural Oscillators, SIAM J. Appl. Math. 50:125—146 R. Fitzhugh (1969), Mathematical Models of Excitation and Propagation in Nerve, in Biological Engineering, H.P.Schwan, ed., McGraw-Hill, New York, NY pp.1-85 P. Frankel and T. Kiemel (1993), Relative phase behavior of two slowly coupled oscillators, SIAM J.Appl.Math 53:1436—1446 Z. Gajic, D. Petkovski and X. Shen (1990), Singularly Perturbed and Weakly Coupled Linear Control Systems, Lecture Notes in Control and Information Sciences, Springer-Verlag Gantmacher (1959), Applications of the Theory of Matrices, Interscience, New York J. Grasman (1987), Asymptotic Methods for Relaxation Oscillations and Applica- tions, Springer-Verlag, New York. C.M. Gray (1994), Synchronous oscillations in neuronal systems: mechanism and functions, Journal of Computational Neurosci 1:11-38 C.M. Gray, P. Konig, A.K. Engel and W. Singer (1989), Oscillatory responses in cat visual cortex exhibit inter-columnar synchronization which reflects global stimulus properties, Nature 338:334—337 M. Golubitsky and D. Shaeffer (1979), A theory for imperfect bifurcation via sin- gularity theory, Communications on Pure and Applied Mathematics, 32:21—98 S. Grossberg (1988), Nonlinear neural networks: Principles, mechanisms, and architectures, Neural Networks 1:17—61 180 J. Guckenheimer and D. Holmes (1983), Nonlinear Oscillations, Dynamical Sys- tems, and Bifurcations of Vector Fields, Springer, New York D.O. Hebb (1949), The Organization of behavior. Wiley, New York M.W. Hirsch (1989), Convergent activation dynamics in continuous time networks, Neural Networks 2:331—349 A.L. Hodgkin and AF. Huxley (1954), A quantitative description of membrane current and application to conduction and excitation in nerve, J.Physiol., 117:500—544 J.J. Hopfield (1982), Neural networks and physical systems with emergent collective computational abilities, Proc Natl Acad Sci USA 79:2554—2558 F.C. Hoppensteadt and EM. Izhikevich (1995a), Canonical Models for Bifurca- tions from Equilibrium in Weakly Connected Neural Networks, World Congress on Neural Networks, Washington DC, extended abstract, 1:80—83 F.C. Hoppensteadt and EM. Izhikevich (1995b), Synaptic Organizations and Dy- namical Properties of Weakly Connected Neural Oscillators: 1. Analysis of Canonical Model. Biological Cybernetics, accepted F.C. Hoppensteadt (1993), Analysis and simulations of chaotic systems, Springer- Verlag, New York F.C. Hoppensteadt (1991), The searchlight hypothesis, J Math Biol 29:689—691 F.C. Hoppensteadt (1989), Intermittent chaos, self-organization, and learning from synchronous synaptic activity in model neuron networks, Proc Natl Acad Sci USA 86:2991—2995 F.C. Hoppensteadt (1986), An Introduction to the Mathematics of Neurons, Cam- bridge Univ.Press, Cambridge, UK. G. Iooss and M. Adelmeyer (1992), Topics in Bifurcation Theory, Advanced series in nonlinear dynamics, vol.3, World Scientific. E.M. Izhikevich and CC. Malinetskii (1993), A neural network with chaotic be- havior, preprint #17 , Inst.Appl.Mathem.,Russia Academy of Sciences (in russian) E.M. Izhikevich and CC. Malinetskii (1992), A possible role of chaos in neu- rosystems, Dokl.Akad.Nauk 326:626—632 (translated in Sov.Phys.Docl.(1993) 37(10) October 1992:492-495) 181 Ya.B. Kazanovich and RM. Borisyuk (1994), Synchronization in a neural network of phase oscillators with the central element, Biol Cybern 71:177—185 N. Kopell and D.Somers (1995), Anti-phase solutions in relaxation oscillators cou- pled through excitatory interactions, J .Math.Biol. 33:261—280 N. Kopell (1986), Coupled oscillators and locomotion by fish. In: Levin S. (Ed) Nonlinear Oscillations in Biology and Chemistry. Lecture Notes in Biomathematics, Springer-Verlag. J.M. Kowalski, G.L. Albert, B.K. Rhoades and G.W. Gross (1992), Neuronal net- works with spontaneous, correlated bursting activity: Theory and Simulations, Neural Networks 51805—822 V.I. Kryukov (1991), An attention model based on principle of dominanta, In: Holden AV, Kryukov VI (eds) Neurocomputers and attention. I. Neurobiology, syn- chronization and chaos, Manchester University Press, Manchester, 319-352 Z. Li and J.J. Hopfield (1989), Modeling the olfactory bulb and its neural oscillatory processings, Biological Cybernetics 61:379—392 C. von der Malsburg C and J. Buhmann (1992), Sensory segmentation with coupled neural oscillators, Biol Cybern 67:233—242 R.E. Mirollo and SH. Strogatz (1990), Synchronization of pulse-coupled biological oscillators, SIAM J.Appl.Math. 50:1645—1662 J.C. Neu (1979), Coupled chemical oscillators, SIAM Journal of Applied Mathe- matics 37:307—315 G.M. Peponides and RV. Kokotovic (1983), Weak connections, time scales, and aggregation of nonlinear systems, IEEE Trans. on Systems, Man, and Cybernetics, SMC-13:527—532 T. Poston and 1. Stewart (1978), Nonlinear modeling of multistable perception, Behavioral Science 23:318—334 P. Rakic (1976), Local Circuit Neurons, MIT Press, Cambridge, Mass A. Rapaport (1952), Ignition phenomenon in random nets, Bulletin of Mathemat- ical Biophysics 14:35—44 G.M. Shepherd (1983), Neurobiology, Oxford University Press, New York G.M. Shepherd (1976), Models of LCN Function in the Olfactory Bulb, In: Rakic P (Ed.) Local Circuit Neurons, MIT Press, Cambridge, Mass 182 CA. Skarda and W.J. Freeman (1987), How brain makes chaos in order to make sense of the world, Behav Brain Sci 10:161-195 D. Somers and N. Kopell (1993), Rapid synchronization through fast threshold modulation, Biological Cybernetics 68:393—407 D.W. Storti and R.H. Rand (1986), Dynamics of two strongly coupled relaxation oscillators, SIAM J.Appl.Math. 46:56-67 I.N. Stewart and P.L. Peregoy (1983), Catastrophe theory modeling in psychology, Psychological Bulletin, 94:336-362 I. Tsuda (1992), Dynamic Link of Memory — chaotic memory map in nonequilib- rium neural networks, Neural Networks 5:313—326 H.R. Wilson and JD. Cowan (1973), A Mathematical Theory of the Functional Dynamics of Cortical and Thalamic Nervous Tissue, Kybernetik 13:55-80 H.R. Wilson and JD. Cowan (1972), Excitatory and inhibitory interaction in localized populations of model neurons, Biophys J 12:1—24 M. Zak (1989), Weakly connected neural nets, Appl Math Lett 3:131-135 "Illllllllllltlllf