1

 

LIBRARY
Michigan State
University

 

 

PLACE iN RETURN BOX to remove this checkout from your record.
TO AVOID FINES return on or before date due.

F___=

DATE DUE DATE DUE DATE DUE

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

:
“7W

 

 

 

 

MSU Is An Affirmative Action/Equal Opportunity Institution

cmma—o.‘

Analysis and Implementation of
Continuous-Time Feedback Neural Networks
with Multiple-State Neurons

Bo Ling

A DISSERTATION

Submitted to
Michigan State University
in partial fulfillment of the requirements
for the degree of

DOCTOR OF PHILOSOPHY

Department of Electrical Engineering
1993

ABSTRACT

Analysis and Implementation of
Continuous-Time Feedback Neural Networks
with Multiple-State Neurons

by

Bo Ling

This work has focused on the design and analysis of two types of continuous-time
feedback neural networks, namely, the Hopfield-type feedback neural network and the
dendro-dendritic artificial neural network (DANN). The Hopfield-type feedback neural
networks with two-state neurons are widely used in various applications, such as pattern
recognition, optimization, etc. The more recent DANN architecture has been used in pat-
tern association and as an edge detector. All of the networks considered in this dissertation

have multiple-state neurons.

For the Hopfield-type continuous-time feedback neural network with multiple-state
neurons, we have proposed an analytical recursive algorithm to find a set of weights which
will exactly store the desired gray-level patterns as asymptotically stable equilibria. We
introduce a criterion which will ensure that the designed Hopfield-type neural network to
be implementable as an electronic circuit. The criterion, moreover, allows the resistances
in the network to be made as large as desired. This flexibility has the practically appealing
consequence of using very large resistances and hence considerably reducing the power
consumption in the circuit implementation. We also give explicit bounds on the variation

of equilibria due to the variation in weights.

A cellular structure of the Hopfield-type feedback neural networks is also consid-
ered. We propose a large network structure which connects two or more smaller identical
networks (or cells) together. A parallel recursive algorithm for this type of locally con-
nected neural network is introduced. This parallel algorithm finds the weights for all cells

simultaneously.

For the DANN architecture, we show that multiple patterns can be stored as asymp-
totically stable equilibria with only positive weights. We further formulate the problem of
finding all positive weights as a standard linear programming problem. Thus, based on the
given patterns, all positive weights can be easily determined. We also point out that the all

positive weights can be solved via a nonlinear (e. g. quadratic) programming approach.

We have designed a CMOS circuit for a multiple-state neuron which operates in sub-
threshold. With this neuron, the network power dissipation is considerably reduced. As a
prototype, a DANN architecture using 6 neurons has been designed. Each neuron has four
states. Numerous PSpice simulations have shown that this prototype DANN network
works quite well. A Tiny-Chip using 6-neuron DANN architecture has been designed and
will be fabricated via MOSIS. All internal signals can be accessed through the pins of this
chip for thorough laboratory testing.

ii

ACKNOWLEDGMENTS

The author would like to express sincere thanks to Professor Fathi M.A. Salam for
his inspiration and guidance. Thanks to Professor H.K. Khalil, Professor J. Deller, and

Professor C.R. MacCluer for their comments and suggestions.

iii

Table of Contents

Chapter 1 Introduction ............................................................................................. 1

Chapter 2 Hopfleld-type Continuous-time Feedback Neural Networks

with Multiple-state Neurons .............................................................. 6
2.1 Background .................................................................................................... 6
2.2 Stability analysis ........................................................................................... 8
2.3 An analytical method for finding weights ..................................................... 12

2.4 A recursive algorithm for finding weights based on

multiple patterns ........................................................................................... 18
2.5 Network capacity, "storable" and "unstorable" patterns ............................ 20
2.6 Parameter determination for implementable networks .............. ' ................. 22
2.7 Persistence of equilibria under weight variations ........................................ 35
2.8 A CMOS circuit of 2n-state neuron operating in sub-threshold ................ 47
2.9 Computer simulation of a four-state neural network with 16 neurons ....... 53
2.10 A schematic layout for an n-neuron feedback neural network ................... 59

Chapter 3 Cellular Hoptieid-type Continuous-time feedback Neural

Networks with Multiple-state Neurons .......................................... 61
3.1 Background .................................................................................................... 61
3.2 Structure of the cellular Hopfield-type neural networks ............................. 63
3.3 A parallel algorithm for finding weights ....................................................... 67

Chapter 4 Dendro-dendritic Artificial Neural Networks with Multiple-state
Neurons ..................................................................................................... 69

4.1 Background .................................................................................................... 69

iv

4.2 Stability analysis for a DANN with positive weights ................................ 73
4.3 An analytical method for finding positive weights

via linear programming ................................................................................. 74
4.4 Computer simulation of a three-neuron DANN network ............................ 79

4.5 Network implementation and layout of a three-neuron DANN network 81

4.6 A schematic layout for an n-neuron DANN network .................................. 85
Chapter 5 A Tiny-Chip Layout of a Six-Neuron DANN Network ................... 87
5.1 Computer simulation of a six-neuron DANN network ......... 37

5.2 PSpice simulation of a six-neuron DANN network with

resistor-connections ....................................................................................... 90

5.3 PSpice simulation of a six-neuron DANN network with nMOS

transistor-connections ................................................................................... 94

5.4 A Tiny-Chip layout of a six-neuron DANN network ................................... 98
Chapter 6 Summary, Conclusions, and Future Work ....................................... 102
List of References ......................................................................................................... 104

List of Figures

Figure 2.6.1 A circuit schematic of a continuous-time neural network ........................ 22
Figure 2.6.2 Relation between u and t) ......................................................................... 24
Figure 2.6.3 A simple 2-state neuron circuit .................................................................. 32
Figure 2.6.4 The input-output characteristic of the circuit shown in Figure 2.6.3 ........ 32
Figure 2.6.5 A 2-state neuron Operating in sub-threshold ............................................ 33
Figure 2.6.6 The input-output characteristic of the circuit shown in Figure 2.6.5 ........ 33
Figure 2.6.7 The characteristic of the bias. current and the input voltage of the circuit

shown in Figure 2.6.5 .................................................................................. 34
Figure 2.8.1 A 2n-state neuron circuit operating in sub-threshold ............................... 48
Figure 2.8.2 The sub-circuit serving as a building block ................................................ 48
Figure 2.8.3 A schematic diagram of the 8-state neuron circuit .................................... 49
Figure 2.8.4 The Io'Vin characteristic of an 8-state neuron with W: 80, W: l ............. 49
Figure 2.8.5 The modified transconductance circuit ........................................................ 50
Figure 2.8.6 Four Io-Vin curves of a 4—state neuron circuit with different

values of W ................................................................................................... 52
Figure 2.9.1 The characteristics of a 4-state neuron ...................................................... 54
Figure 2.9.2 The characteristics of a 4-state neuron with positive output ................... 57
Figure 2.10.1 A schematic layout for a 4-neuron network ............................................... 59
Figure 2.10.2 A schematic layout of an n-neuron network with 4»state neurons .......... 60
Figure 3.2.1 Cellular Hopfield-type neural network built on two small network ......... 64
Figure 4.1.1 One circuit configuration type of a DANN with 3 neurons ........................ 69
Figure 4.1.2 Another circuit configuration type of a DANN with 3 neurons ................. 72
Figure 4.5.1 The schematic diagram of a 4-state neuron with voltage input

Figure 4.5.2

and voltage output ....................................................................................... 81
The PSpice simulation of the circuit in Figure 4.5.1 ................................... 82

vi

Figure 4.5.3 The DANN architecture with 3 neurons ..................................................... 82

Figure 4.5.4 Transient curves of the output voltages of 3 neurons ................................ 83
Figure 4.5.5 A physical layout of the circuit in Figure 4.5.4 ............................................ 84
Figure 4.6.1 A schematic layout for a 4-neuron DANN architecture ............................. 3 5
Figure 4.6.2 A schematic layout of an n-neuron DANN with 4—state neurons ............ 86
Figure 5.1.1 The characteristics of the sigmoidal function s(x) ...................................... 87
Figure 5.2.1 The schematic diagram of a 6-neuron DANN circuit ................................. 91
Figure 5.2.2 The schematic diagram of a 4-state neuron circuit ..................................... 92
Figure 5.2.3 The input-output characteristic of a 4»state neuron circuit ....................... 92
Figure 5.2.4 The simulation of a 6-neuron DANN network circuit ................................ 93
Figure 5.3.1 A single MOS transistor used as a programmable weight ....................... 94
Figure 5.3.2 The Ids - Vds transistor characteristic with V83 = 5V ................................. 95

Figure 5.3.3 The schematic diagram of a 6-neuron DANN with MOS-transistor

connections ................................................................................................... 96
Figure 5.3.4 A simulation result of the 6-neuron DANN network in Figure 5.3.3 ....... 97
Figure 5.4.1 The schematic diagram of a 6-neuron DANN network .............................. 99
Figure 5.4.2 The layout of a 6-neuron D ANN chip ......................................................... 100
Figure 5.4.3 The pin configuration of the 6-neuron DANN chip ..................................... 101

vii

CHAPTER 1

Introduction

Many models for the neural networks have been proposed since the work of McCul-
loch and Pitts [37] in 1943. Those models are often referred to as artificial neural net-
works. The Hopfield-type continuous-time feedback neural network [21] is one of several
popular neural networks. In this network, the input of each neuron is the sum of the output
signals of other neurons processed through proper weights. As proposed in. [21], there is
no self-feedback of the output of each individual neuron. One key issue in the design of
the Hopfield-type feedback neural network is to find the symmetric weight matrix based
on desired multiple patterns. The symmetry of the weight matrix is important in the design
I of the usual Hopfield-type feedback continuous-time neural network. Because of its struc-
ture simplicity, the Hopfield-type continuous-time feedback neural network has been the

object of numerous proposed implemented using VLSI technology.

Another interesting type of continuous-time feedback neural network is the Dendro-
dendritic Artificial Neural Network (DANN) [48, 49]. In this network, all neurons are con-
nected by either resistors or nMOS transistors which are controlled by their gate voltages
[49]. The input and output of each individual neuron is also connected by a feedback
nMOS transistor. The gate voltages of all those transistors can be set globally. It has been
shown [30, 48, 49] that multiple patterns can be stored as asymptotically stable equilibria
of a DANN with only positive weights, which is a very promising feature of DANNs. In
the Hopfield-type feedback neural network, weights could be negative in order to store
certain given patterns. Thus extra inverters are usually needed to implement those nega-
tive weights [21], which in turn would greatly increase the number of transistors in the

network. In DANN, because of the positive weights, all weights can be realized simply by

1

2

resistors or MOS transistors. Consequently, the network implementation is more compact
[47]. Moreover, compared with the Hopfield-type feedback neural network, the DANN
network has two other major advantages: (1) It requires potentially fewer weights; (2) It
has symmetric weights; (3) All weights can be nonlinear resistive elements. The realiza-
tion of symmetric weights is a bottleneck in the implementation of Hopfield-type feedback

neural network.

Many theoretical results on the Hopfield-type feedback neural network have been
reported in the literature (e.g. [46], [26], [14]). Although those results have established the
foundation of the Hopfield-type feedback neural network, there are still important
unsolved problems in the design of this type of neural network. In parts of this thesis, we

will propose solutions to some of those fundamental problems.

One important problem is the analytical algorithm for finding symmetric weights.
Michel and his co-workers have reported several results on this problem. In [26], a design
method is proposed for finding a symmetric weight matrix given analog (i.e. real number)
patterns, but this method does not ensure the existence of the weight matrix. In [14],
another method is proposed for finding a weight matrix which is not symmetric. The sym-
metry of the weight matrix, however, is important in the design of Hopfield-type feedback
continuous-time neural network. It ensures the global stability of the designed network

and the convergence to equilibria only. '

For a given structure of a Hopfield-type feedback neural network, there is a capacity
associated with it which limits the number of desired patterns that can possibly be stored.
Yet, even though the network capacity may not be reached, some patterns may not be suc-
cessquy stored. These "unstorable" patterns may cause all other patterns not to be stored

since there may not exist a symmetric weight matrix which makes the given set of patterns

3

stored together. Thus, it appears to be practically useful to separate those "unstorable" pat-
terns from the given set of patterns. In this case, one can still find a symmetric weight

matrix~ which will make part of a set of desired patterns to be stored as equilibria.

A stored pattern will be retrieved by other key patterns only if it is an asymptotically
stable equilibrium of the network. It is known [46] that a pattern, once stored, is asymptot-
ically stable if it corresponds to values within very "flat" regions of the sigmoidal function.
In such a "flat" region, the slope of the sigmoidal function is very small, but we don’t
know quantitatively how small the slope should be. It would be helpful if we knew the
upper bound on those small slopes. With this upper bound, we would be able to design
multiple-state neurons which will make the network capable of processing multiple-level
patterns. This upper bound would be a useful guide in designing a (simple) CMOS multi-

ple-state neuron circuit [32].

The analog VLSI design of the multiple-state neuron circuit is an interesting prob-
lem. With multiple-state neurons, the neural network may process gray-level images
instead of binary-value images. In [60], a multiple-state neuron circuit is designed. This
circuit operates above threshold which would cause a relatively large power dissipation in
the network. To reduce the power dissipation to the micro watt level, it became a goal of
this research to design a simple neuron circuit operating in sub-threshold. Moreover,

because of the. limited chip area, a simple multiple-state neuron circuit is desired.

Not all the (theoretically) designed Hopficld-type feedback neural networks can be
implemented as electronic circuits. One has to ensure that the theoretical design maps the
resistance to nonnegative values. Therefore, one needs to ensure a designed network to be
implementable in the form of a circuit. In [14], a successful synthesis is proposed to

resolve this problem, but, based on this approach, the resistance in the corresponding neu-

4

ral circuits may be small. This in turn would cause large network power dissipation. We
must find a new method which ensures that the designed Hopfield-type feedback neural

network is implementable and the resistors in the network are sufficiently very large.

In the Hopfield-type feedback neural network, the designed weights are realized by
hardware devices. Due to the limited hardware precision, the implemented weights will
not match the designed weights exactly. Thus, the stored pattern in the neural circuit will
not be the desired pattern. Since the weights are realized by certain circuit devices, we can
estimate their variation. Thus, it would be helpful to estimate the corresponding variation
of equilibria due to the variation of weights. That is, we desire to quantify the sensitivity
of the equilibria to the variations in weights. With the information on the variation of equi-
libria, we are able to predict the variation of the stored patterns of the implemented neural
networks. This would enable us to quantify the robustness of the equilibria to variations in

weights.

In real applications such as pattern recognition, a large number of neurons may be
required. However, due to the limited chip area, the number of neurons implemented on a
single chip can be quite limited. Therefore, we need to find a way to connect small net-
works (or chips) together to form a larger network. We may label such networks cellular
neural chips (CNC). In this larger network, each small network (chip) performs a certain
task. To build this larger network, we must find the connection structure of small cells, and
a parallel algorithm which makes all cells find their own weights simultaneously. This par-
allel algorithm is important because it ensures the "programming" time of the large net-

work will not increase significantly as its size becomes larger.

In this dissertation, based on multiple desired patterns, we introduce a new recursive

algorithm to find weights for the Hopfield-type continuous-time feedback neural networks

5

with two-state or multiple-state neurons [27]. We propose a criterion which will ensure the
designed Hopfield-type continuous-time feedback neural network with two-state neurons
to be implementable [29]. With our approach, the resistance in the network can be made as
large as necessary. We also estimate the variation of the equilibria based on the variation
of weights [28]. The cellular Hopfield-type feedback neural network chips is introduced in
[27]. A parallel algorithm for all cells to find their respective weights simultaneously is
also provided, too. We design a CMOS VLSI multiple—state neuron circuit operating in
sub-threshold [32].

For the DANN network, we formulate the problem of finding positive weights as a
standard linear programming problem [30]. As a prototype, a DANN with six neurons has
been designed and implemented . Each neuron has four states. PSpice simulation of this
prototype DANN network has shown it works well. We also design the circuit layout of
this six-neuron DANN network. A Tiny-Chip of this six-neuron DANN network has been
designed and will be fabricated via MOSIS.

This dissertation contains six chapters. In Chapter 1, we have briefly described the
research problems addressed. In Chapter 2, we focus on the Hopfield-type continuous-
time feedback neural network. The analytical recursive algorithm for finding weights that
store a given set of patterns is proposed. A cellular Hopfield-type feedback neural network
is discussed in Chapter 3. In Chapter 4, we formulate the. problem of finding positive
weights for a DANN as a standard linear programming problem. In Chapter 5, A Trny-
Chip of a 6-neuron DANN network has been designed. Finally, in Chapter 6, summary

and some final remarks are presented.

CHAPTER 2

Continuous-Time Feedback Neural Networks
with Multiple-State Neurons

2.1 Background

The mathematical model for the Hopfield-type feedback neural network [21] is

6114- " 1
i=1 '
V,- = 5,04,) (2.1.1ii)

for i = 1, 2, ..., n, where 0,, ’1') 0, ate R and v,- e (-A, A) with 0 < A e R. s,- : R —) (-A, A)
is a monotone increasing Cl-function with 5,-(0) = O, s,(x) = A (or -A) if and only if x ->
+oo (or -oo). n denotes the number of neurons in the network (2.1.1). I i is the external input.

For simplicity, assume I,- = O for all i. Wij are the weights, for i, j = 1, ..., n.

In compact matrix form, (2.1.1) can be written as

du -1
Ca; = WV-R “ (2.1.2i)

v = 5(a) (2.1.211)

where u = [u], ..., u,JT e R", v = [v], ..., vn]T e R", C = diag(c1, ..., c") e R""”, R =
diag(r1, ..., rn) e Rnx”, and s(u) = [31(u1), ..., .r,,(u,,)]T e R", and T denotes the transpose of

a matrix.

7

The Hopfield energy function [21] is given as follows:

1‘“ _
:55.- 1 (x) dx (2.1.3)

n
i=1 ‘

1 n n

Many theoretical results on the Hopfield-type neural network have been reported in
the literature (e. g. [46], [26], [14]). Those results have established the foundation of the

Hopfield-type neural network. Some of the most important properties are:

(1) For the energy function defined in (2.1.3), for all v e (-A, A)", dE(v)/dt S 0, and
dE(v)/dt = 0 if and only if v is an equilibrium point of (2.1.2).

(2) For any v e (-A, A)", there exists a unique solution of (2.1.2) for t 2 0.

(3) All solutions are bounded.

(4) An isolated local minimum of E(v) is an asymptotically stable equilibrium of (2.1.2).
(5) There are only finite number of equilibria of (2.1.2).

(6) Let v0 be an equilibrium point of (2.1.2). v0 is a local minimum of E(v) if and only if
the Hessian matrix JvE(vO) is positive definite.

(7) Let v0 be an equilibrium point of (2.1.2). v0 is isolated if detUVb-(VOD e o.

(8) No limit cycles exit.

Those properties are related to the existence of the solutions and the stability of equi-
libria of (2.1.2). Since any desired patterns will be stored as stable equilibria, it is essential
for them to be asymptotically stable.

2.2 Stability Analysis

Suppose v1, v2, ..., v’" are m given patterns. Here, we may assume the weight matrix
W is known. In the next two sections, we will introduce an analytical method to find this

weight matrix. In order to retrieve vi, 1' = 1, 2, ..., m, we must make vi asymptotically stable.

For the Hopfield-type neural network described in (2.1.2), it is known that the isolated
local minimum points of the energy function (2.1.3) are asymptotically stable. It can be
observed that a stored pattern can be either local maximum or local minimum. In other

words, an equilibrium might not be stable.

It is also known that a pattern, v, is a local minimum of energy function (2.1.3) if and
only if the Hessian matrix JVE(v) is positive definite. And, v is an isolated equilibrium if
det(JVE(v)) ¢ 0. Thus, if we can find a condition which guarantees JVE(v) > 0, we know

that v is asymptotically stable. It can be easily shown that
Jew) =- W+R"J,.(v) (2.2.1)
where h = s'l, Jh(v) = diag(h1’(v1), ..., hn’(v,,)), and W is the weight matrix for all of those

m patterns, v1, v2, ..., vm. Define B = W - R'llh(v). Then Jv5(v) is positive definite if and

only if B is negative definite. The elements of B satisfy:

bi} = Wij for i,j = l, 2, ..., n, iatj; (2.2.2i)

bi,- = -r,-'1h,-'(v,-) for i = 1, 2, n. (2.2.2ii)

Stability Criterion ([14], [46]):

Suppose v = M, ..., vn]T is an equilibrium point of network (2.1.2). Then v is

asymptotically stable if

n

h'.-(V.~)>'t 2 ani , fori=l,...,n. (2.2.3)
j=lj¢i

Proof: Suppose (2.2.3) is true. Then

I:

lbiil> 2 lbi/i , fori=l,...,n.
j=l,j¢i

By Gershgorin’s eigenvalue estimation theorem, the matrix B has all negative eigenvalues.

B is negative definite because of its symmetry. This completes the proof. I

Let u,- = h,(v,-). Since h,- = s51, the inequality (2.2.3) is equivalent to

 

s} (at) < .1 . for i = 1, n. (2.2.4)
r i Z Iwill
j=lJ¢i

For given m patterns, v1, v2, ..., v“, we have a recursive algorithm (introduced in
Section 2.4) to find the corresponding weight matrix W which only depends on the values
of the sigmoidal functions evaluated at those m patterns, and the values of those m patterns
themselves. The values of the derivatives of sigmoidal functions at those patterns do not
contribute to the calculation of weight matrix W. In other words, the weight matrix W will
not be changed if we vary the derivatives of the sigmoidal functions at those m patterns, v1,
v2, ..., v’". The stability criterion stated above actually put the requirement 9n]! on the deriv-

atives of sigmoidal functions at the m patterns. This makes it possible to design a Hopfield-

10

type neural network in two steps:

Step _1: Find the weight matrix W for m patterns, v1, v2, ..., v’";
Step 2: Adjust the derivatives of sigmoidal functions based on (2.2.4) to ensure the stability

of those m patterns.

In reality, v is continuously evaluated in (-A, A)". The stability criterion requires that
si’(u,-) be less than some number. This is usually impossible. For example, si’(u,-) would be
very large if u,- is close to the origin. But, in most neural applications, the given patterns are
binary, i.e., their elements take two difierent values, such as +1 and -1. And the sigmoidal
functions of all neurons are chosen to be identical, i.e., sl = = sn = so. Thus, for m pat-
terns, v" =[vk1, ..., vkan fork = 1, 2, ..., m, v",- = +1) or -t) for some t) > O,i=1, ..., n. Define
u = s0'1('u). Thus, we have u",- = +11 or ~11. For the binary input patterns, we have the fol-

lowing simple sufficient stability criterion:

Stability Criterion for Binary Pattern:
Suppose v = [v], ..., vn]T with v,- = +1) or 4) is an equilibrium of network (2.1.2). Define

u = so'l(u). Then v is asymptotically stable if

 

S601.) < 1 (2.2.5)

It

Max{ri 2 [wt-1]}

lSiSn j=1,j¢i

In general, the upper bound of (2.2.5) is quite small. The above criterion actually
requires the derivative of so at +11 or -p. be bounded above by some positive number,
denoted by (a, which depends on the given pattem, v. Since a) is pattem-dependent and the

pattern length is n, there are only 2" possible binary patterns. It is very easy to find a (no

11

such that any possible binary pattern, once stored, is asymptotically stable if so’ql) < (00.

For m patterns, v1, v2, ..., v’", we can find a weight matrix W such that these m patterns
can be stored. Let SV,-, i = 1, 2, ..., C(2",m), be a set of m possible binary patterns. For each
pattern set, SVi, we can exactly evaluate the upper bound 0),. Define 010 = Min{(l),-, i = 1, 2,
..., C(2",m)}. If we design the sigmoidal function so such that so’(u) < (no, then any pattern
in any possible pattern set, SVi, i = l, 2, ..., C(2",m), will be stored as a locally asymptoti-
cally stable equilibrium of the Hopfield-type neural network (2.1.2).

If we further assume all elements of R are identical, i.e., r1: r for i = 1, 2, ..., n. Then

(2.2.1) can be simplified as
. Jvz(v) = - W + r'lh’m)! (2.2.6)

Since W is symmetric, there exists orthogonal matrix P such that W = PAPT, where A =

diag(11, 7.2, ..., 7e"), and it; for i = l, 2, ..., n are the eigenvalues of W.

Theorem 2.2.1: ([46])
Suppose v = [v1, ..., vn]T with v,- = w or -t) is an equilibrium of network (2.1.2), and all
elements ofR are identical. Define it = so'1(t)). Then JVE(V) is positive definite if so’(u) <

1/(rkmax), where km” is the maximal modulus eigenvalue of W.

12

2.3 An Analytical Method for Finding Weights

First, let us consider the single pattern case. Let v = [v1, ..., vn]T be a given pattern,
where n denotes the length of the pattern, superscript T indicating the transpose of a matrix.
In the associative memory network design, we wish to find the weight matrix W in network
(2.2) such that the given pattern v will be stored as an equilibrium. Moreover, we desire v

to be asymptotically stable such that it can be retrieved by some patterns other than v.

Mathematically, in order to be an equilibrium of network (2.1.2), v must satisfy

Wv = R‘1u (2.3.1)

where W is the weight matrix which is symmetric and whose diagonal elements are all
zeros; R = diag(r1, ..., rn) with the resistor r,- connected to the ith neuron; v = s(u), s is the
sigmoidal function which is a strictly increasing C 1 function. Let h = s'l. Obviously, his
also a strictly increasing function. Since u = h(v), equation (2.3. 1) can be rewritten as fol-
lows:

Wv = R'lh(v). (2.3.2)

In equation (2.3.2), only the elements of W are unknown. Recall that the elements of W sat-
isfy [21]:

(l) wij = “’17 for l,j = 1, ..., n, l¢j (symmetry);

(2) wt,- = O for i = 1, ..., n (zero diagonal elements).

Therefore, among n2 elements of W, there are only (n2 - n)/2 elements needed to be deter-

13

mined. Let us denote this number by nw:

(n2 - n) (2.3.3)

NIH

”w:

This number, nw , represents the total number of unknowns in (2.3.2).

The diagonal elements of the weight matrix can be non-zero. If we use non-zero diag-
onal elements, then the number of unknowns would be (n2 + n)/2, instead of (n2 - n)/2. If
the number of neurons, n, is small, the non-zero diagonal elements will help to increase the
capacity of network which limits the number of desired patterns to be potentially stored. If
n is large, the non-zero diagonal elements will not significantly affect the neural network
performance since nw >> n. In this case, we would rather set them to be zero, which, in turn,

reduces the number of transistors in the neural circuit implementation.

By proper algebraic operations, we can transfer the matrix equation (2.3.2) to the

following equivalent matrix equation:

 

 

Ax=R‘lh(v) (2.3.4)
where
’ E E 5 E 1
V2 Vn olxm-Z) 5 E
§ v3 vn 02x (rt-3)
Vlln—r ......................... mvj ................. v [1“ ““““ out-2) x1
: v2In_2 v3ln_3 E vn
L . v"“ 4

’1on

14

x = [x], ..., xnw]T

with

W2j+2 nSiSZn-Z

~
coco-oat
cocoon.

 

 

L Wn-2,i+n-2 nw-Z S l .<_ nw-l

Wit-1J1 l - "w

Notice that nw > n if n > 3. So, in general, matrix A has more columns than rows. It
can also be seen that A is entirely determined by the given pattern v, i.e., A can be com-
pletely constructed once a pattern is provided. If we add non-zero diagonal elements of the
weight matrix, the expression of the matrix A and vector x would be different, but their

expression can be easily derived.

Let us denote b = R'1h(v) which is a constant vector for the given pattern, v. Thus,

equation (2.3.4) can be rewritten as
Ax = b (2.3.5)

In this matrix equation, there are nw unknowns in n equations. Thus, we expect the difficul-
ties in finding its solutions. Before continuing, let us investigate some properties of the

matrix A which will be useful for helping us find solutions of (2.3.5).

Lemma 2.3.1:

Suppose v = [v], ..., v,JT is a given pattern. A is defined in (2.3.4). Ith' at O for i = 1,

15

..., n, rank(A) = n.

Proof: Suppose v,- at O for all i. From the structure of A, we see that each column of A has
only two non-zero elements because v,- ¢ 0 for all i. Let a,- be the ith column of A, i = 1, ...,
nw Then a1, a2, ..., an are n linearly independent vectors, which implies rank(A) = n. This

completes our proof. I

Notice that AAT is non-singular; in other words, A is of full row rank. In general, (2.3.5)
may have more then one solution. If there exist solutions, they can be expressed in terms of
the Moore-Penmse pseudo-inverse of a matrix [6]. The following theorem deals with the

existence of solutions of (2.3.5):

Theorem 2.3.2 (Existence Theorem):
Suppose v = [v], ..., vn]T is a given pattern. A is defined in (2.3.4). Then equation

(2.3.5) has solutions if v,- ¢ 0 for i = l, ..., n.

Proof: Suppose v,- ¢ 0 for all i. By Lemma 2.3.1, rank(A) = n, which implies b e R(A)
where R(A) denotes the range of A. Therefore, Ax = b is consistent. It follows that (2.3.5)

has solution(s). This completes the proof. I

Theorem 3.2 actually guarantees that any continuous-valued single pattern with non-
zero elements will be certainly stored, but, as we will show later, multiple patterns might

not be stored all together in sequence.

We must note that Theorem 2.3.2 only states the existence of solutions of (2.3.5). But,
in general, the solution is not unique. As a matter of fact, the general solution set of (2.3.5)

can be expressed as {A+b + N(A)} (N(A) represents the null space of A) which is a linear

16

manifold in R”. Therefore,

x = A+b + y ' (2.3.6)

is a solution of (2.3.5), where A+ = AT(AAT)'1, y 6 MA). In this work, we choose

x=A+b (2.3.7)

as the solution of (2.3.5), which is the minimum-norm solution. Oncex is solved in (2.3.7),

the weight matrix W can be easily reconstructed as follows:

 

’0 x1 x2 -------- xn-Z xn-l
x1 0 In x2n-3 x2n-2
_ -------- (2.3.8)
W ..
_Xn l X”; x3n-3 ........ xnw O J

 

 

This W is the weight matrix of network (2.1.2) for the given pattern v which will make v
exactly stored as an equilibrium of (2.1.2).

Now, let us consider the multiple-pattem case. Suppose we are given m patterns, v1,
v2, ..., v’". Our goal here is to find weight matrix W such that those m desired patterns will
be stored (if possible) as equilibria of the Hopfield-type neural network (2.1.2).

For each v", i = 1, 2, ..., m, there exists Ai, b,- and x, as we shown early, such that Ax
= bi. To find weight matrix W for storing v1, v2, ..., v’" as equilibria of (2.1.2), we need to
find the common solutions of Apr = b,- for i = l, 2, ..., m. In other words, we need to find if

the following matrix equation

17

 

Al bl
A2 x = by, (2.3.9)
_Amd _bmJ

 

 

 

has solutions. Before proceeding, it is necessary to check whether (2.3.9) has solutions.

Recall that the solution set of A ,-x = b,- is given by {Afbi + N(A,-)} which is an affine
linear space in R”. Thus, (2.3.9) has solutions if and only if

{Afbl + N(A1)}n{A2+b2 + N(A2) }n...n{Am”bm + N(A,,,)} at Z

in other words, the intersection of the m affine manifolds, {A fbi + N(A,-)} for i = l, 2, ...,

m, is non—empty.

There are some other criteria used to check the existence of the common solutions of
Apr = b,- for i = 1, 2, ..., m. For example, the common solutions exist if and only if [b1T, sz,

..., melT e R([A1T, A21; ..., AmTJT), or the linear mapping of [A112 A21; ..., AmT]T is onto.

The major drawback of solving for the common solution of x fi'om (2.3.9) is the
large computation time. In fact, the dimension of [A 1T, AZT, ..., A mT]T is manw which can
be very large for large values of m and n. In the next section, we will introduce the so-

called Morris-Odell’s Theorem to overcome this computational problem.

18

2.4 A Recursive Algorithm for Finding Weights Based on
Multiple Patterns

Our goal in this section is to check for the existence of solutions and solve for the

common solutions for the following linear equations with reasonable computation time:

 

 

 

 

r" ‘ P '

A1 b1

A2 x = b2 (2.4.1)
LAms me_

Morris and Odell [41] propose a criterion to check the existence of common solutions
of Aix = b,- for i = 1, 2, ..., m. An excellent algorithm for finding the common solutions (if

they exist) is also given. In the following, we sketch their result.

Let m be a positive integer. Suppose A,- is a nxnw matrix and b,- is a nxl vector for i

= 1, 2, ..., m. Define

Ck =Aka-1

dk = bk - Akek-l (2.4.2)
91: = ek-l + Fk-lck+dk

Fk = Fk-l(l ' Ck+ck)

fOIk= 2, 3, ...,m With Cl =A1, d1=b1, 81 =A1+b1 and F1 =I-A1+A1. ThC Morris-Odell’s

Theorem is stated as follows:

Theorem 2.4.1 (Morris and Odell [41])

19

Apr = b,- for i = 1, 2, ..., m have a common solution if and only if Cflfdi = (1; for i =

l, 2, ..., m. The general common solution is x = em + sz where z is arbitrary.

The original theorem given by Morris and Odell in [41] was shown for general matrix

equations. There are some useful corollaries of this theorem in their paper.

Let m be a positive real number, and v1, v2, ..., v’" be m desired patterns. For those m
patterns, we have corresponding m matrix equations: Apr = b,- for i = l, 2, ..., m. Suppose all
of those m patterns can be stored as equilibria of the Hopfield-type neural network (2.1.2);
in other words, Ax = b,- for i = 1, 2, ..., m, have common solutions. Consequently, we have
the symmetric weight matrix associated with those m patterns. Denote this weight matrix
as Wl'm, where the superscript, l-m, simply indicates the weight matrix for those m pat-
terns, v1, v2, ..., v’". We will use this notation even for the case that not all of those m patterns

can be stored.

Now, r new patterns, v’”+ 1, vm+2, ..., W”, are provided. We wish for them, together
with those old m patterns, v1, v2, ..., v’", to be stored as equilibria of the Hopfield-type neural
network (2.1.2). By applying Morris-Odell’s Theorem, we first check to see if v’"" 1 can be
stored, i.e., check C,,,.,.1C+ m+1dm+l = dml, where Cm“ =Am+1Fm, am, = hm“ — A "Hem,
and all Am“, Fm, bml, em are all known. This evaluation involves only simple matrix
operations. If equality holds, v’"+1 can be stored and the common solution is given by em.”
= em + memldml; if not, v’"+1 can not be stored. Remove v""'1 from pattern list and

proceed to v""'2.

This algorithm is summarized as follows:
Step I: For the pattern v1, calculate C1, d1, el, and F1.
Step 2: For a new pattern (say v2), find C2, (12;, e2, and F2, based on (2.4.2). Check if

20

C2C2+d2 = (12 is satisfied. If yes, go to Step 3. If not, remove v2 form the given set of pat-
terns. Then go to Step 3.

Step 3: Check if all given patterns have been proposed. If not, choose a new pattern from
the given pattern list. Go to Step 2. If all patterns have been processed, the solution vector

is given by en.
2.5 Network Capacity, "Storable" and "Unstorable" Patterns

The network capacity deals with the maximum number of arbitrary patterns which
can be stored. In other words, the network capacity, denoted as NC, is the maximum num-
ber of m (the number of given arbitrary patterns) such that the set of equations (2.4.1) will

have at least one solution.

LetA = [A 11‘, AZT, AmT]T, b = [1211”, b}, me1T. Since A,- and b,-, i = 1, 2, m,
are of dimension anW and nXl, respectively, the matrix A and vector b are of dimension
manw and W]. By Linear Algebra, equation (2.4.1) always has solution(s) if mn S n».
This is because, in equation (2.4.1), the number of unknowns is greater than or equal to the
number of equations if mu 5 nw Thus, the capacity NC of the Hopfield-type feedback neu-
ral network based on our design approach is at least l-0.5(n-1)-I where L.-' is defined by

Lx-|=Max{ne Z: n.<_x} (2.5-1)

For large n, the network capacity NC can reach 0.512, which is greater than 01511. As
shown in [14], the capacity can reach n+1. But, the designed network in [14] is not sym-
metric. Since our synthesis results in a symmetric network, the capacity is expected to be

less than n +1.

21

As we stated in the Section 2.4, Morris-Odell’s Theorem can be viewed in a recursive
way. Suppose we are given m patterns, v1, v2, ..., v’". We want to find a symmetric matrix

W such that these m patterns are stored as equilibria. For each v‘, we can form A,- and bi.

For the matrix equations, Aix = b,- for i = l, 2, ..., m, we first check €2€2+d2 = d2
(since C1C1+d1 = d1 is always true). This process is repeated. Suppose Cka+dk = (1,, for k
= 2, 3, ..., p, where 2 .<. p S m and Cp+lC+p+1dp+1 :6 rip“. By Morris-Odell’s Theorem, Aix
= b,- for i = l, 2, ..., p have common solutions given by ep (assume 2 = 0), and A p”): = bp+1
will not have solutions shared by Air = b,- for i = 1, 2, ..., p (although A p+1x = bp+1 always
has solutions). In terms of neural network, the desired patterns, v1, v2, ..., v”, will be exactly
stored as equilibria of the Hopfield-type feedback neural network, but v1” 1 cannot. Thus,
we remove AP+1 from the list, A1, A2,..., Am and rearrange Ap+2, Ap+3, ..., Am as 8p”,
Bp+2’ ..., Bm,1,where Bp+1 =Ap+2, Bp+2 =Ap+3, «me-l =Am. Now, check C,C,-+d,- = di for
i = p+1, p+2, ..., m-l on Bp+l» Bp+2’ ..., Bm_1. This whole process is repeated until allAi’s
have been checked.

When this checking process is completed, we will get a subset of matrix equations,
A jx = bj for r S j S s, where 1 S r, s S m. Those matrix equations have common solution
which is given by e, (assume z = 0). We must note that this subset of matrix equations
depends on the order of checking these equations. The algorithm stated above is also sum-
marized in Section 2.4.

In terms of neural network, we deduce a subset of the desired patterns, {v’, r S j S s,
1 S r, s S m}, which will all be exactly stored as equilibria of the Hopfield-type neural net-
work (21..2). We call these patterns "storable" patterns, and the rest of patterns "unstorable"
patterns. Once the common solution x is found, the weight matrix W can be easily recon-

structed in the same way as we show in Section 2.3.

22

2.6 Parameter Determination for implementable Networks

In this section, we consider the implementable problem (defined later) of the
Hopfield-type feedback neural network with two-state neurons as defined in [21]. For this
type of network, one problem in the implementation is to determine whether a designed
network is implementable as an electronic circuit or not. Mathematically, we can find all
weights for any values of the network parameters if those given patterns can be stored.
The designed network based on this approach can be simulated by a digital computer, but
may not be readily implementable via a circuit since the resistance in the corresponding

neural circuit may be negative.

The mathematical model for the feedback continuous-time neural network [21] can

be expressed as

du

n
' 1
(“t-E; = 2 Wijvj+li- —ui (2.6.13)

r.
i=1 ‘

V.- = Mus) (2.6.1b)

fori= l, ...,n,wherec,-,r,~>0,u,-e Randvie (-A,A) withO<Ae R,s,-:R —) (-A,A) is
the sigmoidal function, n is the number of neurons in (2.7). The schematic diagram of a

circuit realization of (2.7) is shown in Figure 2.6.1.

 

Figure 2.6.1: The schematic circuit diagram of
a continuous-time nemal network.

23

Note that Rij and pi, for i, j = l, ..., n, are passive linear resistive elements, and Ci, i

= l, ..., n, are capacitors. Applying KCL, the equation of the circuit in Figure 2.6.1 can be

easily derived as follows:

d“: “i n 1
.— —- = _ o- e o 20 02
c‘dr +pi 311%“) “1) +1, ( 6 )
which is equivalent to
dui " l 1 n 1 )
c-— = ——v.+I-- _+ _ u. (2.6.3)
‘dt i§1Rii J 1 (pi iglRii '

1
We. = — 2.6.43
‘1 Rij ( )
1 1 ” 1
7—,- - E{-1- 2 R—.. (2.6.4b)

for i = 1, ..., n.

The network (2.6.1) can also be expressed in a matrix form:

Cdu/dt = Wv + 1 - R'lu (2.6.53)

v = 5(a) (2.6.5b)

 

24

where u = [u], ..., un]T, v = [121, ..., vn]T, W = [wij], C = diag(c1, ..., C"), R = diag(r1, ..., r"),

1 = [11, 1,]T, S(u) = [s1(u1), sn(u,,)]T.

For given desired patterns, v1, ..., v’", we expected to find weights, wij, i, j = l, ..., n,
such that those m patterns are stored as asymptotically stable equilibria of (2.6.1). From a
mathematical point of view, we can find all weights for any value of ri, i = l, ..., n, pro-
vided that all these m patterns, v1, ..., v’”, can be stored. This approach will not cause any
problems in computer simulation as pointed out in [14], but, the designed system may not
be implementable as in the circuit of Figure 1 if p,- < O for some i e {1, ..., n}. In this case,
p,- must be realized with a negative resistance. Therefore, for a designed system to be

implementable as the circuit in Figure 1, p,- > O for i = 1, ..., n.

For simplicity, we assume s,(.) = s0(.), and r,- = r0 for all i. This assumption is reason-
able for the circuit implementation. Consider a desired binary pattern, v = [v1, ..., van with
Vi = t) or m for 0 < t) < A. Define u = so'lw). s0'1(.) is an odd function, -u = so’l(-t)). The

relation between it and t) is depicted in Figure 2.6.2.

V = 3004)
.T ;

 

 

 

t:w

 

Figure 2.6.2: Relation between it and '0.

 

25

Assume I,- = O for all i. To be an equilibrium of (2.6.1), v must satisfy
Wv = R'IS‘1(v) (2.6.6)
where W = [wij] is the weight matrix, R = r0! which is a constant matrix, and S’1(v) =

[s0'1(v1), so‘1(vn)]T. In Section 2.3, we have shown that the weight matrix W can be

constructed as

0 x1 x2 ........ xn-z xn-l

x1 0 x" ....... XZn-a xbl-Z (2 6 7)
W: . .

xn-l x2n-3 xZia-3 -------- on 0

where nw = (n2 - n)/2, and x = [x], ..., xnwlT is given as

x = A+R'1S'1(v)

(2.6.8)
where At: AT(AAT)'1. From (2.6.8), we see that
urn” s iiA+llwilR’lllwliS'l(v)llw
s IIATII”II(AAT)'1||,°IIR'1llwIIS'1(v)llm (2.6.9)

For the desired binary pattern v, IIS'1(v)lL,, = p, IIATIIOO = 21). And tut-1",, = r0" since R =
r0]. It is easy to verify that .

26

n-l (D ------ (0
AAT - 112 03 n-l ...... (t)
to to ...... "-1

n-l w ------ w '1
(-)= 0) n-l ...... (o
(0 0) ------ n-l

Thus, II(AAT)’1|L,,, = u'ZIIOIIw It follows that

2
llxllms inane,
r01)

(2.6.10)

(2.6.1 1)

(2.6.12)

- As we stated earlier, a designed system is implementable if p,- > 0 for i = 1, ..., It.

From (2.6.4), it is observed that p ,- > 0 if

1 n
70 > z lwijl

j-
for i = 1, ..., n. Since

NW". 2 2 |w,-,1

i=1

for i = l, ..., n, we see that (2.6.13) is satisfied if

(2.6.13)

(2.6.14)

27

1
—>||W||.,
’0

for i = l, ..., It. From (2.6.7), we know that
IIWII“, S nilxlloo
Combining (2.6.16) with (2.6.12), we obtain

1 2n
IIWII,,.<.-r—0 (733nm;

Therefore, (2.6.15) will be satisfied if

211};

Tueuaa

which is equivalent to

3>2nueu..
u

(2.6.15)

(2.6.16)

(2.6.17)

(2.6.18)

(2.6.19)

The inequality (2.6.19) is an explicit constraint on the network parameters for the imple-

mentable circuit realization.

For a given binary pattern, v, the value of "Oil“, is constant. The inequality (2.6.19)

gives a sufficient condition for a designed system to be implementable, namely that ti/ti is

greater than some known constant. Some values of IIGII“, and 2nliGll” for different n are

listed in Table 2.6.1. From the data in Table 2.6.1, we observe that the value of 2ni|6li°°

28

decreases as It increases. In the circuit implementation, the ratio of 11/11 can be changed by

varying So ' (0).

Table l:
IIOII“ ZnIIIOII”

 

From (2.6.4), p,- can be expressed as

By applying (2.6.12), (2.6.14), and (2.6.16), it is easy to verify that

9:2 1 2 1

1 " 1+nwn
70+ Elwiji 7'— °°
.=1

 

0

1 r0

1 2 2n
—+nIIxII.. l+-—u||9||”

IV

 

 

(2.6.20)

(2.6.21)

Thus, by choosing a sufficiently large value of r0, p,- will be very large. Consequently, the

power dissipation of the entire network can be greedy reduced, which is one of the major

 

advantages of our approach.

Example: Let us look at an example. Suppose we want to store a pattern, v = [2.5,

29

2.5, 2.5, 2.5]T. Thus 1) = 2.5. The matrix A can be constructed. It is easy to verify that

 

 

 

 

F3 1 1 1 0.41677 -0.0833 -0.0833 -0.0833
9 = l 3 1 1 —0.0833 0.41677 -0.0833 -0.0833
1 1 3 1 —0.0833 -0.0833 0.41677 —0.0833
_1 1 1 3_ _-0.0833 —0.0833 -0.0833 0.41677

 

and "9",, = 0.6667. Thus, 2nl|O|l°° = 5.3336. Let r0 = 101(5). First, suppose the sigmoidal
function s0(.) is given such that it = 3. Since 11/]; = 0.8333, the inequality of (2.6.19) is not
satisfied. Since b-- 104[3, 3, 3, 31“, by (3.3), x = 4x10'5[1, 1, 1, 1, 1, if. The weight

matrix W can be constructed from (2.6.7) as

0 1 1 1"
w = 4x 10"5 1 0 1 1
1 1 0 l
_1 1 1 0]

 

 

By (2.6.20), p1 = p2 = [)3 = p4 = -50K9. Therefore, those resistors must be realized with
negative resistance. Now, increase the slope of the sigmoidal function s0(.) such that u =
0.4. In this case, (2.6.19) is satisfied. By applying the same procedure, we find that p1 = p2
= p3 = p4 = 12.7K.Q which is positive as we expect. Therefore, they can be easily irnple-
mented.

Now, consider the multiple-pattem case. Suppose we are given m binary patterns, v1,

30

..., v’". As shown in the Section 2.3, in order to find the weight matrix W, the following m

matrix equations, A 1x = ro'IS'1(v‘) for i = 1, ..., n, must have (at least) one common solu-

tion. The existence of such common solution can be checked by the Morris-Odell’s Theo-

rem. Without loss of generality, assume this common solution exists. Thus,

where b,- = ro'lS'1(v‘). It is easy to see that

 

 

T u
Illa-~41 .. = 7.
and
m T
n h-—-A:1|I..sg,u~u~=2mv
Moreover,

(2.6.22)

(2.6.23)

(2.6.24)

(2.6.25)

where v has the same format as O in (2.6.11). Following the same argument stated earlier

in this section, we see that

 

31

2mti
lell., 5 7-0?“ VII“, (2.6.26)
It follows that p,- > 0 if
E > 2mnn wll... (2.6.27)

where m is the number of given patterns, n is the number of neurons. The inequality
(2.6.27) shows that a designed network is implementable if the ratio of u/u is larger than

some known constant.

Now, let us consider the circuit implementation of this kind of neuron which will
make the inequality (2.6.27) true. As we have shown, the designed system will be irnple-
mentable if the ratio of the network parameters, 11/11, is larger than a specific quantity. This
ratio can be changed by varying the slope of the sigmoidal function around the origin.
Since it may not be practical to adjust the slope of the sigmoidal function on-line, it may
be desired to design a neuron circuit which has a very sharp slope around the origin. This
will guarantee the ratio of Wu to be very large. In this case, we expect our designed sys-

tem to be implementable.

As we stated early, the network power dissipation will be reduced if we choose large
values of r,-, i = 1, ..., n, in the network model. Another way to minimize the network

power dissipation is to design the neuron circuit such that it operates in sub-threshold [38].

A two-state neuron circuit is shown in Figure 2.6.3 which is built using inverters in

series. The input - output characteristic with sharp slope at the origin is shown in Figure

 

32

2.6.4. But, this circuit is not operating in sub-threshold. The network power dissipation

may not be sufficiently small.

Figure 2.6.3: A simple 2-state neuron circuit.

a

:73
w:
’6

:3

,_.
,5

 

Figure 2.6.4: The input-output characteristic of the

circuit shown in Figure 2.6.3.

For sub-threshold operation, we choose the neuron circuit as the transconductance
amplifier [38] shown in Figure 2.6.5. In this circuit, there is one current mirror and one
bias transistor with its gate voltage controlled by the bias voltage Vbias- The voltage V1 is
the input voltage, V2 is an internal voltage which is 0.7V in our simulation. Since the bias

voltage is set at 0.7V. We can reduce one power source if we let V2 and Vbias are equal.

33

 

Figure 2.6.5: A 2-state neuron operating in
sub-threshold.

Figure 2.6.6 shows the characteristic of the input - output. It is observed that the

transition voltage occurs at 0.7V which is the internal voltage V2.

3 41L‘ afrm_t__..a v1 .4 14 Al

‘1
i
i

 

Figure 2.6.6: The input-output characteristic of the

circuit shown in Figure 2.6.5.

34

The bias voltage, me, controls the amount of current flowing through the neuron
circuit. In our simulation, we set Vbias = 0.7V. Thus, the circuit operates in sub-threshold,
which minimizes the network power dissipation. The characteristic of bias current and

input voltage is shown in Figure 2.6.7.

 

Figure 2.6.7: The characteristic of bias current and input

voltage V, of the circuit shown in Figure 2.6.5.

Comparing the circuits in Figure 2.6.3 and Figure 2.6.5, the circuit in Figure 2.6.5

requires more voltage sources and one more transistor, but it operates below threshold.

 

35

2.7 Persistence of Equilibria under Weight Variations

In the Hopfield-type feedback continuous-time neural network [21], patterns are
stored as asymptotically stable equilibria of the network by choosing proper weights. As
we show in the Section 2.3 and Section 2.4, the weight matrix can be found by an analyti-
cal algorithm so that all patterns are exactly stored as equilibria. The Hopfield-type feed-
back neural network can be implemented via VLSI technology due to its structure
simplicity, but, in hardware implementation, there is inevitably a quantitative difl'erence
between the calculated and the implemented weights. This variation in weights has conse-
quent variation in equilibria. Indeed, possible bifurcations may arise, e.g. saddle-node
bifurcations ([9], [17]). Thus, equilibria may lose stability or it may disappear.

The feedback continuous-time neural network [21] is expressed as
Cdu/dt = Tx - R'lslbt) (2.7.1)
where C and R are both constant matrices and S(.) is a bounded C l "sigmoidal" function, x
= $04) = [so(u1), so(u,,)]T. 7' is the weight matrix which represents the calculated weight
matrix. Let T' be the implemented weight matrix. Thus, 8T = T ' - T is the matrix of per-

turbation on T. The perturbed feedback continuous-time neural network can be described

as

Cdz/dt=T'y -R'lS’1(y) (2.7.2)

where y = 5(2).

Let fl.) = r - R'1s1(.). Then (2.7.1) can be rewritten as

 

36

Cdu/dt =flx) (2.7.3)
And its perturbed system can be described as
Cdz/dt = g(y) (2.7.4)

where g(.) =f(.) + 8f(.), and Sfl.) = 8T which is a linear operator. Let x0 be an equilibrium
of (2.7.3). And let yo be the corresponding equilibrium of the perturbed system (2.7.4). Let

L(x0, yo) be the line segment connecting x0 and yo.

In general, the system is designed such that IDfixOM at 0 which makes 1:0 an isolated
equilibrium by the Inverse Function Theorem [52]. For the perturbed system (2.7.4), g(y)
=fly) + 87)). For the system (2.7.3) to have the property of persistence of equilibria, 5T has
to be small enough in the sense of a norm. This means that lDyg(x)i :6 0 for all x within a
neighborhood of x0, which implies that all the eigenvalues of Dfixo) must have non-zero
real parts.

Let us define the set
1) = {x e (-A, A)” I Dxflx) is non-singular}

Obviously, x0 e 1) since lDfixOM at 0. By the continuity of D fix), there exists a neighbor-
hood B(x°) ofxo such that B(x0) C 1).

Lemma 2.7.1:
Let x0 be an equilibrium of (2.7.3). There exists 0 < K < .. such that

37

||(DJ,,i“(z))°lllp S K for all z e B(x0) c D, where ||.||p is some norm in R".

Proof: Since the mapping (p: z —) Dxflz) is continuous and W = (-A, A)" is bounded, there

exists 0 < K1 < co such that

IlDfiz) - 13,10?)le 3 K1

for all z e B(x0). And since the mapping \y: Dxflz) —) (D,J(z))'l is continuous, the range is

also bounded. Thus, there exists 0 < K2 < 00 such that
ll(D,flz))'l - (Dfixon'lllp 5 K2

for all z e B(x0). It follows that
"(Dyan-10p s ||(D,f(x0))’lllp + K2

for all z e B(x0). Let K = |i(D,f(x0))'lllp + K2. Thus, for all z e B(x0), ||(D,f(z))'lllp s K.
This completes the proof. I

Theorem 2.7.2:

Let x0 and y° be the equilibria of (2.7.3) and (2.7.4), respectively. Let 6 > 0. Then
ilyo - roll], 5 8K if Iififllp s e and L(x°, yo) cB(x°), where K is defined in the proof of Lemma
2.7.1.

Proof: Since x0 and y0 are equilibria of (2.7.3) and (2.7.4), g(yo) =fly0) + my") = 0 =
flxo). It follows that fly”) -f(x0) = -8f(y°). By the Mean-Value Theorem [35], there exist 21,
z" e L(x0, yo) such that

38

Dy1.(z‘)

f (yo) -f(x0) (yo - x0)

Djn'(z")

DJ. <21)

= (yO -x0)

01142:)

= 0mm“ - x")

where z = [21,, 2",,1'1” e L(x°, y“). Since Let”, y”) c act") t: 19,0;(2) is invertible.
Thus,

yo - x0 = (Dflz))'l(f(v°) -f(x°))
= -(D,rtz»"8fly°)
which implies
llyo - xollp s II(D,f(z))'lupufif(y°)IIp (2.7.5)

Since 1(D,,t(z))'1np s K by Lemma 3.1 and IIIiif(y°)IIp s e for some t > 0 by assumption, llyo
- xolip S eK. This completes the proof. I

To apply Theorem 2.7.2, we need to estimate the value of K and the size of 30:”),

39

which is generally not easy for the general gray-level input patterns. In the following, we

will develop further results under the case of binary desired patterns.

LetxO = [x10, ..., xnolT be a desired binary pattern with xio = t) or -‘D for 0 < t) e R. In
Section 2.3 and Section 2.4, we have introduced an analytical learning algorithm to find a
symmetric weight matrix T of (2.7.3) such that x0 is exactly stored as an equilibrium.
Thus, in the following, we may assume that T is known and fixed. It is also shown in the
Section 2.2 that x0 is asymptotically stable if so ’ (so-1(0)) < (no, where so(.) is the sigmoi-
dal function of each neuron, (no is a small positive real number which can be evaluated

exactly.

Let too be the positive number defined above. Now, let us define

120,, = {x = [x1, x,,1T e (-A, A)" I so’ (so'l(x,-)) < (Do, i = 1, n} (2.7.6)

where A is the upper bound of the sigmoidal function so. In general, 9010 is non-empty due
to the "shape" of this sigmoidal function so(.), especially, when the neuron is realized by
CMOS inverters or op-amps. Let $01070) be a neighborhood of x0, where or is selected such
that $0100) = {x e (-A, A)" I lit - xollz < a} c 190,0. Obviously, a exists since so ’ (so'1(x,-O))
< coofor i = 1, ..., n.

Letx e can"). Since fl.) = r - R'lslt),

Dxflx) = 7' - R'ID,s*1(x) (2.7.7)

is invertible. We have the following lemma:

Lemma 2.7.3:
Let 2 e can"). If IISTIIP < 1 / II(T - R'leS'1(z))'lllp, T - 1240;1(2) + 57‘ is non-sin-

gular.

The proof of this lemma directly follows the fact that l18T(T - R'IDJ,,.S"1(2))'IIIp S

|8Tllplil(T - R'ID§1(2))'1up < 1.

Theorem 2.7.4:
There exists an open set 2 C Rnx” such that for T e fll, there exists x 6 00,0 which

satisfies flT, x) = Tx - R‘lS‘l(x) = 0.

Proof: Let 0 at z e pm There is T, 6 R” such that fir, , z) = 7', z - R'lS'1(z) = 0 as we
have shown in the Section 2.3. Since M = Tz - R'lDrS‘l(z) is non-singular by assumption,
by the Implicit Function Theorem, there is an open set )4, c R” containing T, such that
for T 6 Fl, there is a unique x e 19“,, such that flT, x) = Tx - R'lS'l(x) = 0. Define

a: u a, (2.7.8)

which is an open set. Therefore, for T e :4, there is some x 6 9m such that f(T, x) = 0.

This completes the proof. I

By the Implicit Function Theorem, there is a domain ‘le containing x, and a corre-
sponding domain ‘11;- containing T such that fix, T) = 0. Moreover, there is a unique func-
tion, say 9, such that x = 9(1) within ‘lex‘ZlT. In addition, there also exists a domain ‘Vx
and a corresponding domain ‘VT such that x 6 VJ, is an asymptotically stable equilibrium
satisfying fix, T) = 0 for some T e ’VT. Our efforts will be focused on the intersection of

‘Uxx‘ZlT and ‘VXX ‘VT.

41

In the following, we assume R = ml for simplicity. Let kl, ..., 1,, be the eigenvalues
of T, hm = maxUtl, ..., An}. Let K = D1510) for some x 6 12m. We assume that too is

small enough such that
(1) llroK'1T||2 < 1,
(2) (”O-1’01 > Amax

These two assumptions can be satisfied by choosing a small value of too. Let us look at an
example. Suppose the desired pattern is given as v = [1, -1, l, -1]T, S‘1(v) = diag(0.5, -0.5,
0.5, -0.5), ro = mm It can be found that km = 0.5x10'4 and iiroTll2 = 0.5. Therefore,
the assumption (1) and (2) will be satisfied if too < 2. In general, as far as stability is con-
cerned, the value of coo is very small, e.g., too = 0.01. Thus, assumption (1) and (2) will
always be satisfied by choosing a sufficiently small value of too.

Lemma 2.7.5: _
Let T 6 R” be a symmetric matrix. Suppose assumption (1) and (2) are satisfied.

Then, for x 6 9,1,0,

|l(T -r(,'ID,,s*1(x))'1n2 < "(T - (oo'lro'lb'lllz

Proof: Let x e (190,0. Define so'lbq) = Iq, 1' = 1, n. Thus, D,S'1(x) = K = diag(lq, kn).
Since x 6 Two, kt > (no1 for i = 1, ..., n. Since T is symmetric, there exists an orthogonal
matrix P such that T = PAPT, where A = diag(hl, ..., A”) and where h,- is the ith eigenvalue
of T.

It is observed that

42

"(7' - (no'er-ln'ln, = "(PAPT - too-#0404"2
= "(A - (coho-10'1", = 1 /((oo'1ro‘1 - kw)
Since liroK‘lTilz < 1,
"(T - nflxfln,

nnrln.
—1’llror1“2"T"2

 

-1
rOkmin

-1
1 - rOkmin” A||2

 

= 1 / (ro'lkmyn - Am”)
Since km > cool, 1 /(ro'1k,,,,.,, - am) < 1 / (coo'lro'l - km) which implies that
"(T -ro’1o,.s*1(rt))'1n2 < "(r - too-#0404112
This completes the proof. I

Let us define

or

no = ll (T-m51r511)'1”2(a+v,/r_1) (2.7.9)

 

43

As we know, flT, x0) = 1x0 - ro'lsloto) = 0, and finer, y) = (r + 8T)y - ro'lS'l(y). In the
following, assume that T+8T e 31 which is defined in (2.7.8). This assumption is generally

satisfied due to the particular nonlinearity of the sigmoidal function, so(.).

Theorem 2.7.6:
There exists y 6 Taco) such that flT+5T, y) = 0 if IISTIIZ S Bo.

Proof: Suppose ll8Tl|2 S Bo. Since T+8T 6 Fl, by Theorem 3.4, there is y e 1)“, such that
flT+8T, y) = 0. Since $0,055 c 00,0, there are two cases: y 6 $0,060) or y 6 0w\ 30,00).
The first case is desired. We want to show that y e 130,0 \ (80,00).
Suppose flT+5T, y) = 0 with y 6 90,0\ $a(x0). Then
(T + snot“ + by) - ro’lS'1(x0 + By) = 0
where y = x0 + 8y. It follows that
(r - ro-lo,s"1(z) + 5T)5y + 8Tx0 = 0

where z 6 L06", y). By Lemma 2.7.5,

"or", s 80 s 1 /u(r - oio'lro'llyln2

s 1 / "(r -ro'lo,1r1(z))'1n2

By Lemma 2.7.3, the matrix T - row; 1(2) + ST is non-singular. Thus,

by = - (r - ro'leS’1(z) + 8T)'15Tx0
It follows that
||5y||2 = "(T - ro'leS'1(z) + 81')‘15er112

s "(7' - mingle) + SD'lllzllfiTllzllxollz

< H (T- ra‘Dxf‘ (z) )“ll,u arnzvh‘t
— 1‘" (T461051 (2))'1l|2||57‘||2

 

which implies that y = x0 + 8y 6 Bauo), contradicting the assumption y e @010 \ 30100)-
Thus, there exists y 6 $0,060) such that f(T+8T, y) = 0. This completes the proof. I

Let 8 > 0. Define

8

(1)./5+ a) II (T - 0165511) -1ll2}

 

70 = Min {[30, (2.7.10)

We have the following theorem:

Theorem 2.7.7:
Lott0 be a binary equilibrium of (3.3), y0 6 60,60) be an equilibrium of (3.4). If

45

"57112 5 yo, then,
llyO - x0", 5 e if [30 2 e / [(b(n)1’2+ot)n(r - wo'lro'ln'lllzj; (2.7. 1 la)
llyo - x0", 5 80(u(n)1’2+ot)u(r - too-#0404112 otherwise (2.7.11b)

Proof : Since x0, y0 6 Taco), Dfiz) is non-singular for z 6 L06", yo). As we know, Dj(z)
= r - ro'lDrS'1(z). By equation (3.5), Lemma 2.7.5, and the fact that "you2 s llxollz + ii8y|l2
s u(n)1’2+ot,
llyo — x0|l2 s (o(n)l”+a)ll5T|lzll(DJ/(z))'lll2

s (U(n)1/2+(1)’YOII(T -ro'll),,.s*1(z))'1n2

s (o(n)m+or)yoll(T - hobo-10'1",
If [so 2 e / [(u(n) 1”+6000: - trio-#0404112], yo = e / [(b(n)1’2+ot)n(r - oro’lro'lb'lllz].
Thus, llyo - x0", s e. Otherwise, yo = 130- In this case, uy0 - x0112 s Bo(t)(n)m+a)ll(T -
(oo'lro'llflliz. This completes the proof. I

Collecting all the previous results together, we obtain our final major result:

Theorem 2.7.8 (Persistence of Equilibria):
Letxo be a binary equilibrium of (2.7.3). Suppose 23“,, at e and II8TII2 5 yo. Then

(1) There exists an equilibrium y0 of the perturbed feedback continuous-time neural net-
work (2.7.4) such that y0 6 Taco).

46

(2) Ilyo - x0112 satisfies (2.7.11).

(3) y0 is asymptotically stable.

It is observed that one has to check that @010 at O in order to apply Theorem 2.7.7.
As we stated earlier, 130,0 is always non-empty for the sigmoidal function so(.). Thus, the
equilibria of the feedback continuous-time neural network (2.7.3) and its perturbed net-
work (2.7 .4) are very close as long as IIOTIIZ is relatively small. Moreover, we are able to

estimate the upper bound on llyO - xollz once we know the upper bound on IISTIIZ.

Let us consider an example. Suppose we are given three patterns:

 

 

 

 

 

 

Let the sigmoidal function so be chosen such that so’1(1) = 3, and so'1(-1) =—3. By using

the analytical recursive algorithm in [2], we find weight matrix T as follows:

T=r0

 

 

It can be easily estimated [2] that too = 0.1. Thus, "(7' - too-#0404", = 0.14294, Based
on the sigmoidal function selected, we choose or = 1.5. It follows that Bo = 3ro‘l. And, it
can be calculated that yo = Min{3ro'l, Zero'l }. It follows that yo = 0.24,:1 if e = 0.1. By
Theorem 3.7, My" - x0", s 0.1 if 1157112 s 0.2ro'l. Obviously, lly° - x0", 5 0.01 if 1157112 5
0.02r0'1.

47

2.8 A CMOS Circuit of 2n-State Neuron Operating in
Subthreshoid

The multiple-state neural network can be used to process gray-level images. In the
associative memory network, patterns are stored as asymptotically stable equilibria, and
then retrieved by some other patterns. As we show in the Section 2.2, neurons with flat
regions are capable of storing gray-level images which are asymptotically stable equilibria
of continuous-time feedback neural networks. The key to their stability is the slope of the
flat region. In other words, for the given pattern v = [v], ..., vn]T, it is asymptotically stable
if

606103)) < .1 (2.8.1)

ri 2 Iwijl

j=l,j¢t‘

 

for i = 1, 2, ..., n. Therefore, the pattern, v = [v1, ..., vn]T, is asymptotically stable if the sig-
moidal function so(.) has flat regions at so'l(v,-), i = 1, ..., n. To make this pattern v asymp-
} totically stable, we can adjust so ’ (so'1(vi)) for i = l, ..., n, based on (2.8.1). In practice, it is
wise to pre-design the neuron circuit such that the slope so ’ (so'1(vi)), for i = 1, ..., n, is

very small. In this case, (2.8.1) will be easily satisfied

Figure 2.8.1 is a schematic diagram of the 2n-state neuron circuit. Figure 2.8.2
shows the subcircuit serving as a building block. There is a current mirror shared by all
building blocks [59]. In general, n building blocks contribute 2n voltage regions where the
corresponding currents are flat. Each block has its own bias transistor which provides suf-
ficient bias current. All bias transistors are controlled by a common bias gate voltage Vb.
By setting small value of Vb (e.g., 0.7V), the neuron circuit operates in sub-threshold. In

each block, the two transistors forming the middle leg are assumed to have identical sizes,

48

and so do the other remaining transistors.

 

 

 

Figure 2.8.1: 2n-state neuron circuit operating in sub-threshold.
There are n building blocks which share a common current
mirror. The gate voltage V), controls all bias transistors.

Figure 2.8.2: The sub-circuit
serving as a building block.

Figure 2.8.3 is a schematic diagram of the eight-state neuron circuit. This neuron cir-
cuit consists of four building blocks. Figure 2.8.4 is the corresponding Ia-Vin characteris-
tic. This curve- shows consecutive flat regions. First the controllable flat regions at the
chosen voltages V1 = 1V, V2 = 2V, V3 = 3V, V4 = 4V. These regions separate very flat
regions which are due to current saturation of each block. Since the slope of the curve in
the saturation region is very small, all patterns selected in this region are bound to be

asymptotically stable.

49

 

 

(“TI £5142] Elia] iv: V4
1 ‘1 l 1

Figure 2.8.3. Schematic diagram of the 8-state neuron circuit.

 

 

 

,_.
j
I;
t
0
a.)

 

Figure 2.8.4: The I,,-V,-,l characteristic of 8-state neuron
with W: 80, w: 1.

To understand the behavior of this 2n-state neuron circuit, let us analyze the circuit
shown in Figure 2.8.5 which is similar to the modified transconductance circuit in [12]. In
the sub-threshold region of operation [38], 11 = onexp((ltV,-,l - V)/Vo) and 12 = onexp(lth
- V)/Vo), where Io is a fabrication parameter, the value of K is about 0.7, and Vo = kT/q =

25mV. For simplicity, we will omit Vo in the expression of current since Vo basically

50

 

Figure 2.8.5. The modified
transconductance circuit.

scales all voltages. The bias current can be expressed as I b = onexp(ltVb)(l - exp(-V)).
The current flowing through the middle two transistors is Iq = WIoexp(lth-)(exp(-VS) -
exp(-Vj)) = WIoexp(xV,-n)(exp(-V) - exp(-VS)). By eliminating Vs, [Q can be solved as

 

xv“ (V1-y
, = Wise v V (2.8.2)
4 at “+2: 1
Sincer =11+12 +Iq, we have
xv“ xv)
fl = a": ‘ +‘ (2.8.3)

(Vuetvi

e‘v‘ (cm: + e‘vi) + cz‘v“ + cm'1+ (2 + :‘1) e
W

And since 10,- = 11 - 12 = on(exp(KV,-,,) - exp(KVJ-))exp(-V), we know

51

2xV

fa-

2V
lcV K’

‘ e

 

I .= wloe

a, (2.8.4)

V V V 2 V V W V V
e‘c '(e‘c “'-l-e'c ’) +e K "+e2x ”r- (2+—)ex "ex’
w

In the 2n-state neuron circuit shown in Figure 2.8.1, all building blocks share the same
current mirror. Consequently, the output current, 10, is the summation of all I oj: j = l, ..., n,

i.e.,

n 21W“ 2xV

n
ttV e -e ’
I = 21.: w] e ‘ 2
0 01 0 V V V V v w
j=l i=1 c“(c""+e"1)+c2""+e2"l+(2+—)e
W

 

o, ..v, (2.8.5)
12

The slope of the Io-Vin characteristic at V]- is given by

 

 

 

 

 

B dlo n KWIoeKV. 286
= = ( . . )
J ”in V; j=1 2+o.5 (g) +e"(V*’VI)
In this paper, for simplicity, we choose w = 1. Thus,
xV,
p], = :1; = n ”0‘ W _V) = i _KV “0 _KV (2.8.7)
in V} i=12+0.5W+e ‘ ’ i=1¢ '(2+0.5W)+c I

It is observed that B]- is inversely related to W which is the ratio of width and length

52

of those middle transistors. In order to have small value of B], we can make those middle

transistors having larger ratio of width and length. Figure 2.8.6 shows some curves with W
= 20, 40, 60, and 80. It is seen that the slope [31- becomes smaller as Wincreases. In general,
we desire the slope of each flat region to be small. This can be satisfied if we design the

2n-state neuron circuit with larger value of w (e.g., W = 80 or W/w = 80.).

._.
1

c.
S
o
.3
r:
a:
L
L
:1
L)

t‘

1.3
Voltage input

 

Figure 2.8.6: Four Ia-Vin curves of 4-state neuron circuit with
different values of W. The curves from left to right have

W: 20, 40, 60, and 80. It can be seen that the curve with

W = 80 has the smallest slope in the flat region.

Compared with the multiple-state neuron circuit in [60], this circuit is much simpler:
Moreover, this circuit operates in the subthreshold region. Thus, the power dissipation is

expected to be very small.

53

2.9 Computer Simulation of 4-State Neural Network
with 16 Neurons

We have built a Hopfield-type neural network with 16 neurons. Each neuron has

four-state. The four-state neuron is realized by the following sigmoidal function:

s(x) = s1(x) - 32(x) - s3(x) + s4(x) (2.9.1)

where

4e2(x-1)

 

s (x) =
l _ _ ..
2(x 1) 2(x 1)

 

 

4e—2(x+1)
320‘) = 2(x+1) -2(x+1)
e +e
S3“) = 4(x-1) -4(x—1)
e +e
2e-4(x+1)
s4(x) =

 

e4(x+1) +e-4(x+1)

The characteristic of s(x) is shown in Figure 2.9.1. It can be determined that s(l) = 1,
s(-1) = -1, s(1.9) = 1.9, and s(-l.9) = -l.9. Thus, the four states of this neuron are defined
as: 1.9,1, -l, -l.9.

54

 

Figure 2.9.1 The characteristic of a 4-state neuron.

Suppose we are given 4 gray-level patterns:

   

Pattern A

 

 

 

 

 

 

Pattern D

55

The numerical values associated with the gray-levels are defined as:
1.9 1.5 1

Based on this set of gray levels, we obtain the pattern vectors associated with these

3

0 -l -1.5 -l.9

 

 

four desired patterns:

Pattern A: [1.9,1,1,-1.9,-1,1.9,-1.9,1,-1,-1.9,1.<.l,1,-1.9,-1,-1,1.9]T
Pattern B: [1.9, 1, -1, -1.9, 1.9, 1, -1, -1.9, 1.9, 1, -1, -1.9, 1.9, 1, -1, -1.9]T
Pattern C: [1.9,1.9,1.9,1.9,1,1,1,1,-1,-1,-1,-1,-1.9,-1.9,-1.9,-1.9]T
Pattern D: [1.9,1.9,1,1,1.9,1.9,1,1,-1,-1,-1.9,-1.9,-1,-1,-1.9,-1.9]T

For simplicity, we choose R = C = In. By applying the recursive algorithm shown in

the Section 2.4, we find the weight matrix W which is given as follows:

W:

0.00 0.28 0.18 -.24 0.08 0.22 -.21 -.13 0.13 -.06 0.12 -.08 -.14 -.18 -.28 -.25
0.28 0.00 0.10 -.01 0.07 0.13 -.04 0.00 0.00 -.05 -.01 -.07 -.09 -.10 -.13 -.16
0.18 0.10 0.00 0.25 -.08 -.05 -.02 0.03 -.03 —.06 0.13 0.08 -.22 -.19 -.10 -.09
-.24 -.01 0.25 0.00 -.03 -.39 0.42 0.13 -.13 0.06 -.18 0.03 -.22 -.25 0.01 -.36
0.08 0.07 -.08 -.03 0.00 0.19 0.08 0.01 -.01 0.01 -.29 -.22 0.12 0.08 -.07 -.21
0.22 0.13 -.05 -.39 0.19 0.00 -.03 0.22 -.22 -.22 -.19 -.19 -.05 0.05 -.13 0.14
-.21 -.04 -.02 0.42 0.08 -.03 0.00 0.13 -.13 -.01 -.24 -.08 0.01 0.02 0.04 -.10
-.13 0.00 0.03 0.13 0.01 0.22 0.13 0.00 -.33 -.20 -.12 -.01 -.17 -.03 0.00 0.24
0.13 0.00 -.03 -.13 -.01 -.22 -.13 -.33 0.00 0.20 0.12 0.01 0.17 0.03 0.00 -.24
-.06 -.05 -.06 0.06 0.01 -.22 -.01 -.20 0.20 0.00 0.00 -.01 0.16 0.06 0.05 -.18
0.12 -.01 0.13 -.18 -.29 -.19 -.24 -.12 0.12 0.00 0.00 0.29 -.15 -.13 0.01 0.20
-.08 -.07 0.08 0.03 -.22 -.19 -.08 -.01 0.01 -.01 0.29 0.00 -.12 -.08 0.07 0.21
-.14 -.09 -.22 -.22 0.12 -.05 0.01 -.17 0.17 0.16 -.15 -.12 0.00 0.22 0.09 -.08
-.18 -.10 -.19 -.25 0.08 0.05 0.02 -.03 0.03 0.06 -.13 -.08 0.22 0.00 0.10 0.09
-.28 -.13 -.10 0.01 -.07 -.13 0.04 0.00 0.00 0.05 0.01 0.07 0.09 0.10 0.00 0.16
-.25 -.16 -.09 -.36 -.21 0.14 -.10 0.24 -.24 -.18 0.20 0.21 -.08 0.09 0.16 0.00

56

All of those four patterns have been stored as asymptotically stable equilibria. To

show this, we choose the Pattern A as an example:

 

 

 

 

 

In the circuit implementation, the implemented neuron function is usually not an odd
function, like the multiple-state neuron circuit proposed in the Section 2.8. In the follow-
ing, we will simulate the Hopfield-type feedback neural network with 16 neurons. Each
neuron has four states. The output of each individual neuron is positive. This neural func-

tion is chosen as follows:

s(x) = 5*(tanh(5*(x-1)) + tanh(5*(x-2)) + tanh(5*(x-3)) + tanh(5*(x-4)))/8
(2.9.2)

57

s(x) is shown in Figure 2.9.2. The four states are chosen as 5, 3.75, 1.25, 0. It can be

determined that s(4.5) = 5, s(3.5) = 3.75, s(1.5) = 1.25, and s(0.5) = 0.

 

 

 

 

 

Figure 2.9.2 The characteristic of a 4-state neuron.

The numerical values associated with the gray-levels are defined as:
5

4.25 3.75 2.5
Based on this set of gray levels, we obtain the pattern vectors associated with those

       

0.75 0

four desired patterns given early:

Pattern A: [5, 3.75, 3.75.0, 1.25, 5, 0, 3.75, 1.25, 0, 5, 3.75, 0, 1.25, 1.25, 5]T
Pattern B: [5, 3.75, 1.25, 0, 5, 3.75, 1.25.0, 5.3.75, 1.25, 0, 5, 3.75, 1.25, 0]T
Pattern C: [5, 5, 5, 5, 3.75, 3.75, 3.75, 3.75, 1.25, 1.25, 1.25, 1.25, 0, 0, 0, 0]T
Pattern D: [5, 5, 3.75, 3.75, 5, 5, 3.75, 3.75, 1.25, 1.25, 0, 0, 1.25, 1.25, 0, 0]T

58
For simplicity, we choose R = C = In. By applying the recursive algorithm shown in

the Section 2.4, we find the weight matrix W which is given as follows:

0.00 0.15 0.14 0.12 0.14 0.21 0.07 0.08 0.17 0.10 0.17 0.09 0.10 0.07 0.05 0.08
0.15 0.00 0.13 0.18 0.13 0.19 0.09 0.10 0.08 0.06 0.07 0.04 0.05 0.04 0.02 0.03
0.14 0.13 0.00 0.34 0.01 0.03 0.09 0.11 0.06 0.04 0.21 0.14 -.08 -.06 0.02 0.03
0.12 0.18 0.34 0.00 0.08 -.16 0.28 0.11 0.01 0.08 -.04 0.05 -.17 -.20 -.06 -.29
0.14 0.13 0.01 0.08 0.00 0.28 0.13 0.07 0.10 0.09 -.18 -.11 0.19 0.13 0.01 -.08
0.21 0.19 0.03 -.16 0.28 0.00 0.08 0.32 -.11 -.10 -.05 -.05 0.09 0.19 0.02 0.28
0.07 0.09 0.09 0.28 0.13 0.08 0.00 0.08 -.01 0.02 -.13 -.04 0.00 -.01 -.02 -.10
0.08 0.10 0.11 0.11 0.07 0.32 0.08 0.00 -.l9 -.12 -.01 0.03 -.12 -.02 -.01 0.15
0.17 0.08 0.06 0.01 0.10 -.11 -.01 -.19 0.00 0.27 0.23 0.08 0.27 0.12 0.08 -.08
0.10 0.06 0.04 0.08 0.09 -.10 0.02 -.12 0.27 0.00 0.09 0.02 0.17 0.07 0.04 -.11
0.17 0.07 0.21 -.04 -.I8 -.05 -.13 -.01 0.23 0.09 0.00 0.35 -.03 -.01 0.10 0.28
0.09 0.04 0.14 0.05 -.11 -.05 -.04 0.03 0.08 0.02 0.35 0.00 -.07 -.04 0.04 0.15
0.10 0.05 -.08 -.17 0.19 0.09 0.00 -.12 0.27 0.17 -.03 -.07 0.00 0.17 0.05 -.06
0.07 0.04 -.06 -.20 0.13 0.19 -.01 -.02 0.12 0.07 -.01 -.04 0.17 0.00 0.03 0.04
0.05 0.02 0.02 -.06 0.01 0.02 -.02 -.01 0.08 0.04 0.10 0.04 0.05 0.03 0.00 0.05
0.08 0.03 0.03 -.29 -.08 0.28 -.10 0.15 -.08 -.11 0.28 0.15 -.06 0.04 0.05 0.00

All of those four patterns have been stored as asymptotically stable equilibria. In

fact, the following two noisy patterns will retrieve the Pattern A:

 

 

 

 

 

59

' 2.10 A Schematic Layout for an N-Neuron Feedback
Neural Network

As we know from section 2.6, the mathematical model for the feedback continuous-

time neural network [21] can be expressed as

dui " 1

(2.10.1)

vi = s,- (u,)

for i = 1, ..., n. A schematic layout for a four-neuron neural network is shown in Figure

2.10.1 where the capacitor and resistor connected to each individual neuron are not shown.

Output

 

V

 

Input I >
ii

 

 

 

 

 

 

 

 

 

D -- neuron I -- weight

Figure 2.10.1. A schematic layout for a four-neuron network.

The neuron in Figure 2.10.1 can be either two-state or multiple-state. Additional inverters

are needed in order to realize the negative weights [21]. The general schematic layout for

60

an n-neuron feedback neural network with four-state neurons are shown in Figure 2.10.2.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

I!" J" {EL [5: *1
11:4 .
.. J i
F“ 1” J—D Stu ii

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 2.10.2. A schematic layout of an n-neuron network with fora-state neurons.

In this general circuit, the voltage states are controlled by V1 and V2. The voltage

Vbias provides the bias voltage for all bias transistors.

CHAPTER 3
Cellular Continuous-Time Feedback Neural Networks

with Multiple-State Neurons

3.1 Background

The mathematical model for a Hopfield-type continuous-time neural network is

d“: n 1
J=1 ’
(3.1.1)

fori= 1, 2, ..., n, where ci, ’i >0, uie R and via (-A, A) with 0<A e R. s,- : R —) (-A, A)
is a monotone increasing C 1-function with si(0) = 0, s,(x) = A (or -A) if and only if x ->

+00 (or —oo). n denotes the number of neurons in the network (3.1.1).

In compact matrix form, (3.1.1) can be written as

du_ __1
E—Wv R u

(3.1.2)
v = s (u)

where u = [u], ..., uu]T e R”, v = [v1, ..., vn]T e R", C = diag(cl, ..., c") e Rm‘", R =
diag(rl, ..., rn) e Rm“, and s(u) = [s 1(u1), sn(u,,)]T e R", and T denotes the transpose of

matrix.

For the network with n neurons, the dimension of the pattern vector is n. In the cir-

cuit implementation of the Hopfield-type neural network, the size of each network is quite

61

62

limited due to the high interconnectivity; in other words, the value of n cannot be too
large. In this sense, it is impossible to build a neural network circuit in a single chip with

as many neurons as is often required in real applications, such as pattern recognition.

The cellular neural network (CNN) (for example, [10], [33]) provides a useful
approach to connect small networks (cells) together. In this large network, each cell is
treated as an individual group performing certain tasks. Due to the large number of neu-

rons in the large network, the network capacity is expected to be large.

In the design of CNN, one needs to consider the structure of the large network, its

Stability and the way to find weights. In this chapter, we will address those basic questions.

3.2 Structure of the Cellular HopfieId-Type Neural Networks

First, we consider the case of two small networks. Suppose we are given two small

Hopfield-type neural networks, S 1 and S2, which are described as follows:

S1:
du4 " 1
1 l _ 1 1 1___ 1
1 _ 1
vi — si(ui)
andS2:
do? N
2 2 2 2 2
'=1 l'

(3.2.2)

 

63
for i = l, 2, ..., n, where n denotes the number of neurons in each network. These two small
networks, S 1 and 32, can be viewed as two independent electronic chips with a fixed inter-

nal Structure. Assume I,1 = 1,-2 = 0.

Let us combine these two small networks to form a large network. The correspond-
ing large network model is expressed as:

dul

Cldt' = W1v1+A1v1+A2v2- (lt‘)"u1

(3.2.3)

C2— = W2v2 +2121:1 +A3v2 - (R2) ”11:2
(3.2.4)
v2 = s(uz)

where Aj = diag(alj, ..., aNj) for j = 1, 2, 3.

Define v = [(vl)T, (v2)T]T, u = [(u1)T, (u2)T]T. Then, it and v satisfy the following

equations:

C3}; = Wv + Kv -R'1u
(3.2.5)

v =S(u)

where w = diag(wl, W2), 0 = diagc(C1, C2), R = diag(R1,R2), and

K = A1142 S(u) = s(ul)
212.43 s(uZ)

64

The model in (3.2.5) represents the cellular Hopfield-type neural network with two cells.

The diagram associated with the model of (3.2.5) is shown in Figure 3.2.1.

 

 

 

 

 

 

 

 

 

Fig. 3.2.1 Hopfield cellular neural network built on two small network

Now, let us extend the structure of the network with two small networks to M small
networks. Suppose we are given M small Hopfield-type neural networks. The cellular

Hopfield neural network is described as follows:

0ka = w"v"+ iA'vk- (R*)"‘u"
dt [:1 "
(3.2.6)
v" = s(uk)

where

Ck = diag(clk, ..., cn"),

11,1 = diag(akr’, obj); .43- : Al} for i, j = 1, 2, M,
R" = diag(rlk, ..., rn"),

W‘t is the weight matrix of the kth small network,

k = l, ..., M.

The overall system is defined as:

cfl‘ = Wv-i-Kv—R'lu
dt

v=S(u)

where

C = diag(cl, C“),

W = diag(Wl, W“),

R = diag(Rl, R”),

u = [041)”, (uM)T]T.

v = [(v1)T. (vMiTlT.

$04) = l(s(u1))T. (s(uM»TlT.

' I 2 M‘

A1 Al 00. Al

1 2 M

K: A2 A2 .42
l 2 M
_AM AM .00 AMJ

 

 

 

(3.2.7)

It is easy to see that the dimension of K, C, R, W is Mann, and the dimension of u and v

is Mn. This means the cellular Hopfield-type neural network represented in (3.2.7) has Mn

neurons. It is observed that the matrix K is symmetric.

' The network (3.2.7) can be transformed to

du_ _ _1
Ca; — (W+K)v R u

v=S(u)

(3.2.8)

66

which has the same structure as the Hopfield-type neural network (3.1.2) if we treat the
matrix W + K as the weight matrix. Therefore, the stability results in the Section 2.2 about
the Hopfield-type neural network (3.1.2) can be all applied to the network (3.2.8). More—
over, by the same argument in Section 2.5, the capacity of this large network (3.2.7) is at

least 0.5Mn. Thus, the network capacity is increased by at least M times.

In a real large network design, K is a symmetric constant matrix. Theoretically, the
elements of K can be arbitrary. But, in the circuit implementation, those values should be
small to reduce the network power dissipation since they are equivalent to the conduc-

tance. One may choose the elements of K in the same order of network local resistance.

In particular, for the larger network with adjacent connections, the matrix K can be

expressed as:

 

 

' O A? O o “
A; .. 0 MA; . .. O
K = ........ : 5...“ "5.5.94...
0 o o 0 . Ali—1
_o o O .. Ag-1 0

Thus, the larger network (3.2.7) represents the locally connected network, which is also

referred to as cellular neural network [10].

In this locally connected larger network, each individual cell can also be locally con-

nected. The mathematical model of this locally connected smaller network is:

0.35: = Wv—R'lu (3.2.9)

 

67

where
P W11 W12 0 0 9
W12 W22 ....... W23 °°°°°°°°°° 0
W = '1‘: ................... . .. “a...“
0 0 0 ............. lSalim-1. Wn-l n
-0 O O ...... ”3n, n — l wn n .i

 

 

By the same argument in section 2.3, for a desired pattern, v, the weight matrix W can be

 

 

by the following equation:
Px = b
where
r E .1
v1 5 v2
v2 0 g v1 v3 0
P = 5 V2
0 5 U vn_2 vn
‘ v E v
- " E n ‘ 1-i n x (271 - 1)
w” 1 S l S n
xi =
Wi-n, i+I-n n+1 S i S 2 n-l

Therefore, the recursive algorithm presented in the section 2.4 can be applied.

3.3 A Parallel Algorithm for Finding Weights

Let v be an equilibrium point of the cellular Hopfield-type neural network (3.2.7).

 

68

Then v satisfies

Wv+Av—R‘IS'1 (v) = 0 (3.3.1)

which is equivalent to

M
W1v1+ EAIIVI — (R1)-ls'l(v1) = 0
1:1

........................................... (3.3.2)

M
WMvM + 22131;“ - (RM)'1s'1(vM) = 0
l = l
where v = [(v1)T, ..., (vM)T]T, T represents the matrix transpose,. It is observed from the
above equations that Wk, k = 1, , M, are independent during the "learning" process,
where each Wk represents the weight matrix of the kth small network. In other words, Wk

can be solved from the following linear equation:

W‘v" = —§A§,v" + (R‘)“:." (3.3.3)
I: 1
The algorithm discussed in the Section 2.3 can be used to find the weight matrix wk for
the given pattern v. Since those M matrix equations in (3.3.2) are all identical in terms of
weight matrices, all weight matrices can be found in the same way. Moreover, those
weight matrices can be found in parallel since all equations are independent in terms of

weight matrices. In this sense, the weights for all cells will be found simultaneously.

For multiple patterns case, we can apply the Morris-Odell’s theorem to find all
weight matrices. The recursive algorithm discussed in the Section 2.4 can be applied in the

cellular Hopfield-type continuous-time feedback neural networks.

CHAPTER 4

Dendro—Dendritic Artificial Neural Network
with Multiple-State Neurons

4.1 Background

The mathematical model of the dendro-dendritic neural network is given as:

n
l
j=l,j¢i ‘

V.- = S.- (“1) (4.1.1b)
for i = l, ..., n, where n denotes the number of neurons in the network; ui is the input of the
ith neuron, v,- is the output of the ith neuron; Ci is the capacitor and r,- is the resistor con-
nected to the ith neuron; s,(.) the activation function which is a bounded monotone C1-

function. One type of circuit configurations of the dendro-dendritic neural network for

three-neuron is shown in Figure 4.1.1 where connections are made by resistors.

VDD Von

V1 . ‘ ul R 2 “2 . E
c c 4'
bel :

__ R23 '1’. 7
R31 ‘ V'”
113 ' . V3
@9113
'9' -.- '1-

? at .
Figure 4.1.1: One type of circuit configurations of DANN with three neurons.

L5

 

r‘;' V
E

69

70

It is reasonable to assume ci = co, r, = ro and si = so for i = 1, ..., n. Thus, (4.1.1) can

be rewritten as follows:

I!
1
CO; = . 2 .Tij(uj_ui) +Tii(vi-ui)+1i_;6ui (4.1.23)
J=ld¢l

V.- = 3004;) (4.1.2b)

In compact matrix form, (4.1.2) reads

coda/dt = - Wu + Tv + I ' (4.1.3a)

v = 5(a) - (4.1.3b)

where W = [Wij]’ WU“ — -le for l #1. and W17: 2 Tij+ 731T = diag(Tu, Tn"),

i=1

I = [11, 1,]T, u = [111, 11,11", v = [v1, v,,]T, and S(u) = [so(ul), so(un)]T.

The energy function is defined as [49]:

’5’2:y=_1i,.,rif“i“i 2T $300)“ 21““ 22(’°+21’ju')2

i=1 i=1 i=1 j=l
(4.1.4)

or in vector form:

E(u)=luTWu-1Tu- 27.17130“) dx (4.1.5)

i=1 0

 

71

which is continuously decreasing with respect to time t and vanishes only at an equilib-
rium of the network (4.1.3). This also implies that no limit cycle (or chaos) of (4.1.3)

exists. Now define the Hessian matrix at a point it as JVE(u) = W - TDuS(u).

Since the activation function so, of each neuron, is a bounded Cl-function, (4.1.3)
has a unique solution for t 2 0 starting from each initial condition u(0). As shown in [49],
system (4.1.3) is a gradient-like system where the isolated local minirna of (4.1.4) are
asymptotically stable. And, by the Inverse Function Theorem, u is isolated if u is an equi-
librium of (4.1.3) such that det(JVE(u)) = det(W - TDuS(u)) at 0. Moreover, it is a local
minimum of (2.4) if JVE(u) > 0. It is also shown in [49] that all trajectories of (4.1.3) con-
verge to a bounded and closed (compact) region. Hence, there is only finite number of iso-
lated equilibria of (4.1.3). In summery, the dendro-dendritic neural network has the

following important properties:

(1) For the energy function defined in (4.1.4), dE(u)/dt S 0, and dE(u)/dt = 0 if and only if

u is an equilibrium point of (4.1.3).

(2) There exists a unique solution of (4.1.3) for t 2 0 for each initial condition u(0).

(3) Isolated local minimum of E(u) is an asymptotically stable equilibrium of (4.1.3).

(4) There are only finite number of equilibria of (4.1.3).

(5) Let u0 be an equilibrium point of (4.1.3). u0 is isolated if det(JVE(u0)) at 0.

(6) Let u0 be an equilibrium point of (4.1.3). u0 is a local minimum of function E(v) if and
only if the Jacobian matrix JVE(u0) > 0.

 

72

(7) All solutions are bounded.

(8) No limit cycles nor other forms of recurrent solutions exit.

Thus, the dendro-dendritic neural network has all the desired properties of the
Hopfield-type neural network. But, compared with the Hopfield-type neural network, the

dendro-dendritic neural network has the following advantages:

(1) Less number of weights. For an n-neuron network, the Hopfield-type neural network
uses a maximum of n2-n weights, the DANN network uses a maximum of n(n+1)/2

weights.

(2) Natural symmetric weights. In the DANN network circuit, “’0' = wfi naturally. In the
Hopfield-type neural network, this property can never be true due to circuit hardware mis—

match.

The circuit shown in Figure 4.1.1 has connections made by resistors. In the neural
chip design, resistors take large area. Thus, we expect to replace the resistors by some
other circuit devices, like for example MOS transistors. Another type of three-neuron
DANN network is shown in Figure 4.1.2 where all neurons are connected by nMOS tran-

sistors which are controlled by their gate voltages.

 

 

Von
V1 ' 1‘1 -
vol-l IE h 1, 6.13.: ha
? '9- (;3V‘( Ei’von v.21)
“3
EFVM

Figure 4.1.2: Another type of circuit configurations of DANN with three neurons.

 

 

73

4.2 Stability Analysis

In the following analysis, we assume so: (-oo, +oo) —> (A1, A2) for A2 2 A1 > 0, so(0)
= 0, and the desired patterns are binary, e.g., v = [v1, ..., anT with v,- = v- or v.... Corre-
sponding to v = [V1, ..., van, we have u = [u1, ..., u,,]T with u,- = u- or up, where so(u_) = v-

and so(u+) = v+.
For the energy function defined in (4.1.5), it is easy to verify that

VE(u) = Wu - I - TS(u)
ng(u) = W - TDuS(u)

where VE(u) is the gradient and JVE(u) is the Hessian of E(u). A stability theorem is

stated as follows:

Stability Theorem: Let 11° be an equilibrium of (4.1.3) where To- 2 0 i #2 j and T,- > 0.
Then u0 is asymptotically stable if so ' (uio) < 1 / (70TH) for all i.

Proof : Suppose Ti]- 2 0 for i it j, Ti,- > 0. Define B = W - TDuS(uO). Then, by,- = w,:,- +
Til-so ' (uio) = Z T ,-,-+ r3l - T ,-,-s’o (14?), bo- = my = - Til for i ¢ j. Suppose so ' (uio) < 1 / (roT,-,-)
i=1

for all i. Then, bi,- > 0. It is easy to see that Ibo-l > z lbui: By Gershgorin Circle
j: 1,1'nj
Theorem [13], B has only positive eigenvalues. Therefore, JVE(uO) > 0, which implies u0

is a local minimum of E(u). Thus, u0 is asymptotically stable. This completes the
proof. I

74

4.3 An Analytical Method for Finding Positive Weights

via Linear Programming

First, consider the single pattern case. Let u = [u], ..., uan be an image pattern,
where ui = u, or 14+. In order to be an equilibrium of the dendro-dendritic neural network
(4.1.3), it must satisfy:

(4.3.1)
- Wu+TS(u)+I=0

which is equivalent to
(4.3.2)

W'u+TS(u)=R'lu-l

where wij’ = - wij = To for i at j, wii' = - W“. + r04, 7' = diag(r‘l 1, TM), R’1 = r045",
where En is the identity matrix, I = [11, ..., I n]T. For simplicity, one may set I ,- = 0 for i = 1,
..., n. Thus, (4.3.2) results in:
(4.3.3)
W ' u + TS(u) = R'lu = ro'lEnu

By proper algebraic operations, (4.3.3) can be transformed into
(4.3.4)

where x = [x1, ..., xN]T, N = n(n+1)/2 (the number of unknowns in (4.3.4)). and b = ro’lu,

75

_so(u1) -u1 O

 

 

=
fl
l
t
N
0
O
“‘\““‘\““‘\“‘

 

A =
O O .. un-un-l
_ so(un) _un ul—un ,un—l-un _

. Till 1SiS n

T1,i+l-n n+1 S I S 2’1'1
xi = , . . (4.3.5)

L \

Tn-Ln l=N

Without loss of generality, assume so(ui) at “1' for i = l, ..., n. From the structure of A

in (4.3.4), it is easy to show the following proposition:

Proposition 4.3.1: Let u = [u1, ..., an]T be an image pattern with u,- = u- or n+1: 0, A as
defined in (4.3.5). Then, rank(A) = n.

The minimum-norm solution of (4.3.4) is given by x = A+b where A" = AT(AAT)'1.
Once x is solved, all weights, Tij for i, j = 1, ..., n, can easily be reconstructed via (4.3.5).
Those weights will make u exactly stored as an equilibrium of (4.1.3).

Now, consider the case of multiple patterns. Suppose we are given m binary patterns,

it], ..., am. For each vector u', we have

76

- wit" + TS(u‘) = 0

which can be transferred to the equivalent matrix equation: A ix = bi. Thus, in order to find
weights, Tij for i, j = 1, ..., n, for storing those m patterns, Aix = bi, i = 1, ..., m, must have

common solutions. In other words,

 

 

 

 

A1 b1
2 2
A x = b (4.3.6)
A'" 8'"

must have solutions. In the Section 2.4, we have shown how to apply the Morris-Odell’s
Theorem to check the existence of solutions and find common solutions of (4.3.6). Thus,

we may assume that common solutions of (4.3.6) exist.

Define A = [(A1)T, (Am)T]T, b = [(1:1)T, (bm)T]T. Then, the common solutions
of (4.3.6) is given by x =A+b + N(A), where MA) is the null space of A. Based on the Sta-
bility Theorem in Section 4.2, we wish to find non-negative solutions of (4.3.6). In other

words, we need to find solutions of:

(4.3.7)

It is observed that the solution of (4.3.7) is not unique. The elements of the solution
x are the weights in our neural network. It is desired to have small values of those weights
so that they can be implementable as large resistive elements to achieve minimized power

dissipation in the network. In other words, the problem can be stated as:

77

‘ I!
Minimize 2 xi (4.3.8a)
i: 1
st Ax = b (4.3.8b)
x 2 0 (4.3.80)

which is a standard linear programming problem.

The solutions of (4.3.8b) and (4.3.8c) are called feasible solutions. An optimal feasi-
ble solution is one which satisfies (4.3.8). An optimal basic feasible solution is an optimal
feasible solution which has exactly p non-zero elements, other elements are all zero,

where p = rank(A). It can be Shown [34] that

(l) The feasible solution set K is convex.
(2) If K is non-empty, there exists at least one extreme point of K.
(3) The convex set K possesses at most a finite number of extreme points.

(4) Optimal values of the objective function occurs only at extreme points of K.

Without loss of generality, assume so(u+) > u, and so(u,) < u,. This assumption can
be always satisfied since we are free to set the values of u+ and u-. With this assumption,

we have the following theorem:

Theorem 4.3.2: Let u = [u], ..., u,,]T be a binary pattern with ui = u, or u. Then, there

exists an optimal basic feasible solution of (4.3.8).

Proof : Let A be defined as in (4.3.4). Partition A = [A1, A2], where A1 = diag(so(u1)-u1,

so(un)-u,,). Define x1 = Al'lro'lu = diag(l/(so(u1)-u1), l/(so(un)-u,,))ro'1u. Then, x =

78

[x1, 0]T is a basic feasible solution of the linear programming problem (4.3.8) with A and
b defined as in (4.3.4). Thus, the feasible solution set K is non-empty, which implies that

an optimal basic feasible solution of (4.3.8) exists. This proves the result. I

Theorem 4.2 guarantees the existence of optimal basic feasible solution of (4.3.8) for
single input pattern. In the multiple-pattem case, the situation is quite different. How to
find conditions which guarantee the existence of the feasible solution of (4.3.8) for multi-
ple-pattems case remains as a future work. Real-time experiments [47] have shown multi-

ple binary patterns can be stored and retrieved.

Among all numerical algorithms about linear programming, the simplex method is
the most significant algorithm. It proceeds from one basic feasible solution to another in
such a way as to continuously decrease the value of the objective function until a mini-
mum is reached [34]. Standard packages of the simplex method are available. Therefore, it
is easy to find weights for the dendro-dendritic neural network once multiple patterns are

provided.

79

4.4 Computer Simulation of a Three-Neuron DANN Network

Consider a three-neuron dendro—dendritic neural network. The binary input pattern
takes two values: 5 volts and -5 volts. And define u+ = 2 volts, u- = —2 volts, which implies
so(u+) = 5 > u+ and so(u-) = -5 < u-. Suppose a binary pattern v = [5, 5, -5]T is given. Cor-
responding to this pattern, u = [2, 2, -2]T. Since the number of neurons is three, the maxi-

mum number of weights N = 6. The matrix A in (4.3.4) is constructed as:

Let ro = 1 K9. Then b = ro'lu = [0.002, 0.002, -0.002]T. The linear programming

problem is formulated as follows:

Minimize x1+ x2 + X3 + X4 + x5 + x6,
s.t. Ax = b,

x120,x220,x320,x420,x520,1:620.

The solution [x1, x2, x3, x4, x5, x6]T = [0.00067, 0.00067, 0.00067, 0, 0, 0]T is an
optimal basic feasible solution. Thus, the weights are: T11 = 0.00067, T22 = 0.00067, T33 =
0.00067, T12 = 0, T13 = 0, T 23 = 0. The resistors of 1.5 K52 are needed to implement those
weights. From the dendro-dendritic neural network (4.1.1), we know there are no connec-

tions among the three neurons. In other word, for this example, the network is decoupled. ‘

From a mathematical point of view, [x1, x2, x3, x4, x5, x6]T = [0.00067, 0.00067,

0.00067, 0, 0, 0]T is an optimal basic feasible solution which minimizes the quantity x1 +

80

x2 + x3 + x4 + x5 + x6. But from an implementation point of view, this optimal basic feasi-
ble solution may not yield good robust performance, e. g. fault tolerance, since the network
with those weights are basically decoupled. There is a trade-off between small values of
weights and good system performance. If we focus on a better system performance, we
may only want to find feasible solutions, instead of optimal basic feasible solutions. In this
example, we can find a solution [x1, x2, x3, x4, x5, x6]T = [0.002, 0.002, 0.002, 0.001,
0.001, 0.001]T which is only a feasible solution. The corresponding weights are: T11 = T22
= T33 = 0.002, T12 = T13 = T23 = 0.001. In this case, the network is not decoupled. We can
also define the objective function (4.8a) as to minimize {x12 + ...+ xNz} and apply nonlin-
ear programming algorithms. This will result in optimal solutions which will not make the

network decoupled.

81

4.5 Network implementation and Layout of Three-Neuron
DANN Network

We will design the DANN network with multiple-state neurons. For simplicity, we
will only design a network with three neurons. Each neuron has four states. In the Section
2.8, we have shown the design of the CMOS circuit of 2n-state neuron. The schematic dia-

gram of four-state neuron with voltage input and voltage output is shown in Figure 4.5.1.

 

 

file if

Vent

 

 

 

 

 

WM” V”

IL IL'IL-fl wm-eo

 

 

 

 

 

 

 

 

 

 

 

 

 

 

ll'

11‘
13v GIL IL
-i-

T

 

 

 

Figure 4.5.1 The schematic diagram of 4-state neuron with
voltage input and voltage output.

In this circuit, the resistor 320K and the voltage source 2.0V in the output stage con-
verts the current to voltage. A large resistance is required in order to vary the output volt-
age over the range of 0 - 5V. The bias voltage is set at 1.2V. One may reduce this bias
voltage. The smaller the bias voltage, the larger the resistance in the output stage. In the
chip design, one may implement this large resistor outside the chip since this large resistor

will require a large layout area.

82

The PSpice simulation of this four-state neuron circuit is shown in Figure 4.5.2.

s.tlv 7-------------------------------------------~--------------------------;

 

 

Figure 4.5.2 The PSpice simulation of the circuit in Figln'e 4.5.1.

In the DANN network, the "sigmoidal" function does not necessarily need to be an
odd function. Thus, the four states can be defined at u = 0.5V, 2V, 3V and 4V. With this
four-state neuron, we build the DANN network shown in Figure 4.5.3.

Unit 1 Unit 2

 

Figure 4.5.3. The DANN network with three neurons.

83

In this network, each neuron has self-feedback. All neurons are fully connected.
These connections are made possible by MOS transistors which are controlled by the gate

voltages. These gate voltages can be set globally.

We have made some PSpice simulations for this DANN network. In those simula-
tions, we set all gate voltages at certain level. Some initial voltages are applied to the input
nodes. After the transient process is finished, the output voltages of all three neurons con-
verge to desired values. Figure 4.5.4 shows the transient curves of the output voltages of

those 3 neurons.

  
 

 

 

8.0V
I
I
2'" ........ A ............... ¢ 1
WV :
z . ,
I
I i
I .
‘ 'w 1' .................................
I
I ; ‘ i _ _ ' l
l f " , l
I 1
0V ------- l- -------- r- ------ w- ------ '1 -------- 1 -------- 1 -------- 'r -------- I
_ 0e 0.600 Mlle flue Mlle “in ljue the “no

Figure 4.5.4. The transient craves of the output voltages of three neurons.

In this simulation shown in the Figure 4.5.4, we set the initial input voltages of unit
1, unit 2 and uint 3 at 3.5V, 1.8V and 0V, respectively. The steady state of the output of
unit 1 converges to 4.306V, unit 2 to 1.912V, and unit 3 to 0.516V. These three voltages,
0.516V, 1.912V and 4.306V are the three states of our multiple-state neuron shown in the
Figure 4.5.1 with some small shift in voltage. The gate voltages of self-feedback transis-

84

tors are all equal to 5.0V. For other three connection transistors, their gate voltages are set

at 1.5V.

The network layout of the circuit in Figure 4.5.4 is shown in Figure 4.5.5. In this lay-
out, the resistor and power source in the output stage of each neuron is not included. A
large area of the chip is required in order to implement this large resistor. To test this chip,

one can externally connect the resistors and power sources to the chip through pins.

 

Figure 4.5.5. The layout of the circuit in Figure 4.5 .4.

85

4.6 A Schematic Layout for an N-Neuron DANN Network

_ As we know from Section 4.1, the mathematical model for the DANN network [49]

can be expressed as

d“.- " 1
j: 1,j¢i ‘ (4 61)
Vi = Si (ui)

for i = 1, ..., n. A schematic layout for a four-neuron neural network is shown in Figure

4.6.1 where the capacitor and resistor connected to each individual neuron are not shown.

 

D -- neuron I -- weight

Figure 4.6.1. A schematic layout for afour4-netmon DANN network.

The symbol of neuron in Figure 4.6.1 can be either two-state or multiple-state. The

weights can be realized by either resistors or nMOS transistors. The general schematic

86

layout for an n—neuron feedback neural network with four-state neurons are shown in Fig-

ure 4.6.2.

 

 

Figure 4.6.2. A schematic layout of an n-neuron DANN with four-state neurons.

In this general circuit, the voltage states are controlled by V1 and V2. The voltage

Vbias provides the bias voltage for all bias transistors.

CHAPTER 5

A Tiny-Chip Layout of a Six-Neuron DANN Network

5.1 Computer Simulation of a Six-Neuron DANN Network

We have built a DANN neural network with six neurons. Each neuron has four-state.

The four-state neuron is realized by the following sigmoidal function:

s(x) = 0.847(tanh(5(x — 0.5)) + 1) + 0.415(tanh(5(x - 1.5)) + 1)
+ 0.405(tanh(5(x - 2.5)) + 1) + O.389(tanh(5(x - 3.5)) + 1) (5.1.1)

for x e [0, 5]. In the expression of (5.1.1), all parameters are chosen such that the charac-
teristic of s(x) is approximately matched with the characteristic of the four-state neuron
circuit shown in next section. The characteristic of s(x) is shown in Figure 5.1.1. The four
states are chosen as: 1V, 2V, 3V and 4V. It can be measures that s(l) = 1.694V, 3(2) =
2.514V, s(3) = 3.325V, and 3(4) = 4.103V.

 

4.6

 

 

 

 

2 8 4 6
hurt

0 I

Figure 5.1.1. The characteristic of sigmoidal function s(x).

87

88

Suppose we are given the following four-level pattern:

 

The numerical values associated with those four gray levels are:

4.103 3.325

 

 

 

 

 

 

1.964

Based on this set of gray levels, the pattern vector is given as
[4.103, 3.325, 2.514, 3.325, 1.694, 4.103]T
The corresponding input pattern vector is
[4, 3, 2, 3, 1, 4]T
For simplicity, we choose R = r01. It can be found that

x =r0'1[44.95, 9.51, 3.02, 9.51, 0.014, 44.95, 0.09, 0.09, 0.09, 0.09,
0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09,]T

is a feasible solution of (4.3.8). Thus, the weights are:

T11 = 44-95704: T22 = 9.517'0'1, T33 = 3.02,-0'1, T44 = 9.51,.0-1,
T55 = 0°014r0-12 T66 = 44.9570-1, T12 = 0.091-0'1’ T13 = 0.09,.0-1’

89
T14 = 0.09r0'1. T15 = 0.09r0'1, T16 = 0.09r0'1, T23 = (109,04,
T24 = 0.09ro'1, T25 = 0.09r0'1, T26 = 0.09ro'1, T34 = 0.09, 0-1,
T35 = 0'09ro'1, T36 = 0-09ro". T45 = 0095‘, T46 = 0.094,-1
T56 = 0.09r0’1.

To test whether this pattern has been stored as asymptotically stable equilibrium, we
use the pattern itself and its noisy patterns to retrieve the stored pattern. The numerical

values for the noisy patterns are defined as:

3.92 3.13 2.14 1.56

   

 

We found that the given pattern

 

can be retrieved by itself and the following noisy patterns:

 

which shows that the given pattern has been indeed stored as asymptotically stable equi-

librium.

90
5.2 PSpice Simulation of a Six-Neuron DANN Network

with Resistor-Connections

We have designed a six-neuron DANN network circuit in which all connections of
each individual neuron and other neurons are mode by resistors. Each neuron has four

states. We use this DANN circuit to store the four-level pattern given in Section 5.1.

The schematic diagram of this six-neuron DANN network is shown in Figure 5.2.1.
In this circuit, the resistors of each individual neuron are identical with resistance r0. Since

we store the pattern given in Section 5.1, the values of all resistors are found as:

R11 = 0.0222r0, R22 = 0.105r0, R33 = 0.332r0, R44 = 0.105r0,
R55 = 69.4r0, R66 = 0022er R12 = 11.11r0, R13 = 11.1er,
R14 =11.11r0, R15 = 11.11r0, R16 = 11.11r0, R23 = 11.11r0,
R24 = 11.11r0, R25 =11.11r0, R26 = 11.11r0, R34 = 11.11r0,
R35 =11.11r0, R36 = 11.11r0, R45 = 11.11ro, R46 = 11.11ro,
R56=11.11r0.

where r0 is the local resistance. In Figure 5.2.1, the neuron circuit shown by the box serves
as a building block. Its schematic diagram is depicted in Figure 5.2.2 in which, in addition
to VDD and GND, the signals V1, V2, V0 and Vbias are shared by all other neurons. The
input-output characteristic of the neuron circuit is shown in Figure 5.2.3 where the volt-
ages 1V, 2V, 3V and 4V are chosen as 4 states in the multiple-state neural network. It can

be measured that

s(l) = 1.694V, s(2) = 2.514V, s(3) = 3.325V, s(4) = 4.103V

Its input-output is very similar to the one in Figure 5.1.1 as far as the four states

91

Al“
V‘m

 

Neuron

 

 

 

#1 %_

 

 

 

.WF
AAAAA
‘ E

4

 

 

 

 

 

E
i:
c? ,.
a;
5
I
[W

 

 

Neuron

 

 

 

 

 

 

 

 

 

 

 

 

 

Lib" .

 

1:
3_
fig ‘E
:

l—wvw—

 

 

 

 

 

 

i:
ti
rm-

 

E

L..._, L..._. L?" L...,__. 7,... .
l
r...

 

 

 

 

 

 

 

 

 

 

 

 

AI—
g_
as. ‘E
8
i
I
rm-

Yl Y) Yo

Figure 5.2.1. The schematic diagram of a six-neuron DANN circuit.

92

 

“““

 

 

I
l
l
l
l
l
l
l
l
l
l

 

 

 

 

n‘
H

 

 

 

 

 

 

 

 

 

 

ILA-l

 

 

 

 

 

 

 

“ya-n

a.

11'
IL

 

 

 

 

 

 

llll'llll'll

 

VI | VIIIV v.4.”

Vi"

Figure 5.2.2. The schematic diagram of a four-state nem'on circuit.

45v -r--------------

    

........

------.----P--..------

.------------'-------------‘-..----------¥-.

  

13V

33V

89V 43V

, uv

Figure 5.2.3. The input-output characteristic of a four-state neuron circuit.

93

are concerned. This similarity allows us to use the theoretical calculation results to build

our circuit.

To test this circuit, we set the initial voltages at the input nodes of each neuron as:
neuron #1 - 4.3V, neuron #2 - 3.2V, neuron #3 - 2.2V, neuron #4 - 2.8V, neuron #5 - 0.9V,
neuron #6 - 3.6V. After the transient process is finished, the final voltages at the corre-
sponding nodes converge to: neuron #1 - 4V, neuron #2 - 3V, neuron #3 - 2V, neuron #4 -
3V, neuron #5 - 1V, neuron #6 - 4V, which represents the stored pattern vector: [4, 3, 2, 3,
1, 4]T. This experiment shows that the designed 6-neuron DANN circuit works well as

expected. The simulation result is shown in 5.2.4.

SflN

- ------4
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I

-L/
4-
I
I
I

I

llnfl

P

 

 

I}...

 

 

 

any 1 - c - e :
f I

I I

I I

l I

I— - . I

I ' At 4 ; _ - '

2.0V -{ ' I
I I

I I

I I

I I

I I

I I

I I

my 4 k c 1
I i

I I

I I

I I

I I

l I

l I

w+ ------------- 1 ............. ‘ ............. fl .............. r ............. '
0. Nu. 4000 “no “It. 100m

Figure 5.2.4. The simulation of a 6-neuron DANN network circuit.

It is noted that the resistor r0 of each neuron has to be very large. In this simulation,

we choose r0 to be lMegQ

94

The network in Figure 5.2.1 also stores other patterns. To show this, we set the initial
input voltages at arbitrary values. Figure 5.2.5 shows that the network converges to the
pattern [4, 2, 2, 3, 1, 4]T. And Figure 5.2.6 shows that the network converges to the pattern
[4, 2, 2, 2, 1, 4]T.

 

 

 

 

 

  
    

-J-------

in

 

 

In

I.“ ‘5
i
I
I
I
I
‘" + ........ r ............... q- ....... W ooooooooooooooooo ‘ oooooooo ' ........ '
0- 60m IOU. 1“». ”In so» two- .IOIII a...

Figure 5.2.6. The network converges to pattern [4, 2, 2, 2, 1, 4]T.

95
5.3 PSpice Simulation of a Six-Neuron Network with
nMOS Transistor-Connections

A neural network can be used to store given patterns by having proper weights. This
requires that those weights are programmable. The disadvantage of the circuit with resis-

tor-connections shown in Figure 5.2.1 is that the weights are not programmable.

As shown in [49], there is another type of configuration of DANN where all connec-
tions of each individual neuron and other neurons are made by nMOS transistors. In this
configuration of DANN, the nMOS transistors are used as the programmable weights. The

values of weights can be set on-line by varying the gate voltages of those transistors.

An nMOS transistor used as a programmable weight is shown in Figure 5.3.1.

V8
Vs —L|__| Vd

Figure 5.3.1. An nMOS transistor used as a programmable weight.

For the fixed gate voltage, the characteristic of Ids - Va: is plotted in Figure 5.3.2
where Vgs = 5V. The level of Ids mainly depends on Vgs, Vds’ and W/L. There are basically
two regions in the I d, - Vds region: ohmic region and saturation. The nMOS transistor can
be used as a nonlinear resistive device if it operates in the ohmic region. The resistance of

a nMOS for fixed Vgs is determined by:

dVds 1

 

 

R =.
d“ d1“ d’ds (5.3.1)
dVds

96

      
    

 

 

Id:
Iv 3
,0 I Vgs = 5V
‘ J Ohmic Region Saturation Region
,.. 5
2-- 5
,.. E
.. g
In as u u u a; n I; a u u V43

Figure 5.3.2. The 14, - V4, characteristic with Vgs = 5V.

Therefore, the level of Rds depends on Vgs,

Rds- And, Rds decreases as W/L increases.

V43, and W/L, too. The larger Vgs, the smaller

The schematic diagram of a six-neuron DANN network with nMOS transistor-con-
nections is depicted in Figure 5.3.3. In this circuit, the neuron circuit shown by the box
serves as a building block. Its schematic diagram is depicted in Figure 5.2.2 in which, in
addition to VDD and GND, the signals V1, V2, V0 and Vbias are shared by all other neu-
rons. The input-output characteristic of the neuron circuit is shown in Figure 5.2.3 where
the voltages 1V, 2V, 3V and 4V are chosen as 4 states in the multiple-state neural network.

It can be measured that
s(l) = 1.694V, s(2) = 2.514V, s(3) = 3.325V, 3(4) = 4.103V

Its input-output characteristic is very similar to the one in Figure 5.1.1 as far as the four

states are concerned.

VON

 

“h vnu v.

Figure 5.3.3. The schematic diagram of a six-neuron DANN with nMOS transistor-connections.

98
A PSpice simulation of this DANN network in Figure 5.3.3 is shown in Figure 5.3.4.

 

 

 

t‘v1 ....... . . . . ........

 

I I
I IV + -------------------------------------------------------------------- l
to nu «In an «a Iqu mu III. III-- Time

Figure 5.3.4. A simulation result of the six-neuron DANN network in Figure 5.3.3.

In this simulation, the resistor r0 of each individual neuron is chosen as 1000K
which is the same as it is in the simulation of Figure 5.2.4. All the gate voltages are: VG11
= 4V, V622 = 3V, VG33 = 2V, V644 = 3V, V655 = 2V, V666 = 5V, V612 = 3V, V613 =
2V, V614 = 4V, V615 = 1V, V616 = 4V, V623 = 2V, V624 = 3V, V625 = 1V, V625 = 3V,
V634 = 2V, VG35 = 1V, VG36 = 2V, V645 = 1V, V646 = 3V, VG56 = IV. The initial input
voltages in the order of neuron #1 - neuron # 6 are: 4.2V, 3.2V, 2.2V, 3.2V, 1.2V, 4.2V.
From Figure 5.3.4, we know that the final input voltage of neuron #6 does converge to
about 4.1V which is expected. The final voltages of neuron #2 and neuron #4 converge to
about 3.6V. The other three final input voltages converge to 3.7V as well. As we see, the

voltage states of each neuron in the circuit of Figure 5.3.3 has shifted from desired states.

This simulation shows that the six-neuron DANN with nMOS transistor-connections
has the potential to store gray-level pattern. Due to the nonlinearity of nMOS transistor
used as resistors, some on-chip learning circuits have to be designed if we expect the
DANN network with nMOS transistor-connections store desired gray-level patterns. This

work is left for the future research.

99
For the same set of gate voltages, we test the network in Figure 5.3.3 with difierent
initial conditions. Figure 5.3.5 shows the simulation result with the initial input voltages in
the order of neuron #1 to neuron #6: 4V, 1V, 1V, 2V, 2V, 3V.

 

 

 

 

Figure 5.3.5. Simulation of the network in Figure 5.3.3 with initial input
voltages: 4V, 1V, 1V, 2V, 2V, 3V.

Figure 5.3.6 is another simulation of the same network with the set of initial input voltages
given as: 3V, 2V, 1V, 4V, 4V, and 3V.

l" T""'°"""""""""""""°""°""'°"""'""""'°'°"'”"3

 

 

 

I
‘” ‘b-.----.-'---....*-----.-* OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO .

Figure 5.3.6. Simulation of the network in Figure 5.3.3 with initial input
voltages: 3V, 2V, 1V, 4V, 4V, 3V.

100
We also test the network in Figure 5.3.3 with different gate voltages with the initial
input voltages given as the desired patterns: 4V, 3V, 2V, 3V, 1V, 4V. Figure 5.3.7 shows the
simulation with the gate voltages of all self-feedback transistors given as 5V, all others are

held the same as before.

u"

 

 

Ifl‘ ......... r ....... .1... ....... .1 .......................... ‘ ........ ' ........ .

Figure 5.3.7. Simulation of the network in Figure 5.3.3'with the gate voltages
of all self-feedback transistors set at 5V.

Figure 5.3.8 shows another simulation of the same network with the gate voltages of

all self-feedback transistors set as 1V, all others are the same as before.

 

 

 

 

 

 

 

 

 

 

Figure 5.3.7. Simulation of the network in Figure 5.3.3 with the gate voltages
of all self-feedback transistors set at 1V.

101

5.4 A Tiny-Chip Layout of a Six-Neuron DANN Network

A VLSI layout was designed for a six-neuron DANN network circuit on MOSIS
Tiny—Chip as a prototype chip. This layout is fabricated by MOSIS using an n-well 2pm
CMOS process with 2.2 x 2.2 m2 chip area in a 40-pin package.

Among those 40 pins, 4 pins are used as VDD and GND. The rest of 36 pins can be
used to access the internal signals. For this 6-neuron DANN network, there are total 2x6 «1-
6x(6+1)/2 = 33 gate voltages. Thus, 33 pins will be used to access those gate voltages and
input, output voltages. The other three pins are used to connect Vbias’ and two-state volt-
ages, V1 = 1V and V2 = 3V. Those three voltage signals are shared by all six neurons. The
resistor and voltage source in the output stage of each neuron are connected to the chip

externally.

The schematic diagram of this six-neuron DANN circuit is depicted in Figure 5.4.1,
which has been designed to maximally use the limited chip area. Each neuron is repre-
sented by the black box which circuit is shown in Figure 5.2.2. Based on this configura-
tion, the VLSI layout of this six-neuron DANN network is designed, which is shown in
Figure 5.4.2. In this layout, the resistors and capacitors of each individual neuron are not

included. The pin configuration of this DANN chip is shown in Figure 5.4.3.

This six-neuron DANN chip will be sent to MOSIS for fabrication. All internal sig-
nals can be accessed through pins. To test this chip, one needs to connect a 200K resistor
and 2.5V power source to the output of each individual neuron to convert four-state output
current signal to four-state voltage signal. The bias voltage has to be set at 1.2V in order to
have the four-state input-output characteristic of each neuron. This characteristic can be

measured by setting all gate voltages to 0V and measm’ing input-output relation.

102

«a»

.3!

an.» 39»

v5»;

as» 33» 33»

a;

I»

 

a» $9»

*3» a»

Figure 5.4.1. The schematic diagram of a six-neuron DANN network.

103

 

Figure 5.4.2. The layout of a six-neuron DANN chip.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

10

 

 

 

ll

 

 

12

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

“5
v6
V066

“6

VG 1 5
V025
V016
V626
V036
V046
V656
V512

V045

GND
V03 5
ul

V01 1

V1

104

Four-State Neurons

Ip with SIX

DANN Neural Ch

 

 

V05 5

 

 

V5

 

“4

 

3| alts [e

 

V644 ‘

 

 

V4 36

 

 

 

VDD 35

 

 

 

 

Vcl 33

 

 

 

Vbias 32

 

 

V014 31

 

 

 

 

 

 

 

 

65‘
8

 

 

GND

N
UI

IBIBBIS’L

 

 

V033

 

“3

 

 

V2

 

 

V022

 

 

Figure 5.4.3. The pin configure of the six-neuron DANN chip.

 

CHAPTER 6

Summary, Conclusion, and Future Work

This work has focused on the design and analysis of two types of continuous-time
feedback neural networks, namely, the Hopfield-type feedback neural network and the
Dendro-dendritic artificial neural network (DANN). The networks considered in this dis-

sertation mainly have multiple-state neurons.

For the Hopfield-type continuous-time feedback neural network with multiple-state
neurons, we have proposed an analytical recursive algorithm to find weights which will
exactly store the desired gray-level patterns as asymptotically stable equilibria. We intro.
duce a criterion which will ensure the designed Hopfield-type neural network to be irnple-
mentable. Moreover, the resistance in the network can be made as large as necessary. We

estimate the variation of equilibria based on the variation of weights.

The cellular Hopfield-type feedback neural network is also considered. We propose
a large network structure which connects some small identical networks (cells) together. A
parallel recursive algorithm for this type of locally connected neural networks is intro—
duced. This parallel algorithm makes all cells capable of finding their own weights simul-

taneously.

For the DANN network, we show the stability of the network with positive weights.
We show that multiple patterns can be stored as asymptotically equilibria of DANN with
only positive weights. We further formulate the problem of finding all positive weights as
the standard linear programming problem. Thus, based on given patterns, all positive

weights can be easily found.

105

106

We have designed the CMOS VLSI multiple-state neuron circuit which operates in
subthreshold. With this neuron, the network power dissipation is reduced. As a prototype,
a DANN network with three neurons has been designed. Each neuron has four states. The
PSpice simulation has shown that this prototype DANN network works well. A six-neuron

DANN Tiny-Chip has been designed and fabricated by MOSIS.

The following problems are left for future research:
(1) The persistence of equilibria for DANN with transistor-connections.
(2) The persistence of equilibria for Hopfield-type network with transistor-connections.
(3) Existence of optimal feature solutions of DANN for multiple patterns.
(4) Design a new output stage of the multiple-state neuron circuit to convert the current to
voltage.
(5) Design an on-chip learning circuit to find gate voltages for DANN with multiple-state

neurons and transistor-connections.

Identifying applications is the key to the future of the continuous-time feedback neu-
ral networks. Feedback continuous-time feedback neural networks have shown their
potential in various applications such as optimization, vision, conuol, edge detection, and

computer numerical computations, to name a few.

References

[l] Eyad H. Abed, "A simple proof of stability on the center manifold for Hopf bifurca-
tion", SIAM Review, Vol. 30, No. 3, Sep. 1988, pp. 487-491.

[2] Eyad H. Abed, "Local bifurcation control", pp. 225-241 in Dynamic Systems
Approaches to Nonlinear Problems in Systems and Circuits, ed. Fathi M.A. Salarn, Mark
L. Levi, SIAM, Philadelphia 1988.

[3] Dirk Aeyels, "Stabilization of a class of nonlinear systems by a smooth feedback con-
trol", Systems & Control Letters 5 (1985), pp. 289-294.

[4] Phillip E. Allen, Douglas R. Holberg, CMOS Analog Circuit Design, Holt, Rinehart
and Wmston, Inc., 1987.

[5] Stephen T. Barnard, "A Stochastic Approach to Stereo Vision", in Proc. National Con-

ference on Al (AAAI-86) (Philadephia, PA., Aug. 11-15), pp. 676-679.

[6] Adi Ben-Israel, Thomas N. E. Greville, Generalized Inverses: Theory And Applica-
tions, Robert E. Krieger Publishing Company, 1980.

[7] Andrew Blake, Andrew Zisserman, Visual Reconstruction, The MIT Press, 1987.
[8] R.W. Brockett, "Asymptotic stability and feedback stabilization", pp. 181-191 in Dif-

ferential Geometric Control Theory, ed. R.W. Brockett, R.S. Millman and HJ. Sussmann,
Birkhauser, Boston (1983).

107

108

[9] Shui-Nee Chow, Hack K. Hale, Methods of Bifurcation Theory, Springer-Verlag,
1982.

[10] Leon 0. Chua, L. Yang, "Cellular neural networks: Theory", IEEE Trans. on Circuits
Syst., Vol. 35. pp. 1257-1272, 1988.

[11] Leon 0. Chua, Tamas Roska, "Stability of a Class of Nonreciprocal Cellular Neural
Networ ", IEEE Trans. on Circuits Syst., Vol. 37, No.12, December 1990.

[12] Tobi Delbruck, ""Bump" Circuits for Computing Similarity and Dissimilarity of Ana-
log Voltages", CNS Memo 10, California Institute of Technology, Computation and Neural
Systems Program, May, 1991.

[13] D.K. Faddeev, V.N. Faddeeva, Computational Methods of Linear Algebra, W.H.

Freeman and Company, 1963.

[14] Jay A. Farrell, Anthony N. Michel, "A synthesis procedure for Hopfield’s continuous-

time associative memory", IEEE Trans. on Circuits Syst., Vol. 37 , July 1990.

[15] W. Eric L. Grimson, "Computational Experiments with a Feature Based Stereo Algo-
rithm", IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. PAMI-7, No. 1,
Jan. 1985, pp. 17-34.

[16] L.T. Grujic, A.N. Michel, "Exponential stability and trajectory bounds of neural net-
work under structural variations", IEEE Trans. on Circuits Syst., Vol. 38, No. 10, Oct.

1991.

109

[17] Jack K. Hale, Ordinary Differential Equations, Robert E. Krieger Publishing Com-
pany, 1980.

[18] J. Hertz, A. Krogh, R.G. Palmer, Introduction to the Theory of Neural Computation,
Addison-Wesley Publishing Company, 1991.

[19] M.W. Hirsch, S. Smale, Difi’erential Equations, Dynamical Systems and Linear
Algebra, Academic Press, 1974.

[20] 1.]. Hopfield, "Neural networks and physical systems with emergent collective com-

putational abilities,", in Proc. NatlAcad. Sci. U.S.A., Vol. 79, pp. 25542558, 1982.

[21] 1.]. Hopfield, "Neurons with graded response have collective computational proper-
ties like those of two-state neurons," in Proc. Natl. Acad. Sci. U.S.A., Vol. 81, pp. 3088-
3092, 1984.

[22] Zahid Hussian, Digital Image Processing - Practical Applications of Parallel Pro-
cessing Techniques, Ellis Horwood, 1991.

[23] Hassan K. Khalil, Nonlinear Systems, Macmillan Publishing Company, 1992.
[24] N.N. Krasovskii, J.L.‘ Brenner, Stability of Motion, Stanford University Press, 1963.
[9] Solomon Lefschetz, Difi’erential Equation: Geometric Theory (2nd edition), Inter-

science Publishers, 1963.

[25] Solomon Lefschetz, Difl’erential Equation: Geometric Theory (2nd edition), Inter-
science Publishers, 1963.

110

[26] J .H. Li, A.N. Michel, W. Porod, "Qualitative analysis and synthesis of a class of neu-
ral networks", IEEE Trans. on Circuits Syst., Vol. 35, No. 8, Aug. 1988.

[27] B0 Ling, Fathi M.A. Salam, "A cellular network formed of Hopfield networks", in
Proc. of the 35th Midwest Symposium on Circuits and Systems, Washington, DC, Aug. 9-
12, 1992.

[28] Bo Ling, Fathi M.A. Salam, "Persistence of equilibria under weight variation of feed-
back continuous-time neural networ ", Proc. of the IEEE International Symposium on

Circuits and Systems, Chicago, Illinosis, I993.

[29] B0 Ling, Fathi M.A. Salam, "Parameter determination for an implementable feed-
back neural networ ", Proc. of the IEEE International Symposium on Circuits and Sys-

tems, Chicago, Illinosis, 1993.

[30] B0 Ling, Fathi M.A. Salam, "An analytical learning algorithm for the dendro-den-
dritic artificial neural network via linear programming", Proc. of the IEEE International

Conference on Neural Networks, San Francisco, California, 1993.
[31] Bo Ling, Fathi M.A. Salam, "State feedback stabilization of nonlinear system via the
neural network approach", Proc. of the American Control Conference, San Francisco, Cal-

ifornia, 1993.

[32] Bo Ling, Fathi M.A. Salam, "Zn-state CMOS neuron circuit operating in subthresh-
old", Proc. of the 1993 World Congress on Neural Networks, Portland, Oregon, July, 1993.

[33] B0 Ling, Fathi M.A. Salam, Shanti Vedula, "Analog VLSI feature matching circuit

111

for stereo vision process", in Proc. of the 35 th Midwest Symposium on Circuits and Sys-

tems, Detroit, Michigan, August, 1993.

[34] David G. Luenberger, Introduction to Linear and Nonlinear Programming, Addison-
Wesley Publishing Company, 1965.

[35] Jerrold E. Marsden, Elementary Classical Analysis, W.H. Freeman and Company,
1974.

[36] Larry Matthies, Takeo Kanade, "Kalman Filter-based Algorithms for Estimating
Depth from Image Sequences", International Journal of Computer Vision, 3, 209-236,
1989.

[37] W.S. McCulloch, W. Pitts, Bull Biophys., 5, 1943, pp. 115-133.

[38] Carver Mead, Analog VLSI and Neural Systems, Addison-Wesley, 1989.

[39] A,N, Michel, J.A. Farrell, W. Porod, "Qualitative analysis of neural netwo ", IEEE
Trans. on Circuits Syst., Vol. 36, No. 2, Feb. 1989. ‘

[40] Kenneth S. Miller, Donald M. Leskiw, An Introduction to Kalman Filtering with
Applications, Robert E. Krieger Publishing Company, 1987.

[41] Gerald L. Morris, Patrick L. Odell, "Common Solutions for ti Matrix Equations with
Applications", Journal of the Associations for Computing Machinery, Vol. 15, No.2, April
1968, pp. 272-274.

112

[42] C. Neti, M.H. Schneider, E.D. Young, "Maximally fault tolerant neural networks",
IEEE Trans. on Neural Networks, Vol. 3, No. 1, Jan. 1992.

[43] J.M. Ortega, W.C. Rheinboldt, Iterative Solution of Nonlinear Equations in Several
Variables, Academic Press, 1970.

[44] Alex P. Pentland, "Dynamic Vision", Pattern Recognition by self-organization neu—

ral network, ed. Stephen Grossberg, MIT Press, 1991.

[45] Tomaso Poggio, Vincent Torre, Christof Koch, "Computational vision and regular-

ization theory", Nature, Vol. 317, Sep. 1985.

[46] Fathi M.A. Salam, Y. Wang, M. Choi, "On the Analysis of Dynamic Feedback Neural
Nets," IEEE Trans. on Circuits Syst., Vol. 38, No. 2, Feb. 1991.

[47] Fathi M.A. Salam, Y. Wang, "A Real-Time Experiment Using a 50-Neuron CMOS
Analog Silicon Chip with On-Chip Digital Learning", IEEE Trans. on Neural Networks,
Vol. 2, No.4, July 1991.

[48] Fathi M.A. Salam, "A model of neural circuits for programmable VLSI irnplementa-
tion of the synaptic weights for feedback neural nets", 1 989 IEEE International Sympo-
sium on Circuits and Systems, Portland, Oregon, May 1989, pp. 849-851.

[49] Fathi M.A. Salam, "New artificial neural models: basic theory and characteristics",
1 990 IEEE International Sym posium on Circuits and Systems, New Orleans, Louisiana,
May 1990. PP. 200-203.

113

[50] Fathi M.A. Salam, Yiwen Wang, "Neural circuits for programmable analog MOS
VLSI implementation", Proc. of 32nd Midwest Symposium on Circuits and Systems,
Champaign, Illinois, August, 1989.

[51] Fathi M.A. Salam, Yiwen Wang, "Learning scheme for analog CMOS neural chips",
Memorandum No. MUS/EE/A90/07, Department of Electrical Engineering, Michigan
State University, East Lansing, MI 48824, 1990.

[52] Michael Spivak, Calculus on Manifolds _ A Modern Approach to Classical Theo-
rems of Advanced Calculus, New York, W.A. Benjamin, 1965.

[53] M. Stevenson, R. Winter, B. Widrow, "Sensitivity of feedforward neural networks to

weight error", IEEE Trans. on Neural Networks, Vol. 1, No. 1, March 1990.

[54] Demetri Terzopoulos, Andrew Witkin, Michael Kass, "Syrnmetry-Seeking Models
and 3D Object Reconstruction", International Journal of Computer Vision, 1, 211-221,

1987.

[55] J. Tsinias, N. Kalouptsidis, "Output feedback stabilization", IEEE Trans. Automat.
Contr., Vol. 35, pp. 951-954, Aug. 1990.

[56] M. Vidyasagar, Nonlinear Systems Analysis, Prentice-Hall, 1978.
[57] Y. Wang and F. Salam, "Custom Analog VLSI Neural Chip with On-Chip Digital

Learning for Pattern/Character Recognition," the second international conference on
Fuzzy Logic and Neural Networks, Iizuka, Japan, July 1992.