'.:‘: .

‘11???
- gas..—

at

‘ s-f-W'
“KAT—~33
i‘ «a: 1%!»-

A

m, ‘ ‘5 deﬁlégﬁ 25-. ntﬁyﬁiﬁﬁ
3‘3 ﬁx ’ $23112 s‘m‘n‘kfé’

;. 3 ‘L
3‘3? Wm? ”29%;.“
‘y
+-"
7:

,.. z» *9
t.

, $5
W

I
J
1A
a
\

‘ “I - ‘
._... a “"0-
4 :i’fT‘i’a \-
ur.m'\-”~‘ -
. g» "'3"

n.
. .t

, X

1‘- "
{gig-mg“; , .
m- b w‘*“~“**€"v‘$:aarn *3:
ﬂag", -~: ' ‘L‘j 13:1: if

“51‘?!- d‘ a‘ V airs-«33“»
>3 u»..— ,. a? ﬁ-‘ﬁztf‘ﬂq-

” .
“ML“ "—'
J \‘ z“. ‘1‘

. 1.
u
“‘2’“:

...u~

. -.-
".nr ' (I
,.,a A. ’ .4 >
,~ wt?“ ' I ‘ I
H , O ofu"|.”"
I “Jr-nu"
.4... .ary
”Mr Hat

'7 p .
I ”Hug
, ”.1"

w

> . n r"
,vlpwa”
rﬁryrfvwla-
afP'“""”
,urtA ' "
v o ' '

r
u-v,

v
_ I

,nt-‘pn
I .vv‘!‘
, . ~..

I. ", I
‘n I ‘ ' b‘
. I..A|V""

v ' v.

’ .1. x'
v ' ,‘.4:|4:‘

—. I» way-,1: ' V” '
{:- Onomrt at" ’ "5' '
1-! u‘ ', . vac v .
..r
..

. o -
n.-
,...or

 

th*"‘

UNIVERSITY LIBRARIE

IlllmllHlllliill‘MI‘Ul illelll

3 129300 00906

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

l|\l|

This is to certify that the

dissertation entitled

CMOS VLSI IMPLEMENTATIONS OF A NEW
FEEDBACK NEURAL NETWORK ARCHITECTURE

presented by

YIWEN WANG

has been accepted towards fulﬁllment
of the requirements for

Ph.D.

degree in Electrical
Engineering

    

Major professor

Date A”?‘8 [3&2

MS U it an Afﬁrmative Action/Equal Opportunity Invitation 0-12771

 

,s W 1
LiBKA‘RY

Michigan State
University

R J

 

 

PLACE IN RETURN BOX to remove We checkout from your record.
TO AVOID FINES return on or before one due.

DATE DUE DATE DUE DATE DUE

 

‘E'f'EB 0313513

Jl__
-l__lﬂ'
H37

1
| J

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

MSU Is An Afﬁrmative ActIorVEquel Opportunity Institution
emana-ot

 

 

 

 

 

 

 

 

 

 

 

 

CMOS VLSI IMPLEMENTATIONS OF
A NEW FEEDBACK NEURAL NETWORK ARCHITECTURE

By

Yiwen Wang

A DISSERTATION

Submitted to
Michigan State University
in partial fulﬁllment of the requirements
for the degree of

DOCTOR OF PHILOSOPHY

Department of Electrical Engineering

1991

ABSTRACT

CMOS VLSI IMPLEMENTATIONS OF
A NEW FEEDBACK NEURAL NETWORK ARCHITECTURE

By

Yiwen Wang

This work develops CMOS VLSI implementations of a new architecture for feed-
back Artiﬁcial Neural Networks (ANN s). The new architecture lends itself directly to
all-MOS implementations and it has been shown to exhibit qualitatively the same
dynamic properties as gradient continuous—time feedback neural nets. Neural proper-
ties of a prototype of this new architecture had been veriﬁed via extensive SPICE

simulations and discrete-component laboratory experiments.

A 6-neuron Tiny-chip of the new architecture was designed and fabricated as a
prototype all-MOS chip of the new architecture for developing and assessing various
potential off-chip learning algorithms. A new dynamic learning algorithm is described
for general dynamic continuous-time models of ANN s. The learning algorithm is spe-
cialized to the new architecture of ANN s and is successfully tested on the 6-neuron
Tiny-chip.

Subsequently a digital MOS realization of the learning algorithm is developed. A
SO-neuron CMOS analog chip with on-ehip digital learning scheme was designed and
fabricated in 6.8mm x 4.6mm chip size with 63,025 transistors and 1225 programme
able synaptic (interconnect) weights via standard 2pm CMOS n-well technology.

Dedicated interface circuitries and software environments had been built to suc-
cessfully demonstrate the use of the prototype chips of the new neural circuit. As an
example of an application of the fabricated neural net chip, real-time experiments are
described in which the chips are used as a coprocessor to a microcomputer. These
experiments entail learning an arbitrary image which can be subsequently retrieved by

images distorted by binary-noise in the order of 20 usecs in real time.

To our knowledge, these are the ﬁrst successful and effective (analog) neural chip
experiments with guaranteed learning capability. The literature on the implementation
of ANN s has dramatically grown, yet, to the best of our knowledge, none of the pro-
posed neural implementations has been shown to execute and substantiate the claimed
learning and retrieval capabilities in real time. The implementations and real-time
experimental results described here have been shown to perform and substantiate the
capabilities attributed to ANN s by pursuing an approach of analysis followed by direct

electronic implementations.

To
my mother; my wife, Meei-Jyh; and my daughter, Ming.

iii

ACKNOWLEDGMENTS

I would like to express my sincere gratitude to my major advisor, Dr. F athi M.
A. Salam, for his inspiration, guidance and support throughout the years of my gradu-
ate studies. Thanks to Dr. Lionel M. Ni, Dr. Timothy Grotjohn, and Dr. Paul Parker

for their valuable comments and suggestions.

I wish to express my sincere thanks and appreciation to my mother and my wife’s

parents for their continued prayers and encouragement.

Form the bottom of my heart, I would like to thank my wife, Mcei-Jyh, for her
love, understanding, and support, my brother, Chen-Wan, and my sister, Yung—Shih,

for their help and encouragement, and my lovely daughter, Ming.

iv

TABLE OF CONTENTS

Chapter 1. INTRODUCTION . ................................................................................... 1
Chapter 2. BACKGROUND ...................................................................................... 8
2.1. Biological Overview of Neural Networks ............................................ 8

2.2. Basic Models of Artiﬁcial Neural Networks ........................................ 11

2.2.1. The Model of a Neuron ............................................................... 12

2.2.2. Models of the Synapse ................................................................ 13

2.2.3 Feedback Models ......................................................................... 13

2.2.4. Feedforward Models .................................................................... 16

2.2.5. General Models ............................................................................ 18

2.3. Overview of the Present Literature ....................................................... 19

2.4. CMOS VLSI Circuits ........................................................................... 21

2.4.1. MOS Transistors .......................................................................... 23

2.4.2. Design Considerations ................................................................. 23

Chapter 3. A NEW NEURAL NETWORK ARCHITECTURE .............................. 26
3.1. Motivation from Biology ...................................................................... 26

3.2. One Neuron with Self-feedback ............................................................ 27

3.3. General N-neuron Architecture ............................................................. 28

3.4. Mathematical Analysis .......................................................................... 29

3.5. Circuit Implementation of the New Architecture ............................... 31

3.5.1. A Neuron ...................................................................................... 31

3.5.2. A Programmable Synapse ........................................................... 34

3.5.3. Storage of Synaptic Weights ....................................................... 34
3.6. SPICE Simulations and Laboratory Experiments

of the New Architecture ....................................................................... 35

Chapter 4. LEARNING OR THE DYNAMIC UP—DATE OF "WEIGHTS" ........ 39
Chapter 5. A PROTOTYPE 6-NEURON CMOS TINY-CHIP ............................... 43
5.1. Layout design ........................................................................................ 43

5.2. Interfacing with a Personal Computer .................................................. 52

5.3. A Learning Scheme ............................................................................... 53

5.3.1. Dynamic Learning Model ............................................................ 54

5.3.2. Software Implementation of the Dynamic Learning .................. 55

5.4. Chip Experimental Results with Learning ............................................ 57

Chapter 6. A 50-NEURON CHIP WITH DIGITAL HARDWARE

LEARNING SCHEME ............................................................................ 61
6.1. A Digital Learning Scheme .................................................................. 61
6.2. Hardware Overview of the 50-neuron Chip ......................................... 61

6.2.1. A Neuron ...................................................................................... 63
6.2.2. A Programmable Synapse ........................................................... 63
6.3. Chip Experimental Results .................................................................... 70

Chapter 7. A REAL-TIME APPLICATION USING THE 50-NEURON

CHIP AS A PATTERN/CHARACTER ASSOCIATOR ....................... 71

7.1. Interfacing and Software Environment ................................................. 70

7.2. Storing a Single Pattern ........................................................................ 74

7.3. Storing Multiple Patterns ...................................................................... 77
Chapter 8. SUMMARY AND CONCLUSIONS ...................................................... 79
8.1. Summary ................................................................................................ 79

8.2. Conclusions ............................................................................................ 81
BIBLIOGRAPHY ................ 82

LIST OF TABLES

Table 2.4.2.1 MOSIS package types ......................................................................... 24
Table 2.4.2.2 MOSIS Standard Frames ..................................................................... 25
Table 3.6.1 SPICE simulation results of a 3-uint prototype circuit ........................ 37
Table 3.6.2 Experimental results of a 3-uinr prototype circuit ................................ 38
Table 5.1.1 Experimental results of a 3-unir subcircuit of the 6-neuron chip ........ 52
Table 6.2.2.1 Control signals for the operating functions ........................................ 67

vii

LIST OF FIGURES

Figure 2.1.1 A highly idealized biological neuron ................................................... 9
Figure 2.2.1.1 Neural model circuits ......................................................................... 12
Figure 2.2.3.1 A typical feedback artiﬁcial neural network model ......................... 14

Figure 2.2.4.1 A typical two-layer feedforward artiﬁcial neural network model 16

Figure 2.4.1 MOS transistor physical structures ....................................................... 22
Figure 3.2.1 One neuron with self-feedback ............................................................. 27
Figure 3.3.1 A 3-um'r prototype circuit of the new architecture .............................. 28

Figure 3.5.1.1 An implementation of one neuron via two CMOS

inverters in series ............................................................................... 32
Figure 3.5.1.2 An implementation of one neuron via an ampliﬁer ......................... 32
Figure 3.5.2.1 Transfer characteristics of an nMOSFET ......................................... 33

Figure 5.1.1 The schematic diagram of a fully connected 6-unit

neural circuit. ....................................................................................... 44
Figure 5.1.2 The layer names and their corresponding symbols ............................. 45
Figure 5.1.3 A VLSI layout of one neuron with self-feedback ............................... 46

Figure 5.1.4 A VLSI layout of the interconnect network for the 6—unir

neural circuit ........................................................................................ 47
Figure 5.1.5 A VLSI layout of the 6-unit neural circuit .......................................... 48
Figure 5.1.6 A die photo of the 6—um't neural circuit ............................................... 49
Figure 5.1.7 A die photo of the 6-unit neural circuit on MOSIS Tiny-chip ........... 49
Figure 5.1.8 A pin conﬁguration of the 6-unit chip ................................................. 50

Figure 5.4.1.a A graphic display after the network had learned the ﬁrst

viii

desired pattern ................................................................................... 59
Figure 5.4.1.b A graphic display after the network had learned the second

desired pattern ................................................................................... 60
Figure 5.4.1.b A graphic display after the network had learned the third

desired pattern ................................................................................... 60
Figure 6.2.1 A die photo of the 50-neuron chip ..................................................... 62
Figure 6.2.1.1 A schematic diagram of a single neuron with self-feedback

and I/O selector ................................................................................. 64
Figure 6.2.1.2 A VLSI layout of a single neuron circuit ......................................... 64
Figure 6.2.1.3 SPICE simulation results of a single neuron circuit ......................... 65
Figure 6.2.1.4 A die photo of a single neuron circuit .............................................. 66
Figure 6.2.2.1 A VLSI layout of a single synaptic weight circuit ........................... 66

Figure 6.2.2.2 SPICE simulation results of a single synaptic weight

circuit for the learning ...................................................................... 68
Figure 6.2.2.3 SPICE simulation results of a single synaptic weight circuit

for the direct assignment .................................................................. 69
Figure 6.2.2.6 A die photo of a single synaptic weight circuit ............................... 70
Figure 7.1.1.a A block diagram of the interface circuitry for the 50-neuron chip . 72
Figure 7.1.1.b A block diagram of the interface circuitry for the SO-neuron chip . 73
Figure 7.2.1 Experimental results of the SO-neuron chip; all initial

conditions with Hamming distances 9 from the character F ........... 76
Figure 7.2.2 Experimental results of the 50meuron chip; all initial

conditions with Hamming distances 10 from the character F ......... 76
Figure 7.3.1.a Experimental results of the 50-neuron chip; all initial

conditions with Hamming distances 3 from the character F ........... 78
Figure 7.3.1.b Experimental results of the 50—neuron chip; all initial

conditions with Hamming distances 3 from the character T ........... 78

ix

CHAPTER 1

INTRODUCTION

Artiﬁcial Neural Networks (ANN s) have recently become very popular due to
their anticipated ability to solve problems such as vision, speech understanding, pattern
recognition with distorted data, information retrieval with partial information, etc..
These networks attempt to mimic the functionality of the nervous system and the
brain. The human nervous system and the brain have the capabilities of creativity,
adaptive learning, error correction, and robustness. It is believed, for instance, that the
massive parallelism and high interconnectivity in the biological neural networks
account for some of their computational capabilities and intelligence. An ANN typi-

cally possesses parallel-like structure manifested as high connectivity.

Although it is perhaps proper that one includes a tutorial introduction to the
field, this has not been possible for several reasons: one being the unbounded diversity
of the ﬁeld, and another is the publication of various recent textbooks covering

numerous orientations and foci [26.48.81].

During the 1960’s, artiﬁcial neurons were extensively studied. In many cases
these were simulations of biological neurons which attempted to take into account all
the many and varied nuances of real neurons. In other cases the neurons were
oversimpliﬁed but the accompanying algorithms were not suitably developed
Recently, very simple neurons have been considered resulting in emphases have shifted

away from the neuron itself toward the interconnections of neurons and the weighting

of their signals. Many researchers have proposed electrical circuits that model the
highly interconnected nervous system and the brain. These circuits are often referred
to as artiﬁcial neural nets. Architectures for artiﬁcial neural nets have been reported in
numerous works, e.g. [1-7] and [8-12]. In particular, [13-26] are concerned with some

aspects of implementations.

However, there exist two major problems that must be attacked before the reali-
zations of such artiﬁcial neural networks can be achieved. The ﬁrst problem is how to
implement those global and dense interconnections among many neuron-like elements,
and the second is how to program such highly dense interconnect weights. One of the
goals of the ﬁeld is to produce hardware, containing millions of neurons, that mimics
the signal processing capabilities of the brain where millions of neurons are present.
The majority of approaches for implementation, however, fall into two classes: (i)
those that are software-based and (ii) those that are hardware-based but with relevance

toward hardware realization and design for real-time applications.

Software-based implementations usually employ an algorithm, which“ is based
on the architecture and understanding of models of neural networks, mapped onto a
conventional digital computer. Software-based implementations are ﬂexible and easily
modiﬁed. However, when large ANN s are represented by a set of differential equa-
tions, simulation on a digital computer may take a considerable amount of computing
time. Consequently, it becomes apparent that the ﬁeld will be driven by the develop-
ment of devices that are simple and can readily be realized in hardware VLSI, optics,

or electro-optics.

Neurons in the human brain are interconnected in 3-D space because it is the
most natural and efﬁcient way of interconnection, but VLSI-based interconnections are
inherently 2-D in nature. The high interconnectivity of these networks makes an elec-
tronic implementation difﬁcult. Several schemes of VLSI implementations of neural

networks have been proposed [13-26]. Optical signals can ﬂow through 3-D space to

achieve the required interconnections between neuron-like logic elements. Psaltis and
Farhat reported an optical implementation of the Hopﬁeld neural network using an opt-
ical vector-matrix multiplier as a programmable interconnector and illustrated the
potential feasibility of optical content addressable associative memory [27]. Salam and
Wang developed a formulation for 2-D array neural net processing which is suitably
mapped into various electro—optical implementations [28,29]. However, due to the
immaturity of the present optical technologies, it is difﬁcult to presently make optical
devices that are compact and easily programmable (to implement large scale artiﬁcial

neural networks).

Other goals of the ﬁeld are to develop learning schemes [10,30-37] which
effectively choose or "program" the interconnections so as to render a relative
conﬁguration of the neural network that conesponds to a pro-speciﬁed set of stored
data. The Backward Error Propagation (BEP) [34] has been somewhat successful as a
supervised learning rule for multilayered feedforward networks with hidden units. The
multilayered feedforward network is a continuous valued mapping from the input to
the output space. Its mapping function is adapted by repeatedly presenting the network
with a set of input-output vector pairs and using an appropriate update law (typically
gradient descent) to modify the mapping until a functional relationship is realized that
approximately satisﬁes the set of input-output pairs. However, as originally intro-
duced, BEP is limited to feedforward networks which gave marginal, albeit, and satis-
factory success. Networks with feedback necessarily raise the problem of overall sys-
tem stability [38]. There are several learning rules such as Hebb’s rule [36] and the
Pseudo-inverse rule [37] for feedback models, but none of the traditional rules has

been shown to be effective in applications.

The overall objective of this research is threefold: Firstly , to establish simula-
tion and experimental foundations for the operation of a certain, recently proposed,

neural network architecture. Secondly , to design and implement the neural net

architecture via CMOS digital/analog VLSI/LSI circuitry. Thirdly , to demonstrate the
performance of the design using the fabricated VLSI chips. The focus for applications
will be on the proper design and implementation of this architecture as a classiﬁcation

device.

CMOS VLSI implementations of a new architecture for feedback AN N s are
described in this thesis. The new architecture has qualitatively the same dynamic pro-
perties as gradient continuous-time feedback neural nets [39,40]. Moreover, the archi-

tecture also has the following features:

i) It reduces the maximum number of connections to n (n + 1)/2, where n is the

number of neuron processors in the network.

ii) It does not require the symmetry of interconnections (in order to ensure the

convergence of all solutions to equilibria only).

iii) More importantly, it does not require the realization of linear resistive elements
for the synaptic weights. Instead, the connections are realized via MOSFET

conductance elements.

iv) It lends itself naturally to direct analog MOS VLSI silicon implementation.

A 3-unit prototype circuit is employed to illustrate and test the characteristics
and the performance of the new architecture. Extensive SPICE simulations are con-
ducted to obtain the transient evolution to the steady states (or stable equilibria).
Discrete—component laboratory experiments are also performed. The results are quali—

tatively identical to the SPICE simulations.

A 6-neuron Tiny-chip of the all-MOS implementation of the new architecture
was designed and fabricated as a prototype chip of the new architecture. Complete
testing of the Tiny-chips has successfully revealed the proper operation of the neural

circuit.

A new dynamic learning algorithm is described for general dynamic
continuous-time models of ANN s [30] which was motivated from our efforts to
develop a learning algorithm for the new architecture. The learning capabilities of the
new algorithm were tested on the 6-neuron Tiny-chip for storing and retrieving arbi-
trary digitized images.

The new learning algorithm has subsequently been specialized to a digital learn-
ing scheme which is realizable in the all-MOS VLSI implementations of ANN s. A
50—neuron CMOS VLSI chip of the new architecture with on-chip digital hardware
learning scheme was successfully designed and fabricated on 6800M x 4600um chip
size using a MOSIS 2-um scalable CMOS technology and a 64-pin standard pad
frame. There are 63,025 transistors, 1225 programmable synaptic weights, and 50 neu-
rons on a single chip. Each neuron consists of two CMOS inverters in series with one
feedback nMOS transistor between the input node and the output node. The gate vol-
tage of the feedback nMOS transistors can be adjusted globally. Either the input node
or the output node of the neuron can be connected to an external pin via a CMOS ana-
log switch. Each synaptic weight can be set up via on-chip digital hardware learning
circuitry or via direct assignment and it can be stored on an on-chip digital flip-ﬂop.

Dedicated interface circuitries and software environments were designed and
developed for a personal computer in order to facilitate testing and verifying the per-
formance of the fabricated chips. Real-time experiments are conducted to successfully
demonstrate the use of the 50-neuron chip with on-chip digital learning as a real-time

pattern/character recognizer and associator.

The thesis is organized as follows. Chapter 2 covers some background material
on neural networks which is pertinent to the content and emphasis of this thesis. It
includes a biological overview of neural networks, some simpliﬁed neural net models,
and an overview of the current literature on the circuit implementation of ANN 3.

Because standard CMOS technology is employed to implement ANN s in this thesis,

Chapter 2 also brieﬂy introduces the basic concepts of MOS transistors and the charac-
teristics and limitations of VLSI/L81 design.

Chapter 3 starts out with an introduction of a new architecture of feedback
ANN s which may be motivated from neuro-biology. The basic building block and the
general n-neuron architecture with various topologies is also described. Then the basic
theory and some characteristics of gradient models are presented as a theoretical foun-
dation for the operation of the new architecture. Circuit implementations of the new
architecture are also discussed in Chapter 3. It ends with extensive SPICE simulations
and discrete-component experiments to verify the neural properties of a prototype of
the new architecture. In Chapter 4, a new learning algorithm for dynamic continuous-

time models of ANNs is described.

Chapter 5 describes a VLSI layout designed for a 6—unit neural circuit on a
MOSIS Tiny-chip as a prototype chip of the new architecture. A dedicated interface
circuitry is built to facilitate the programming of the chip. The new learning algorithm
described in Chapter 4 is specialized for the 6-unit Tiny-chips. The learning algorithm
is implemented in software on a PC; it interacts with the chip in real-time via the dedi-
cated interface circuitry. Using the interface circuitry and the learning schemes, we
successfully demonstrate the use of the Tiny-chips of the new neural circuits as a

recognition and association device.

Chapter 6 describes the design of a 50-neuron CMOS analog chip with on-chip
digital hardware learning scheme. SPICE simulations are conducted to verify the
functionality of the neuron and synaptic weight circuits before the chip was fabricated.
Extensive testing of the 50-neuron chip successfully substantiates the predicted opera-

tion of the neural circuit.

In Chapter 7, a real-time application is conducted using the 50-neuron chip with
on-chip digital learning as a pattern/character associator. An interface circuitry and a

software environment is designed and developed which provides a user ﬁiendly

environment to easily control and operate the fabricated chips. Real-time experiments
are conducted to demonstrate the learning capability of the SO-neuron chip for storing
a single pattern/character or storing multiple pattems/characters via digital hardware

learning circuitry. Finally, summary and conclusions are collected in Chapter 8.

CHAPTER 2

BACKGROUND

Interest in neural modeling and neural computing is not recent. It can be
traced to the work of McCullough and Pitts [41] and Hebb [36] in the 1940’s. By the
early 1960’s, active efforts in neural networks and learning were concentrated within
relatively few research groups in this country and abroad. The two most active groups
here were those of Prof. Frank Rosenblatt [42] at Cornell and Prof. Bernard Widrow
[43] at Standford. More recent work by Kohonen [10,37], Grossberg [44—46], Hopﬁeld
[1-7], Rumelhart and McCleland [34,47], Arima [82,83], and others has led to new
resurgence of the ﬁeld. This interest is due to the development of new network topo-
logies and algorithms, new analog VLSI implementation techniques, and some intrigu-
ing demonstrations as well as by a growing fascination with the function of the human

brain.

2.1. Biological Overview of Neural Networks

The brain can perform many tasks that conventional digital computers still
cannot. For this reason, it behooves us to examine various models of how a large

number of simple devices can work together to perform useful computation.

A neuron is the fundamental unit of all nervous systems [48]. Figure 2.1.1
depicts a highly idealized description of a real neuron [35].

 
 
 
 
 
    

post-synaptic dendrite
\

 

dendrites

pre-synaptic axon

 

 

Cell body ................

Figure 2.1.1 A highly idealized biological neuron.

Usually, all signals from other neurons come into the dendrite. A cell body
integrates or sums all the information that comes in. Action potentials are initiated at
the axon hillock when the cell ﬁres. The ﬁring rate depends on the aggregated signals
received. The action potentials are a sequence of pulses whose frequency depends on
the intensity and duration of signals that excited the cell [10]. Pulses are transmitted to
the dendrites of other neurons through synapses. A synapse consists of the presynaptic
cell and the postsynaptic cell, which are separated by a synaptic gap. Numerous
chemical molecules are involved in transmitting signals through the synapse. If the
action potentials arriving at the synapse exceed a certain threshold, the chemical
molecules in the presynaptic cell are released and transmitted via the synaptic gap to

the postsynaptic cell of another neuron. As the neurotransmitter molecules combine

10

with receptor cells on the postsynaptic cell membrane, the chemical signals transform

into electrical pulses.

Axons are specialized for the conduction of an electrical impulse, called an
action potential. Action potentials originate at the axon hillock, the junction of the
axon and the cell body, and travel to the small branches of the axon terminals [10].
From there chemical signals are passed on to other cells. Some axons are wrapped
with a sheath of myelin. There exist nodes between two pieces of myelins. These
nodes are achieved to reinforce the weakened signals so that they can be transmitted

without diminishing. Thus these nodes are sometimes referred to as signal repeaters.

Dendrites are thinner ﬁbrous projections extending outward from the cell
body. Dendrites contain regions that receive signals from the axons of other neurons,

convert these signals into electrical impulses, and transmit them to the cell body [49].

A cell body receives signals independently as well. Electrical signals that are
generated in the dendrites or cell body spread passively to the axon hillock. If the sig-
nal is great enough, an action potential - an electrical pulse - is generated and is

actively conducted down the axon.

Synapses are specialized sites where neurons communicate with the cells. The
axon terminal of the presynaptic cell contains vesicles ﬁlled with a particular neuro—
transmitter molecules. When the nerve impulse reaches the axon terminal, these vesi-
cles are exocytosed, releasing their contents into the synaptic gap, the narrow space
between the cells [50,84]. The transmitter diffuses across the synaptic gap and com-
bines to receptors on the dendrite terminal of the postsynaptic cell. Upon combining,
it induces a change in the ionic permeability of the postsynaptic membrane that results
in a signal of the electrical potential at the point. With an excitatory synapse, the sig-
nal from the presynaptic cell will be more positive. With an inhibitory synapse, a
nerve impulse in a presynaptic neuron will be more negative to prevent the generation

of an action potential.

11

Usually, the synapses are connected from the axon terminal of the presynaptic
cell to the dendrite terminal of the postsynaptic cell. However, synapses may also
appear between presynaptic dendrites and postsynaptic dendrites, or even between
presynaptic axons and postsynaptic axons [48]. For examples, the Horizontal and .the
Ganglion cells in the retina [26], their synaptic connections among neurons only occur
via their input terminals only. The synaptic connections among neuron units which is
established via dendrites only may be refen'ed to as dendro—dendritic connections.

In a typical state of the art human brain, there are in the order of 1011 to

1012 neurons [51].

2.2. Basic Models of Neural Networks

Models of the biological neurons and their networks, if accurate, ought to cap-
ture at least some of the properties of biological networks. At least a model should
possess an essence of the mechanisms of operations of biological neural nets. Models
for neural nets can be classiﬁed into the following two categories according to their
applications. The ﬁrst kind of models is for the purpose of studying the functions of
the brain. This category may be referred to as reverse engineering. These models are
of primary interest to biologist, psychologists, physiologists, etc.. The other kind of
models is for the purpose of advancing engineering technology. These are often
referred to as Neuro-engineering or Artiﬁcial Neural Networks (ANN s). These models
are inspired by the neurobiology and aim at improving the technological processing of
data or information. We will primarily focus on the second category.

12

2.2.1. The Model of a Neuron

In a single neuron, a train of action potentials is propagating pulses of
electro-chemical activity. If the new has a strong input, these pulses are generated at
a high rate. If the input is weak or absent, they are generated at very low rate. The
mean rate at which these action potentials are generated results in a smooth nonlinear
function of the mean membrane potential, say S,- (u,-) which is typically modeled as a
monotone nondecreasing sigmoid function [1-5] or approximation thereof. Since the
synapses are activated by arriving action potentials, v,- = S,(u,-) becomes an input and
output relationship for a neuron, where u,- could be thought of as the mean soma
potential of a neuron from the total effect of its excitatory and inhibitory inputs, and 'v,
could be viewed as the short-term average of the ﬁring rate of the cell i. We use
model neurons which lend themselves naturally to VLSI implementation that com-
municate by means of voltage levels. Think of a high voltage as representing a high
level of activity, a low voltage as representing a low level of activity. The neurons (or
the processing elements) are modeled as ampliﬁers having a monotone nondecreasing

sigmoid input-output relation, as shown in Figure 2.2.1. 1.

VjTij

 

 

 

 

Figure 2.2.1.1 Neural model circuits.

13

2.2.2. Models of the Synapse

The strength of the synapse from a presynaptic neuron j to a postsynaptic
neuron i can be modelled as a linear parameter W5,- so that the postsynaptic signal is
given by Wij VJ- or ngSJ-(uj ). If the synapse is excitatory, the output from neuron j
will drive neuron i to produce more output, i.e. Wij is positive. If it is inhibitory, less
output will be produced and Wij will be negative. The linear synaptic weights can be
realized in VLSI implementation of an ANN via linear resistors [18, 21] or analog
multipliers [17,20,52-57].

The strength of the synapse in the biological neuron is a highly nonlinear
function in order to describe the chemical substances transmitting signals through the
synaptic gap. The dynamic neural net models supporting with good theoretical
analysis, the synapse may also be modeled as a nonlinear function [39,40] so that the
postsynaptic signal is given by f U (S ,- (uj )) where f if is a nonlinear function describing
the strength of the synapse.

2.2.3. Feedback Models

Figure 2.2.3.1 depicts a usual feedback neural net model, where the outputs of
a set of neurons return to become inputs with an interconnection weight matrix T,,-.

This kind of neural net model has rich dynamics, but is hard to program.

In a biological system, it,- will lag behind the instantaneous outputs V] of the
other cells because of the input capacitance C,- of the cell membrane, the transmem-
brane resistance R,- , and the ﬁnite conductance Tij . Thus the rate of change of u,- in
feedback ANN s is determined by the following resistance-capacitance charging equa-

tion.

“i

du-
c, 7“— = 2 Til-VJ- - ? + 1,. (2.2.3.1)
I

l

14
u,- ='- Si—1[Vi]’
where Rfl = pfl + 2 T,-.
i

which is the net input impedance, C, is the total input capacitance of the neuron i, I,
is an external stimulation or excitation at the input of the neuron i, and Ti} Vj- is the

postsynaptic current from neuron j to neuron i.

Outputs

 

Inputs

 

 

 

 

 

 

 

 

D : Neuron I : Synapse‘l'tj
Figure 2.2.3.1 A typical feedback artiﬁcial neural network model.

We can now construct an electrical network model [1-7]. A cell body is
implemented via an (operational) ampliﬁer whose input-output relation is a sigmoid
function. Axons and dendrites are each replaced by transmission wires. Synapses can

be substituted by conductance devices.

The dynamic behavior of the feedback neural net model can be examined by
considering the energy function of equation (2.2.3.1) [1—7,44,58]. There is symmetry
requirement on the connections, Ti- = Tji for the existence of an energy function.
The so-called potential energy function, which is a ﬁrst integral of (2.2.3.1), can be

derived as

15
1 - V.- -
E = -32. z Tuvivj + ZR,- 1 I) S, 1(V) dV - 21m. (2.2.3.2)
r j r r

Hence the dynamic equations of the neural network model can be rewritten as

 

 

4“: BE
~-— = — —. 2.2. .
Ct 3V, ( 3 3)
The time derivative of the energy function along trajectories is
dE BE dui
— = —— 2.2.3.4
__ 2 815‘ W i dur
' , av, du, dt
= _ 2 LE ﬂ. 2
I C, u,- 3V,-
_ _2 _1_dS.(u.) at: 2
1 Ci du, 3V,-
.<_ 0,
. . . . dSi(ui)
smce ego and S,- (u,-) rs a monotone nondecreasmg function, thus 720. There-
t'

fore, this system is a gradient-like system. That is, this energy function decreases

along trajectories and its time-derivative equals zero at 385' = 0, which is an equili-
t'

brium point of this system.

Some computational problems can be transformed into a "more—or-less"
equivalent optimization problem by a so—called regularization procedure [59,60]. The
energy function of feedback neural net model provides the link between this optimiza-
tion problem and its solution in terms of neural network, since in the network the
potential is automatically minimized (the characteristics of the gradient system) [1-7].

The computational dynamics of typical neural networks are characterized by the

16

existence of several stable states. These steady states correspond to memories or pos-

sible answers of problems.

’0

> ’ozo'w’oy >
> > >
> > >
> b >
Input Layer Hidden Layer Output Layer

D : Neuron O : Synapse
Figure 2.2.4.1 A typical two-layer feedforward artiﬁcial neural network model.

2.2.4. Feedforward Models

Figure 2.2.4.1 depicts a feedforward neural net model, where outputs of any
layer are weighted and summed as an input to a neuron in the next layer. An external
input is applied to the ﬁrst layer, which is called the input layer and fed forward to the
last layer, which is called the output layer. Any layer between the input layer and the
output layer is called a hidden layer. The governing static equation for each neuron

unit in any layer may be represented as

Y; = 5i( 2 Wijxj + 9: ). (2.2.4.1)
J'

where y,- is the output of the i -th neuron, S,- (.) is a nonlinear monotone nondecreasing

sigmoid function, wij is the connection weight from the j -th neuron output of the pre-

vious layer to the i -the neuron input, xj is the j-th output of i a neuron unit in the

17

previous layer, and 0,- is the threshold bias at the input of the i-th neuron.

The feedforward neural nets are quite powerful from the standpoint of being
able to program such a network through a learning scheme such as BEP [34] to do a
useful task. BEP is a continuous valued mapping ﬁ'om input to output, where the
mapping function is adapted by repeatedly presenting the network with a set of input-
output vector pairs and using an appmpriate feedback function (typically gradient des-
cent) to update the mapping until a functional relationship is realized between the

inputs and outputs.

The total squared error function is given by

E = 2 Ep (2.2.4.2)
P
_ i
— 2 )3 2t, —:y,..-> .
p I
where Ep is the squared error function for the desired target p , say t; = [tpl - - - 'pn]

and ypi is the actual output for the target p .

The BEP rule is governed by the following equations:

 

w}; = tog-1 + 2; prg. (2.2.4.3)
P
35,
AP w”. = -n am,- (2.2.4.4)
115”ij ’

where Ap w-- is the change of the weight wij at the k —th iteration for the desired target
p , n is the learning rate which is sufﬁciently small and positive, and ypj is the output

of the previous layer. If a unit j is in the output layer, SP,- is given by

at
SPI = d—uiﬂpi — ypil' (2m245)

18
If a unit j is in the hidden layer, SP,- is given by

at
Opi = Egﬁplwu, (2.2.4.6)

where l is the index for the units in the next layer to which a unit i is connected.

A modiﬁed BEP learning rule was proposed [33] in order to realize on-chip
learning circuits using standard CMOS technology. Using this modiﬁed BEP learning
rule, sigmoid-derivative circuits are not necessarily required for the implementation of
feedforward ANNs with learning.

Both feedback and feedforward models are interesting from the point of view
of artiﬁcial neural computation. Technologically, they emphasize the same things, a
set of sigmoid I/O neurons and a far larger set of interconnects that describe the rela-

tionships between the output of some neurons and input of others.

2.2.5. General Models

Besides the feedback and feedforward models, there are other ANN models
such as the combination of the feedback and feedforward models [38], silicon retina
[26,61] and cochlea model [26], self—organizing adaptation model [10]. The focus of
the silicon retina is trying to mimic the biological functions of primate retina. Another
mathematical model of ANN s is introduced by Hoppensteadt [62] using voltage con-
trolled oscillators (V COs).

19

2.3. Overview of the Present Literature

The interest in feedback neural networks has been revived by the recent publi-
cation [5] by Hopﬁeld and Tank. In this publication, the authors were able to demon-
strate the applicability of neural networks ideas to the design of A/D converters, signal
decision circuits, and linear programming circuits.

Hopﬁeld neural network is used for VLSI implementation due to the simple
architecture and well-deﬁned network behaviors. A major component of VLSI imple-
mentation of artiﬁcial neural networks is the array of connections between neurons, the
synapses. The transfer function of the neuron itself is a monotonic sigmoid function
which can be easily implemented in VLSI by an ampliﬁer or two logical inverters in

series.

Neural network hardware based on the silicon VLSI technology is actively
being pursued by several research groups. Hubbard et al. [21] demonstrated a thin
ﬁlm synaptic array in submicron feature size fabricated by e-beam lithography. Selec-
tively deposited amorphous silicon resistive elements at the nodes provide the resistive
array synaptic connections which will be useful as an associative read-only memory.
Sivilotti er al. [22] at California Institute of Technology had fabricated a program-
able neural network chip with twenty-two neurons and +1, 0, and -l synapses. This

circuit is able to perform an on-chip learning by employing the truncated Hebb’s rule.

Since digital circuits have proven themselves in VLSI, there is one chip [25]
developed by T. K. Miller III et al. , which was implemented with fully digital cir-
cuits operating in a stochastic manner.

Researchers at AT&T Bell Laboratories [23] fabricated a 54-neuron CMOS
chip with programmable +1, 0, and -l synapses. Such an implementation uses static
RAM cells as prespeciﬁed memories. The stored data are presented in the interconnect
matrix and only one output is evaluated by the largest inner product value between the

input vector and the stored vectors. Recently, Bell labs [16,85] fabricated a 32768

20

programmable binary connections neural-net chip which can be integrated with a digi-
tal signal processor and fast memory for image processing. They also fabricated a
programmable neural network chip with over 130,000 connections and a 4.5x7rnrn2
chip size using a 0.9urn CMOS technology [86]. A character recognition application
was performed using the chip in conjunction with a digital signal processor.

Sage er al. [24] at MIT Lincoln Laboratory developed chips based on Metal-
Nitride-Oxide-Semiconductor (MN OS) and Charge-Coupled Devices (CCDs) technol-
ogy. Such an implementation would achieve analog synaptic weights via variable
charge storage. Agranat er al. [15,87,88] at Caltech fabricated a neural network
integrated circuit with 65536 analog programmable synapses and 256 fully intercon-

nected neurons using CCD techniques.

Researchers at Jet Propulsion Laboratory [17] fabricated a 32x32 synapse chip
using analog multiplier circuits and a 32-channel variable-gain neuron chip. Research-
ers at Naval Research Laboratory [55,90] fabricated both 32x32 and 128x64 pro-
grammable analog vector-matrix multiplier chips to implement multilevel artiﬁcial
neural networks. A VLSI chip from Intel Corporation [20] has 64 neurons and 64x64
programmable synaptic weights which was implemented with fully analog circuitry

using analog multiplier and ﬂoating gate device techniques.

Researchers at Lockheed Research Laboratory [18] built an analog neural net-
work breadboard consisting of 256 neurons and 2048 programmable synaptic weights
with S-bit resolutions. Muller et al. [91,92] at the University of Pennsylvania
developed a general purpose analog neural computer which is composed of intercon-

nected modules containing arrays of neurons, modiﬁable synapses and switches.

In a different approach, Mead et al. [26,61] at Caltech integrated sensor
arrays and processing elements to emulate some of the spatial and temporal properties
of neural networks in the eye. Recently, they developed a working analog VLSI chip
[19] that implements a model of early auditory processing in the brain.

21

As of the present time, there are no reported application results for most of
these chips, except the reported AT&T results [85,86]. More importantly, there is no

on-chip hardware learning scheme for any of these chips.

There are only a few silicon ANN chips with on-chip learning. There is a
single neuron chip developed by Ricoh Corporation research group [89] with on-chip
learning and a 8.39x8.03mm2 chip size which was also implemented with fully digital
circuits using a 1.5m CMOS technology and operating in a stochastic manner.
Schneider et al. [93] at University of Manitoba fabricated a 25-neuron chip with 600
on-chip in situ learning Hebbian synapses and a 5.5x4.6mrn2 chip size using a 3m
CMOS technology. However, there are no reported testing results for the functionality

of these two chips and there are no reported application results.

Arima et al. [82,83] at Mitsubishi Electric Corporation, developed two on-
chip self-learning neural network chips. One has 125 neuron units and 10K synapse
units with a l3><13mrn2 chip size using a lwn CMOS technology and the other has
336 neurons and 28K synapses with a 14.5)(14.5mrn2 chip size. There are only two
examples of association in the reported application results and both two examples can
not successfully retrieve the complete desired patterns respectively after the network
learned 10 desired patterns. We have reported our testing experiments of the 50-
neuron chip with on-chip learning in [13], in June 1991. The 50-neuron chip can store
a desired pattern and successfully retrieve it via all initial conditions with Hamming

distances less than 9 from the desired pattern.

2.4. CMOS VLSI Circuits

It has been emphasized that a large number of neurons are needed in an ANN
to produce emergent useful computations in the neural network sense. Since then,

many workers have sought to implement the feedback (and the feedforward) AN Ns

22

using VLSI/LSI silicon electronics.

Over the past few years, Complementary Metal Oxide Silicon (CMOS) tech-
nology has played an increasingly important role in the world integrated circuit indus-
try. An MOS structure is created by superimposing several layers of conducting, insu-
lating, and transistor forming materials. After a series of processing steps, a typical
structure might consist of levels called diffusion, polysilicon, and metal that are
separated by insulating layers. CMOS technology provides two types of transistors, an
n-type transistor (nMOS) and a p-type transistor (PMOS). Typical physical structures
for the two types of MOS transistors are shown in Figure 2.4.1. For the nMOS
transistor, the structure consists of a section of p-type silicon separating two diffused
areas of n-type silicon. The area separating the n region is capped with a sandwich
consisting of an insulator and a conducting electrode called the gate . The transistors

have two n-type diffused areas, which are designated the drain and the source.
GATE

 

 

 

 

 

mm
DRAW
INSULATOR
/ GATE
SOURCE _L
’ C. , E .. I K/l DRAIN—j—r—SOURCE
p—OOPED
Won some CIRCUIT SYMBOL
n-TMNOIS‘I’OR
GATE
0 ”UK! '5—
DRAIN J—t—sounce
CIRCUIT SYMBOL

 

 

 

Figure 2.4.1 MOS transistor physical structures.

23

2.4.1. MOS Transistors

Recall that a MOS transistor current-voltage characteristic function, say Id, ,
denotes the current ﬂowing from the drain to the source. Denote the source, gate,
drain, and threshold voltages respectively as v,, v , vd, and v,. Then, according to the

square-law theory, the current characteristic function Id, is given as

Cutoff: if (vg - vs — v,) S 0
Ids(vd,v,.vg) = o (2.4.1.a)

Triode: if (vg — vs — v,) 2 (vd — vs)

1,, (v, ,vsyg) = %c,,(W/L)[2(vg — v, - v,)(v,, — v,) - (v, - v,)2] (2.4.1.c)
Saturation: if (v8 5 v, - v,) S (vd — vs)

1,, (v, ,v,,vg) = 12‘-C,,(W/L)[vg — v, - v,]2, (2.4.1.c)

where W/L is the ratio of gate width to length, Cox is the oxide capacitance per unit

area, and u is the mobility of carriers.

2.4.2. Design Considerations

In order to design CMOS VLSI layout, Magic design tools has been
employed. Magic is the backbone of the Berkeley integrated circuit Computer Aided
Design (CAD) software system. With Magic, designers can paint geometry using a
mouse and a graphic display system. The layers painted are not the actual mask layers
used in fabrication. The actual CIF layers are generated by Magic from the abstract

layers.

24

Both the number of pins and the chip area of the neural network circuitry
increase as the number of neurons increases. If we want to put more neurons on a sin-
gle chip with the same chip area, then there is a trade-off between the pin count and
the circuit area. The more pins we put on the same chip, the more area is used to

locate these pins and correspondingly we have less area to locate the network circuitry.

The minimal acceptable pad geometry by MOS Implementation System
(MOSIS) [63] is an 88 x 88 micron glass cut box over a 100 x 100 micron metal box.
The pad geometry is not scalable while the advanced technology is used. In addition,
bonding pads should be placed along the edges of the project, with at least 200 micron
center-to—center spacing. Thus, the network circuit area dramatically decreases as the

number of pins is increased.

The VLSI design project can be packaged in a 28, 40 or 64 pin DIP (dual In-
line Package) or an 84, 108, or 132 PGA (Pin Grid Array) package. Table 2.4.2.1
tabulates the relation among pin count, package type, and cavity size. The MOSIS Ser-
vice offers Standard Frames that specify bonding pad locations and their minimum
sizes. Table 2.4.2.2 shows the available Standard Frames for different project sizes.

25

Table 2.4.2.1 MOSIS package types.

 

 

 

 

 

 

 

 

 

MOSIS
PACKAGE TYPES
PIN PKG TYPE CAVITY SIZE
COUNT

28 0.6“ DIP .310 x .310“
40 0.6“ DIP .310 x .310“
64 0.9“ DIP .400 x .400“
84 1.1“ PGA .350 x .350“
.470 x .470“
108 1.2“ PGA .350 x .350“
.450 x .450“
132 1.4“ PGA .350 x .350“
.450 x .450“

 

 

 

 

Table 2.4.2.2 MOSIS Standard Frames.

 

 

 

 

 

 

 

 

 

Project Size Frame Name (by package pin count)

sq. mm 28 40 64 84 108 I32
7.9x9.2 - - - - - - 64P79x92 84P79x92 - - - - -
6.9x6.8 - - - 40969x68 64P69x68 84969x68 - - - - -
4.6x6.8 — - - 40P46x68 - - - — - - - - - - -
4.6x3.4 28P46x34 40P46x34 - - - - - - - - - - -
2.3x3.4 28PC23x34 - - - - - - - - - .. - .. .. -
2.22x2.25 - - - 40PC22x22 - - - - - - - - - _ -

 

 

 

 

 

 

 

 

CHAPTER 3

A NEW NEURAL NETWORK ARCHITECTURE

A new architecture of feedback artiﬁcial neural nets was proposed in [39,40]
which has been shown to exhibit qualitatively the same dynamic properties as gradient
continuous-time feedback neural nets. This model (i) has a maximum number of con-
nections equal to n(n+1)/2, where n is the number of neurons, i.e., we reduce the
routing overhead for VLSI design. (ii) The synaptic weights are naturally "symmetric"
since there is a single element connecting unit i to unit j. Then, it becomes possible
to view gradient models as valuable and implementable in hardware. In addition to
these two features, this model utilizes simple nMOS transistors as its synaptic connec-
tion, where its conductance is controlled via the gate voltage. It doesn’t require the
realization of linear resistive elements for the synaptic weights. This new architecture

lends itself naturally to analog all-MOS VLSI implementation.

3.1. Motivation from Biology

This new neural network architecture is motivated from biological neural nets
where neurons have dendro-dendritic connections, i.e., connections among neurons
which occur via dendrites only. Consider neural units which communicate via their
dendrites, i.e. the synaptic connection between neuron units is established via the den-

drites. Such synaptic connections are referred to as dendro-dendritic connections.

26

27

Indeed, the Horizontal and the Ganglion cells in the retina are examples of networks
with dendro—dendritic connections [26].

3.2. One Neuron with Self-feedback

Each neuron is a processing device with input u,- and output v,- related by the
usual sigmoid function 5,. It is modeled as an operational ampliﬁer (op-amp) with a
capacitive element C,- and a resistive element R,- at the input node. The neuron may
be realized via two CMOS inverters in series instead of an op-amp. For VLSI imple-
mentation, the capacitive element C,- and the resistive element R ,- at its input node are

eliminated, since the parasitics compensate for their roles.

There is a self-feedback MOSFET with gate voltage Vfb, between input u,-
and output vi. We shall refer to the neuron with self-feedback as a neuron unit. It
represents the building block for the new neural network architecture. The CMOS cir-
cuit model of this neuron unit is depicted in Figure 3.2.1. This neuron unit is capable

of processing two stable equilibria. It is, in fact, a model for a ﬂip-ﬂop.

ui vi

1‘” R _ in...

Figure 3.2.1 One neuron with self-feedback.

 

.4,— -L—

28

3.3. General n-Neuron Architecture

The input of unit i is connected to the input of unit j via a single MOSFET
with gate voltage V6,,- . External input I,- may be injected at node i. The architecture
simply connects neuron unit via their input nodes. The CMOS circuit architecture, for
a 3¥unit prototype circuit, is depicted in Figure 3.3.1. One of the important advan-
tages of this architecture is that between any two neuron units there is a single physi-
cal connection. It preserves the symmetry in the connection required in the theory of
gradient dynamical systems. Moreover, the architecture reduces the maximum number
of connections to n (n + 1)/2, where n is the number of neuron processors in the net-
work and it does not require the realization of linear resistive elements for the synaptic

weights.

vl ul - -
1n
Vfbl :1.- + I

VGl3 v023

 

      
 
 

 

 

 

Figure 3.3.1 A 3-unit prototype circuit of the new architecture.

29

We can also connect the output of unit i to the output of unit j via a single
MOSFET with gate voltage VGij [64]. This output-connected topology has the quali-

tatively same dynamic properties as the input-connected topology.

In planar VLSI implementation, it is difﬁcult to make a global interconnection
among all units in a network, especially when the number of units is very large. Gen-
erally these networks are implemented in an array structure by connections to nearest
neighbors. Retinal structures have a high degree of symmetry and connectivity, and
thus they are attractive for VLSI implementation. This topology has been used to
design the silicon retina in [26]. In order to form a retinal structure network, our units
are tiled in a hexagonal array with six connections converging into the input node of

each unit [64].

3.4. Mathematical Analysis

From Kirchhoff’s Current Law (KCL), applied at every input node, one

obtains the mathematical model for the new architecture as follows:

dui " “i
Ci—dt— = 2 Ids(uj ,u;.VG,~j) + Id,(v,- ,u;,Vfb,-) - 'E- + I,- (3.413)
j I
Vi = Slut), (3.4.1b)

where S,- is a sigmoid function which represent the input and output relation of the
double inverter, 14, is the MOSFET current-voltage characteristic function which is
described in equation (2.4.1), C; and R,- are the parasitic capacitor and resistor, and I,
is an external bias.

With appropriate assumptions and approximations [40], we may consider the

MOSFET characteristic function Id, to be an odd function of (vd - v, ). Moreover,

we note that in this architecture, the gate voltages VGij and VG j; are identically the

30

same, i.e. VGiJ- = VGj-i.

Hence the dynamic equations of the new neural circuit model can be rewrit-

ten as
dui ’3 u,-
Ci 7 = 2 Ids (“j—“i,VGij) + 14; (Vt—“itVfbi) " R— + Ii (3.4.23)
1' i
Vi = S.- (ut). (3.4.213)
Deﬁne “U = u‘. — u}. and “ﬂ = “j - “i- BCCEIUSC VGij = VGji’

Id, (uij,VG,~J-) = —Id, (uji,VGj,-). One can obtain a ﬁrst integral or energy function [40]
of equation (3.4.2) as

1
E=-§ZX

ijset

gull. Ids( xi! VGij )dxi " 2 Lu‘ Ids( 550;) — y,, Vfbi )dy, (3.4.3)

2i R‘ 1'“.

Hence the dynamic system (3.4.2) can be rewritten as

c. __L. = .. ——._ (3.4.4)

= ___' (3.4.5)

SO,

Therefore, this system is a gradient-er system endowed with the properties attributed
to gradient dynamic systems [40, 65]. A theorem [58,65-67] conﬁrms that all solu-

tions converge to equilibria only. No oscillations or any other complicated behavior is

31

permitted in this system as consequence of being a gradient system. Thus the overall

dynamic system is structurally stable.

3.5. Circuit Implementation of the New Architecture

If artiﬁcial neural nets are to be used in large scale, then the medium of their
implementation becomes critical in enhancing the quality of their performance.
Indeed, implementation in large scale is the vehicle through which artiﬁcial neural net-
works would reveal their computational powers. Hardware implementation is natural,

convenient, and more powerful where the mechanism of the proposed neural networks

would be directly realized, and suitably accommodated, into VLSI hardware.

3.5.1. A Neuron

The transfer function of a neuron itself is a monotonic nondecreasing sigmoid
function which can be easily implemented in CMOS VLSI via an op-amp or two logi-
cal inverters in series. The circuit schematic diagram and DC transfer function of a
neuron which is implemented via two CMOS logical inverters in series are depicted in
Figure 3.5.1. 1. An advantage of the implementation using two CMOS logical inverters
in series is its simplicity because a neuron consists of only four MOS transistors. The
circuit schematic diagram and DC transfer function of a neuron which is implemented
via a CMOS operational ampliﬁer are depicted in Figure 3.5.1.2. An advantage for an
op-amp is that the threshold value of the sigmoid function can be adjusted after fabri-

cation.

32

 

 

 

 

 

 

 

5W+ # 4= ,4, .

. x V l ’ 5

.J j : g

u —1 v E E
_, 3 a :

l 1— 3.94.. t .LJ 4 e -+

" "' 9.9V 1.94 2.94 3.94 4.914 5.94 U
(a) Circuit schematic diagram (b) DC transfer function of I/O relationship

Figure 3.5.1.1 A implementation of one neuron via two CMOS inverters in series.

4
i
..J

5
s.

a 5.94%
l

———oVout

 

-_—--------—-----—--- h

 

 

 

:2«pm------—------

—; 9.94 t t 4 e 4Vin+
9 1.9V 2.9V 3.94 4.9V 5.94
(a) Circuit schematic diagram (b) DC transfer function of I/O relationship

Figure 3.5.1.2 A implementation of one neuron via an ampliﬁer.

33

 

Ids

329.9% -------------- 4- ------------- + ------------- 4 .............. t. ............. 3:38:283
. ' SB .
des=2.ov

29am»:r . Jr
EVds=1.ov

199a“; . i,

“3.3 13v 234 afav ' "';.*t'v'""""';'.a¢ V9

Figure 3.5.2.1 Transfer characteristics of a nMOSFET.

34

3.5.2. A Programmable Synapse

It is necessary that synaptic weights can be programmed so as to render a
relative conﬁguration of the neural network which corresponds to various set of stored
memories. Linear resistive elements in electronic neural network can be realized via
analog multipliers which have been introduced [17,20,52-57] to implement the pro-
grammable synaptic weights using a standard CMOS technology. The new circuit
architecture does not require the realization of linear resistive elements for the synaptic
weights. Thus a programmable synapse can be easily realized via a single nMOS
transistor [64,68]. The conductance of an nMOS transistor is capable of being
adjusted via controlling the gate voltage of the nMOS transistor. For a ﬁxed drain to
source voltage, the characteristic of nMOS conductance is depicted in Figure 3.5.2.1.
Although the programmable synapse of the new architecture is modeled as a nonlinear
function because the conductance of nMOS transistor is a nonlinear function of the
gate voltage, the dynamic behavior of the new architecture is supported with good

theoretical analysis.

3.5.3. Storage of Synaptic Weights

The conﬁguration of a neural network is characterized by the synaptic
weights. Therefore, it is necessary to store synaptic weight values in order to preserve
the desired conﬁgurations. There are several approaches to implement the storage of
synaptic weight values, such as on-chip digital memory [23], ﬂoating-gate device [69-
73], CCDs [24,74], capacitors [75], and external digital memory with AD and D/A
converters [17,76].

On-chip digital memory restricts its programmability, but it is useful when the
network has a suitable digital learning scheme and is used in a static mode such as

associative memory and pattern classiﬁer applications. The ﬂoating-gate devices are

35

attracting considerable attention as a nonvolatile analog memory. However, its control
circuitry and scheme are still complicated. Furthermore, the accuracy and repeatability
is difﬁcult to control. To maintain the analog weight, a twin capacitor storage [74]
cell had been proposed utilizing the CCD technique but its control scheme is still com-
plicated. External digital memory with A/D and D/A converters combines two popular
methods: Storing weights as digital words that are converted to analog values, and
storing each weight as a charge on an on-chip capacitor. Rather than implementing a
D/A converter for each weight, only one off-chip, high-precision converter is time
multiplexed to serially refresh all the capacitor charges on one chip. A disadvantage
of this approach, that the analog weight values are quantized by digital word length
and precision of the digital-to-analog converter, is more than compensated by the ease

with which the weights can be manipulated by the digital host computer.

In this work, on-chip digital memory or ﬂoating gate devices were used as the

storage of synaptic weights.

3.6. SPICE Simulations and Laboratory Experiments of the New Architecture

A 3-unit prototype circuit in Figure 3.2.1 has been employed to illustrate and
to test the performance of the proposed circuit. SPICE was used to simulate the tran-
sient evolution to the steady states (or stable equilibria). SPICE parameters for 2-

micron technology are provided by MOSIS.

The voltages (across the three capacitors) at the input nodes of the 3 unit s are
simultaneously initialized. Then the circuit evolves through its transients before set-
tling to the steady state values. The simulation has exhaustively been repeated using
various initial conditions and veriﬁed the convergence of the resulting solutions to

equilibria.

36

Once the interconnect conﬁguration is set, i.e., the gate voltages of intercon-
nect MOSFETs and the gate voltages of the self-feedback MOSFETs are set, the net-
work possesses a speciﬁc set of stable equilibrium states as its only limit set. As one
alters the interconnect conﬁguration, the network will alter the set of equilibria accord-
ingly. In fact, one can adjust the interconnect conﬁguration to obtain any different
number of stable equilibrium states which in turn correspond to different number of

memory data.

By only adjusting the gate voltages VGij of interconnect MOSFETs, the 3-
unit prototype circuit can exhibit all possible number of eqm1ibrium states; for a ﬁxed
interconnect conﬁguration, the number of possible equilibria ranged from two distinct
stable equilibria to eight distinct stable equilibria. Simulation results are depicted in

Table 3.6. 1.

When the gate voltages Vfb, of the MOSFETs across the double inverters are

adjusted, the circuit can then posses a single stable equilibrium state.

In order to verify the SPICE simulation, a discrete-component realization of

the 3-unit prototype circuit has been built and its performance tested in the laboratory.

Every CMOS inverter is realized by one inverter of the 74C04N which is a
single stage and unbuffered CMOS inverter. Every n-channel MOSFET conductance
element is realized via a 2N4351. Table 3.6.2 depicts the experimental results of a 3-

unit prototype circuit.

In the discrete-component realization, the component variations such as thres-
hold voltages and transistor size, will make the experimental results different from the
SPICE simulations. However, the experimental results are qualitatively identical to the

SPICE simulations.

37

Table 3.6.1 SPICE simulation results of a 3-uint prototype circuit.

 

Vfb1=5.0V,i=l,"',3

 

 

 

 

 

 

 

 

 

 

 

 

Double Inverter nMOS W/L = 4.0/2.0
pMOS W/L = 12.0/2.o
Interconnect Transistor nMOS W/L s 4.0/2.0
Feedback Transistor nMOS W/L = 8.0/2.0
(VG 12, VG 23, VG 31) # Of Stable Equilibria Stable EQUIIIDTIB (V1, V2, V3)
(5.0, 5.0, 5.0) 2 (0,0,0).(1,1,1)
(2.6,2.7,5.0) 3 (0,0,0),(l,l,1),(l,0,l)
(0.5,0.5,5.0) 4 (0,0,0),(l,1,1),(l,0,l)
(0, l, 0)
(2.8,2.8,2.8) 5 (0,0,0),(1,1,1),(1,0,l)
(1, 1,0),(0, 1,1)
(2.7.3.3,22) 6 (0,0,0),(1,1,1),(1,0,1)
(1,1,0),(0,l, l),(l,0,0)
(2.6,2.6,0.0) 7 (0,0,0),(l,l,l),(l,0,l)
(l, 1,0),(0, l,l),(0,0, l)
(l, 0, 0)
(00.00.00) 8 (0,0,0),(l,l,l),(1,0,l)

 

 

(l, 1,0),(0, 1,1),(0,0,l)
(l,0,0),(0, 1,0)

 

 

38

Table 3.6.2 Experimental results of a 3-ta‘nr prototype circuit.

 

Vfb,=5.0V,i=1,---,3

 

 

 

 

 

 

 

 

 

 

 

 

Double Inverter the MM74C04N
Interconnect Transistor nMOS the 2N4351
Feedback Transistor nMOS the 2N4351
(VG ,2, V623, VG 3,) # of Stable Equilibria Stable Equilibria (v1, v2, v3)
(5.0, 5.0.5.0) 2 (0,0,0),(l, 1,1)
(1.4,5.0,1.75) 3 (0.0.0).(1.1.1).(1.0.1)
(0.0,0.0,5.0) 4 (0,0,0),(l,l,1),(l,0,l)
(0. 1. 0)
(1.35, 1.9.1.75) 5 (0,0,0),(l,1,1),(1,0,1)
(l, 1,0),(0, l, l)
(1.4, 1.85.1.75) 6 (0,0,0),(1, 1,1),(1,0,l)
(l, 1,0),(0, 1, l ),(l,0,0)
(l.4,l.9,0.0) 7 (0,0,0),(1,l,l),(1,0,l)
(1,1,0),(0,l, l),(0,0,1)
(l, 0, 0)
(0.0,0.0,0.0) 8 (0,0,0),(l,1,l),(l,0,l)

 

 

(1, 1,0),(0, l,1),(0,0,1)
(l,0,0),(0, 1,0)

 

 

CHAPTER 4

LEARNING
OR THE DYNAMIC UP-DATE OF "WEIGHTS"

Learning is a key feature of artiﬁcial neural nets that critically affects
hardware designs. Learning schemes are usually used to determine the weights from a
set of training examples [30,33,34]. In some cases, neural networks have been shown
to be capable of learning the input-output relation from a ﬁnite set of examples,
deﬁned as the training set, simply by minimizing a given measure of the error from the
training examples over the network’s parameter-space. This minimization is called the
learning process. Because the learning algorithms generally takes orders of magnitude
longer than the operation of ’reading ’ the network, most applications requiring fast
adaptability cannot yet be implemented in hardware. Consequently, non-adaptive
applications requiring weights to be set only once (not in real time) should be
developed ﬁrst [21]; in that case learning algorithm efﬁciency is not an important
issue.

Different types of learning deﬁnitions may be formulated [31,32]. We limit
attention here to the so-called supervised learning [31], and in particular for dynamic
feedback models of ANNs. In the context of ANNs, supervised learning loosely
means that the network acquires a desired set of data (vectors) as stable equilibria in

the case of dynamic feedback ANN s.

39

40

Even in theory, learning for neural nets, particularly feedback neural nets,
remains a very difﬁcult problem. The main reason is that the models of ANN s are
nonlinear. When the model of the ANN is dynamic, as Oppose to static, its dynamics
will in_te_rac;_t with the dynamics of a learning algorithm (or an update law). This type of
problem has plagued the development of the so-called robust adaptive algorithms for
linear systems in the area of adaptive control [77]. The situation is perhaps worse in

ANNs because the model of an ANN is nonlinear.

Learning for static models, such as the case for feedforward ANNs [34], is
conceptually simpler because the model does not have dynamics -- it is only a static
map. Thus the problem of coupling between the model and a dynamic learning algo-
rithm does not exist. This, to some extent, explains the relative success of the gradient
learning algorithm popularized as the error back-propagation [34].

However, in electronic implementations, the implementation of a static model
(or a map) results in a dynamic model, nonetheless. This is caused by physical reality
and by the presence of parasitic capacitors, resistors, transistors, etc.. Consequently,
one is faced with the possibility of interaction or coupling between the dynamics of the

ANN and its dynamic learning algorithm.

In this work, a dynamic learning algorithm is introduced for general dynamic

continuous-time models of ANNs.
The learning algorithm

Consider the general dynamic model of ANN s. (In fact, it may be any general
dynamic model.)

i.- = 7" =f.-(x.p). «Hal

1’: = Sit-(xi). (4.11))

41

where x = [ltl,...,tt,,]T is the n—d state vector, p = [p1,...,p,]T is an l-d parameter
vector which needs to be updated (dynamically). (Supervised) learning means that a

1.

given vector, say it ‘ (or a set of m vectors x ,...,x"’") becomes a stable equilibrium

point. It is assumed, of course, that solutions of equation (4.1a) exist and are unique

for each initial condition 1(0), as it is often assumed for differential equations.

Suppose that we desire the learning algorithm to "learn" a given vector x‘.

First, we deﬁne the energy function for the learning scheme as follows:
1 " l "
= EVE-Eat. )2: 3211- f(x,p)2. (4'2)

The learning scheme updates each parameter according to the dynamic equation:

de _ BE _ " . ain‘rP) _ " afi(xsp)

7t— --'aEf--i§xt apj --i§fi(x.p) apj (4.3a)
Thatrs,

if": -f(x p)* Df(x 12) (43b)

where f (x, p) = [f1(x,p),...,f,,(x, p)]T is the vector ﬁeld of equation (4.1a) and
Df (x , p) denotes the vector composed of the partial derivatives on the right-hand side
of equation (4.3a). Note that in the case that p is a matrix, each element of p will be
speciﬁed by two indices.

The algorithm proceeds as follows. Clamp, i.e. ﬁx 1: in equation (4.3) at the
desired value, say it = x' . Let the dynamics of equation (4.3) evolve until they con-
verge to a value for the parameter, say p'. For x = x’ , the system of equation (4.3) is
a gradient dynamic system and hence p I must be an equilibrium point. In computer
simulations or in implemented circuit hardware, the dynamics realistically converge to
only stable equilibria. Hence, in practice, p I is a stable equilibrium point. The vector
p' is also a minimum point of the energy function in equation (4.2), where x = x' .

When equation (4.3b) converges to p ' , we have

42

0 =f(x*.p’) .Df(x'.p").

This means that (at least) one of the following is true:
Case (1): f(x’,p') = 0.

Case (2): Df(x',p') = 0

Case (3): the vectors

f(x'.p') and Df(x’.p').
are perpendicular to one another.

If case (1) is true, then the point it. is an equilibrium point of (11) for
p = p'. If the "physical" system or circuit does not permit case (2), then case (1) is
the only possibility. In addition, if the model of equation (4.1a) is ﬂexible enough, then
it can support the (augmented) equilibrium (x' , p') as stable equilibrium. Idealy if we

I! n
plug p' in equation (4.2), E = -;—2 (i,- )2 = ‘2': fi(X, p")2 = 0. Hence inserting p'
i=1 i=1

in equation (4.1) renders the desired vector x ' a stable equilibrium of equation (4.1)

The desired vector can be retrieved by initial conditions sufﬁciently close to x '.

CHAPTER 5

A PROTOTYPE 6-NEURON CMOS TINY-CHIP

A 6-neuron Tiny-chip of the all-MOS implementation of the new architecture
has been designed and fabricated. Complete testing of the Tiny-chip [64,78] has suc-
cessfully substantiated the predicted operation of the neural circuit. A dedicated inter-
face circuitry has been built to facilitate the programming of the chip via learning
algorithms developed onto a software environment with graphics display on a Personal
Computer (PC). Using the interface circuitry and the learning schemes, we demon-

strate the use of the Tiny-chips of the new neural circuits as a classiﬁcation device.

5.1. Layout design

A VLSI layout was designed for a 6-unit neural circuit on MOSIS Tiny-chip
as a prototype chip of the new architecture. This layout was fabricated by MOSIS
using an n-well 2 um CMOS process, with 2.2 x 2.2 mm2 chip area in a 40-pin pack-
age. The schematic diagram of this 6-unit neural circuit is depicted on Figure 5.1.1.
Every node in the neural circuit is connected to an external pin to facilitate testing.

This includes the input and output of every neuron and the gates of every connecting

 

MOSFET. For the 6-neuron chip the number is 2x(6) + (6)x((26)+1) = 33. Analog

bonding pads are used in this design, i.e., there are no buffers between the I/O pins

and our designed circuit. Thus, we can easily verify the characteristics of this neural

43

. it

V)

e

>
‘9 j.

.285 go: c.3470 “.2858 .33 a we game 3328 BE. 2d am

4..

 

 

at

 

 

 

 

- - ..w
.. - ..a- .am can lam; Hui
.. re

 

 

Y T El

. , 13.191.
:1
41—.

 

 

41

 

4t

 

 

-H . _ _ . H
._ ... ...... ...... ...... .42.. a.

 

 

YT_“
.21
3.

11%|

 

 

 

 

 

 

 

41

 

41

 

 

 

 

 

.1 1%“

h. H H
1.4 in same same team a: H
c1

 

 

Ta.
9

’ if

D I
u-
>

45

.1
ﬂ

 

l‘l

 

 

 

 

 

 

 

Ndif:-C{

 

 

 

 

 

... up. .n . ”an“
m may mm.
» mm".m.u.n.n.n...mmmm

 

D109

 

  

,u 3x3/3f3/3/3,
agﬁééfq,
f3 3 3 3 3.». _ .3 3
,xx :x. fixxx.
qééé:ééa,
3.3.3.5.. 3- 3 3 3..
f .Y .‘x ... ..x ..., ..x ..x 7
.../..._.....,.,.,,...,.-,.,..,-§.
ﬁSﬁﬁQﬁﬁﬁ:

J
)iii‘tritr‘tr

      
          

9019-6,

  

-c

Mae]!

-0

Pdiff

P-Diff

 

 

 

 

 

 

 

 

 

 

 

 

 

Figme 5.1.2 the layer names and their corresponding symbols.

Vdd!

I:
a.

Vfbi

Vi

 

Figure 5.1.3 A VLSI layout of one neuron with self-feedback.

47

 

s A §
9625 VGSNGSS

A § A
961N015 902

L
was

3 §
98119612

Figure 5.1.4 A VLSI layout of the interconnect network for the 6—unit neural circuit.

'1!“°J!° [91“9ul1W‘9 9'43 5° 1005121 ISM V S'I'S omﬁyd

48

  
    
      

   
       

         
  
 

   
 
   
  

  

 

      

 

c 5 . 5 g 5
o o 0 ﬁ. w M E u- S
‘f ‘3 ix \3 \2 1: : : :
I' .V U‘ "3 :V :: :“.~"".\‘:~’ :ﬂ’k :ﬂ 9‘ “I I U' “I ‘ ‘l 6’“ “0"“
:: 3 g: :‘V t‘ v‘ “:‘h:“:lh:“: «"3 "\J " 0' W 0‘ 9‘ ~’k‘ “I“ “‘3'“
.. \ \ L V ’ ‘W‘ﬂw'ﬂ‘ A l l N h
yﬁﬂiw } Q ﬁﬁwwm} : ‘ H waﬁmk‘ 3 \ \ ﬁﬁﬁ
“a“ “ "31"."."1' a a a u‘a u at I - t . o'a .
I“ :x 11 {M‘tﬂ‘tﬁ‘ 9H \ m: i“
“'ktw ‘\ .. \ﬁ“\\'1: \\‘ ‘3 . \‘ \ ‘uﬁ ‘
\A ‘ \ c 1‘" I: “ \\ﬁ‘ \1 ‘ 1' “ i‘ ' 1* \7-*\ A“, .\ ~ “- - - -‘ ‘1“
N 0 I‘N‘ .‘ r." g- _k A ”u a gm 73411 N 4‘; .;. V a ..u g}
‘uw QWA¥VA“a. \mw mmﬁw Mun' Jaw
“*\u1 . ‘1‘ h‘ K \ 1 “‘3. ‘ _ r ’ he
9“» u ‘t‘ *0 ~. ‘
ﬂ \
\H
\‘:‘\ﬁ
1‘ v .‘
\
\‘A x \'
\H , -1 1
I“ 1,1 .
I , 5 a
\‘ﬁ‘ 1 v‘ 3'11" v" ‘9‘ '5“ ‘1
-- * - . . -ﬁx g
u ‘

   

ZINIM
V V

cam
Y
L.
”on

m
r/

”3%

Y
b. x k
HMHMSM

neon saw:
V § V V

sconcm
3.
been

 
  
     

yﬁpﬁ
‘A' J N."

{\H‘E‘xu‘ i

M
3010
in
“HA

49

"UNI! II!" |.'

.5 I” “mummy-out:

cl
'"iICl-"Io I Haw.

 

Figure 5.1.6 A die photo of the 6-unit neural circuit.

a:

canal u (muting ’

‘n

i: y
‘1 C

l'rlv_
hr .4
V,. i.
’ 3

I h

“u \‘ukk.llglli'3lui’;;

...-.._~.__-.,.._. -..—......

i
!

 

Figure 5.1.7 A die photo of the 6-um‘t neural circuit in MOSIS Tiny-Chip.

50

ﬂ
Q

I.)

v6

v4

Vfb4

VDD

114

Vle

v2

0ND

V025

V023

V012

V026

VDD

V046

V024

v1

mmmmmmmmﬂmmmmmmmm

 

O

Vfbl

 

V045

V056

V034

V036

0ND

V014

V016

V035

V013

VDD

V015

v5

0ND

v3

Vfb3

u3

ul

40

awn

6

35

IEJLJULEJ

E
22']
E]

t»
O

N
0

[#3112113] {ﬁlElL'é‘llElULJ

t4
.—

 

 

Figure 5.1.8 A pin conﬁguration of the 6-neuron chip.

51

circuit and examine the performance of this chip design. Figure 5.1.2 depicts the layer
names and their corresponding symbols. Figure 5.1.3 depicts the VLSI layout of one
neuron with self-feedback. Figure 5.1.4 depicts the interconnect network with 15
nMOS transistors for the 6-unit neural circuit. Figure 5.1.5 depicts the VLSI layout of
the 6—unit neural circuit corresponding to Figure 5.1.1. Figure 5.1.6 shows the die
photo of the 6-unit neural circuit corresponding to Figure 5.1.5. Figure 5.1.7 shows
the die photo of a 6-unit neural circuit in a MOSIS Tiny-chip with a 2.2 x 2.2 mm2
chip area and a 40—pin package. Figure 5.1.8 depicts the pin conﬁguration of the 6-

neuron chip.

In the testing process, we set the gate voltage of each feedback MOSFET to
zero, then we measured the I/O characteristics of each double inverter. The resulting
I/O function is indeed a sigmoid curve with a switch-point around 1.57 volts. Using
the new SPICE parameters obtained by MOSIS from measurements of the MOSIS test
structures on the selected wafers of this speciﬁc fabrication lot, the SPICE simulations
have almost identical result. We used a 3-unit subcircuit for more testing to compare

the results to the SPICE simulations.

We can adjust the gate voltage of the interconnected MOSFET conductance
elements among the 3-unit subcircuit to obtain a different number of distinct steady
states or stable equilibrium points. The experimental results are depicted in Table
5.1.1. The testing result is highly repeatable. We tested all four chips obtained from
MOSIS for the same circuit design. The results for all four chips are similar to one

another.

Complete testing of the Tiny-chips has successfully revealed the proper opera-
tion of the neural circuit [64,78].

52

Table 5.1.1 Experimental results of a 3-unit subcircuit of the 6-neuron chip.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Vfb. = 5.0 volts for i=1,2,3.
Initial States
Number of
“’61” V023' V631) 000 001 010 011 100 101 110 111
Stable Equilibria

Steady States
(5.0,5.0,5.0) 2 000 000 000 000 000 000 000 111
(32.3.5.5.0) 3 000 000 000 000 000 101 000 111
(0.0,0.0,3.7) 4 000 000 010 010 000 101 010 111
(3.13.134) 5 000 000 000 011 000 101 110 111
(32.3.5.0.0) 6 000 001 000 011 000 101 110 111
(2.7.2.100) 7 000 001 000 011 100 101 110 111
(0.0,0.0,0.0) 8 000 001 010 011 100 101 110 111

 

 

 

 

 

 

 

 

 

 

 

 

5.2. Interfacing with a Personal Computer

To ensure testing of the implementation and also to provide an environment

for developing and assessing various potential learning algorithms, we planned at the

53

design stage for interfacing the Tiny-chips with a PC. An interface [78-80] was
developed such that the gate voltages of the 15 MOSFETs, which are the interconnect
transistors among the input nodes of the 6—unit neural circuit, can be ﬂexibly read or

assigned analog values between 0 and 5 volts.

Only a single 8-bit D/A converter (DACOSOO) is employed to convert the digi-
tal data to the corresponding analog voltages. The analog outputs of DAC0800 are
multiplexed to the gate voltages of the 15 MOSFETs via analog multiplexers
(CD4051). Fifteen Sample and Hold Ampliﬁers (LF398N) are required for holding the
analog voltages at the gates of the 15 interconnect MOSFETs. To keep the precision
of each gate voltage, the Sample and Hold Ampliﬁers should be refreshed within a

certain time constant which depends on the size of the storage capacitor.

Two 8-bit 8-channel AID converters (ADC0808) were employed to simultane-
ously read out the steady states at both input node and output node of a single neuron.

The output of the ADC0808 is stored in an octal latch and then transferred to the PC.

The exact initial values are dependent on the outputs of the octal latch
(speciﬁcally, 74LS373). The analog voltage of the octal latch output is about 3.5 volts
when the corresponding input is high and is about 0.22 volts when the corresponding

input is low.

5.3. A Learning Scheme

The learning algorithm described in Chapter 4 was specialized for the 6-unit
Tiny—chips. The learning algorithm was implemented in software on a PC [30]; it
interacts with the chip in real-time via a special purpose interface described in the pre-

vious section.

54

5.3.1. Dynamic Learning Model

Assumption: In equation (3.4.1), assume that I,- = O, and the local resistance,
R,- , is quite large. This is reasonable since R,- represents an input to a double inverter

[1]. Thus, equation (3.4.1) can be rewritten as

du. n
Ci 7;" = 2 1&(uk,u,-'VGh-) + Id‘,(vi'u,-.Vfb,-). (5.3.1.1)
A:
We deﬁne the energy function for the learning scheme as the following:
1 " . 2
E = —2 (Ciui ). (5.3.1.2)
21:1

Therefore the learning scheme is governed by the following dynamic equations:

 

 

 

 

 

 

 

 

 

 

 

dVGjr = 315‘
dt aVGﬁ
fag-.2,- , acid,-
— ‘C‘u‘ _aVGj,] - Ciuf [8V6],-
=—Cidi BId,(uJ-,u,-.VGJ-,-) -—Cu Bld,(u,-,uj.VGJ-,-)
‘ 8V0]; J J J b 8V0],-
,BI uo,u,-VG-,- ‘ P—BI u-,u,-VG-,~
=-Cidi d3( 1 . J) -C'-d 41(1 , I)
. . 3143(ujﬂiy0ji)
= [Ciui ”C‘“‘][ avoj,
= mji(uj,ui.VGj;)[CJ-uj-C,-11,-] (5.3..13)

where Cjtij and Cid; may be obtained from equation (5.3.1.1) and iji(uj,u,-.VGj,-) is

the transconductance of a MOSFET which can be expressed as follows:

cutOff: if (VGﬂ "" X "" V‘) S O

55

iji(uj,u,-’VGj,-) = 0 (5.3.1.3a)
Triode: if(VGﬁ -X — v,) 2 |(uj - “0|

iji(uj,u,-’VGj,-) = Kat,- — 11,.) (5.3.1.3b)
Saturation: if (VG); - X - v,) S |(uj — “0'

0:11,, (uj,u,-.VGJ-,-) = K(VG,-,-—X -v,) (5.3.1.3c)

where v, is the threshold voltage, X is the minimum of uj and u,-, and

K = uCox (W/L).

5.3.2. Software Implementation of the Dynamic Learning

The learning scheme is accomplished via software implementation using the
Runge-Kutta fourth order integration routine running on an AT personal computer. A
hardware interface was built so that the gate voltages of the ﬁfteen MOSFETs, which
are the connection transistors among the input nodes of the 6-unit neural circuit, can
be controlled, i.e., analog voltage values read and written onto them. The Tiny-chip
neural circuit can be initialized by the interface at the input nodes; the initial values
are dependent on the outputs of the one Octal latch (74L8373). Each output bit of the
Octal latch controls one input node of the neural circuit. The analog voltage of the
Octal latch output is about 3.5 volts when the corresponding input is high; otherwise
the output is about 0.22 volts. The square law MOS transistor model which was
described by equation (2.4. 1) is employed to simulate the interconnect and feedback
MOSFETs. The parameters of the transistor model are obtained from MOSIS after the
fabrication of the chip. The learning scheme communicates with the actual 6-unit

neural circuit via the computer interface.

The learning procedure is summarized in the following:

56

Step 1:

The desired training pattern is represented in binary

format and is set by the computer interface.

At every input node of the 6-unit neural circuit, "0"
means 0.22 volts and "1" means 3.4 volts as set by

the interface.

The initial values of the gate voltages of the
interconnect transistors are also set by the
computer interface. In this procedure, all gate voltages are initialized

at 5 volts (or the high level).

Step 2:

The actual analog voltages of the input and the output
nodes, obtained from the interface are used to compute the
new gate voltages of the interconnect MOSFETs according

to dynamics of the learning scheme in equation (5.3.1.3).

Step 3:

The new gate voltages of the interconnect MOSFETs are
used in the chip by the interface. Then the process

is repeat. Go to Step 2 until the change of the gate

voltages reaches a chosen stoppage criterion.

A computer graphic environment displays the interconnected architecture as

well as the updating of the analog values for each interconnect gate voltage. After the

57

stoppage criterion is satisﬁed, the personal computer will automatically test all possible
initial conditions (all combinations of high and low) and display them next to their
corresponding steady states at the input nodes of the units. The whole testing pro-
cedure is sequentially displayed in the computer graphics. Using this procedure, one
can verify that a given pattern is indeed stored as a stable equilibrium point. Moreover,
one can test all the binary initial conditions and determine the steady states to which

they converge.

5.4. Chip Experimental Results with Learning

A desired pattern (i.e., a desired stable equilibrium point) is represented as a
binary number (X6X5X4X3X2X1), where X,- (1' =1, - - - ,6) represents the logical
value of the desired steady state at the input node (or at the output node). We set the
gate-voltages of the feedback MOSFETs to logical high (i.e. about 5 volts). When all
the gate-voltages of the 15 interconnect MOSFETs are high, the neural circuit would
have only two stable equilibria, namely (000000) and (111111) [13,14]. All initial
conditions applied at the input nodes of the neural circuit will converge to either one
of these two states. These two states shall be referred to as the no-inforrnation states.
We note that these no—information states will remain regardless of the values of the
gate-voltages of the 15 interconnect MOSFETs [13,14]. The initial conditions applied
at the gates of the 15 interconnect MOSFETs are set at 5 volts. Now, we apply the

outlined learning procedure in section 5.3.2.

Storing a single pattern

The test results show that the learning scheme can successfully store any
desired pattern. The complement of the desired pattern, however, will also be stored.

That is, the neural circuit would now store the desired pattern, its complement and the

58

always-present no-information states. If appropriate bias or restriction on the values of
the gate-voltages of the interconnect MOSFETs are applied, the complement of the
desired pattern can be eliminated.

Storing multiple patterns

Multiple patterns can be applied sequentially. After the ﬁrst pattern is applied,
the learning scheme would converge to a set of gate-voltages for the 15 interconnect
MOSFETs. As the next pattern is applied, the last values of the gate-voltages of the
15 interconnect MOSFETs are used as initial conditions for the learning procedure.
The test result shows that multiple desired patterns can successfully be learned sequen-
tially. We found that each time a new pattern is learned, other patterns of intersection

images among the desired patterns and their complements will also be stored.

Experimental examples

Figure 5.4.1 depicts a graphic display after each of three patterns was learned
in a sequential learning experiment. Figure 5.4.1.a displays the ﬁrst desired pattern,
namely (010111), which looks like the capital letter "T". The display also shows the
steady state analog values of the "converged" gate—voltages of the 15 interconnect
MOSFETs. This set of gate-voltages enables the network to store (as stable equilibria)
the pattern (010111), its complement (101000), in addition to the two no-information
states (000000) and (111111). The right-hand side of the display depicts the roster of
the distinct steady states and the corresponding count of initial conditions converging
to each steady state. In the roster, 11 (distinct initial conditions) converged to the
desired pattern, 11 converged to its complement, 5 converged to the all-one state, and

37 converged to the all-zero state.

Figure 5.4.1.b shows the second desired pattern (111100), which looks like the

capital letter "L", and its corresponding (converged) gate-voltages of the 15

59

interconnect MOSFETs after the network had learned the second desired pattern. I
There are 8 distinct (stable) equilibria stored in the network, the ﬁrst desired pattern
(010111) and its complement (101000), the second desired pattern (111100) and its
complement (000011), the intersection pattern of the ﬁrst desired pattern and the
second desired pattern (010100) as well as its complement (101011), and the two no-
information states. The roster shows the number of initial conditions that converged to
each steady state.

Figure 5.4.l.c shows the third desired pattern, (101111) which looks like the
upper half of the capital letter "O", and its corresponding gate—voltages of the 15 inter-
connect MOSFETs after the network had learned the third pattern. There are 15 dis-
tinct (stable) equilibria stored in the network. We observe that the intersection pattern
of (010111), (111100), and (101111), is not stored in the network, but its complement

is.

 

 

 

u3

 

S. S. #

000000 37

010111 11

101000 11
111111 5

   

   

[:1

u6g

«mvms 1mm” w‘ﬁgvom
tmvms (rmvczs w{§_3f8§jv024
. .Ejvoms @vcss t~['_l_'f_5§3__]VG34
«me (@vmsjm VG23

1@VG% mm! Mme

Desired Pattern: 010111
Figure 5.4.1.a A graphic display after the network had learned the ﬁrst desired pattern.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

      

 

 

 

 

 

 

u3 u2 111 8.8. #

- ' E]: 000000 27

000011 9

010100 9

— _ _ mom 3

u - E, A u A 101000 9

1@V016 «@vms o-l'f_73‘0_8_—_|v014 {911%,} g

«@v0261@v025 (@VG24 111111 1
«millvcaé wlﬂjvoss 1~E§§_5‘IVG34
«@304. ﬂitjvms mm

 

 

 

 

 

 

 

tqﬂjvcss mvmiI-EEVGB

Desired Pattern : 111100

 

 

 

Figure 5.4.1.b A graphic display after the network had learned the second desired pattern.

 

 

 

 

     

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

“3 “2 111 S. S. #
000000 18
000011 3
000111 3
— _ ...... 9
u - - ,_, - 010011 3
(gimme magmas {maven 3:312? 3
k ‘ ‘ 101000 3
l~1172_‘7_=ElvczéHEvms @11ng 101011 1
- - - 101100 3
1413193ch «~("2_T_l’l_Ijvc335 «@VGM 101111 1
- L 3 111000 3
"mm“ 1mm“ TEL—Was {ﬂag ;
'mvox rmvmi mm “1111 1
Desired Pattern: 101111 'F

Figure 5.4. l.c A graphic display after the network had learned the third desired pattern.

CHAPTER 6

A 50-NEURON CHIP
WITH DIGITAL HARDWARE LEARNING SCHEME

A digital realization of the learning algorithms was developed for the new
analog feedback ANN. A VLSI layout of a 50meuron CMOS analog chip with on-
chip digital hardware learning scheme was designed and fabricated with a
6.8mm x 4.6mm chip size using a MOSIS 2-ttm scalable CMOS technology and a
64-pin standard pad frame [14]. Extensive testing of the 50-neuron chip has success-
fully substantiated the predicted operation of the neural circuit.

6.1. A Digital Learning Scheme

Subsequently, the new algorithm described in Chapter 4 is specialized to a
digital learning scheme which is realizable and valuable in the all-MOS VLSI imple-
mentations The digital learning scheme is also tested on the 6-neuron Tiny-chip for

successfully storing and retrieving arbitrary digitized images.

6.2. Hardware Overview of the 50-neur0n Chip

The digital learning scheme can easily be implemented via simple logic circui-

try. The layout design of the 50-neuron chip had been fabricated by MOSIS via a

61

62

 

0n Lx‘rlmdeLV D at“

.aaaaauaaaaaaaag

     
  

Fﬁﬁﬁﬂfgr

he

._ '1'??? H F H" in

" v:

#5:.=:.—.=:—.— “’L—ZE?;éZ-:. £.._ —_ t =.
,

Figure 6.2.1 A die photo of the 50—neuron chip.

 

 

 

63

standard CMOS 2-um n-well technology. Figure 6.2.1 shows the die photo of the 50-
neuron chip. From a conservative estimation, a 124-neuron chip can be designed and

fabricated via a MOSIS 1.21m: CMOS technology on the MOSIS large chip of a
79001111: x 9200um chip size.

6.2.1. A Neuron

Each neuron is represented two CMOS inverters in series with one feedback
nMOS transistor between the input node and the output node. The gate voltage of the
feedback nMOS transistors can be adjusted globally. Either the input node or the out-
put node of the neuron will be connected to an external pin via a CMOS analog
switch. Figure 6.2.1.1 depicts the schematic diagram of a single neuron circuit. Fig-
ure 6.2.1.2 depicts the magic layout design of a single neuron circuit with a
154m x 65pm chip size. Figure 6.2.1.3 depicts the SPICE simulation results of a
single neuron while the gate voltage of the feedback nMOS transistor (Vfbi) is set at 0
volt and the I/O selector is set at 0 volt. The horizontal axis of Figure 6.2.1.3
represents the signals applied at the external pin. The vertical axis of the Figure
6.2.1.3.a represents the signals at the input node of a neuron. The vertical axis of the
Figure 6.2.1.3.b represents the signals at the output node of the neuron which is indeed
a sigmoid function. The vertical axis of the Figure 6.2.1.3.c represents the difference
of between the signals at the input node of the neuron and the external pin which is

less than 500nV. Figure 6.2.1.4 shows the die photo of a single neuron circuit.

6.2.2. A Programmable Synapse

In a single 50-neuron chip, there are 1225 synaptic weights which are pro-

grammable and can be set up via on-chip digital hardware learning circuitry or

 

External Pin

 

j |_‘Lio:erector

 

 

 

k

feedbac

LIP. .m

 

Jlrlll?

I;

 

Figure 6.2.1.1 A schematic diagram of a single neuron with self-feedback and I/O selector.

      
 

—.‘..Vn.1rn..rp_.r..
’5) .3551:
11‘

       

        
 

             

 

a ._ ..
..ﬁ,.u,”,,..
3.255555}
5 b Jruan-b'uﬂrqw.
......m.«.o.......

     
 

       

 

 

     

. v.5...» a. in.
1.5.5.. ...»... .r.
... 55.55:,

       

                 

    

.. if... “4 ;
...... gum .
. \é

. _ .

2. a

. r . , ,
__ . 5”,»,

...... .. _.
EH4. ”1..
H. "....

. ..

.5. ...—9... .
..
a

5‘.
F

   

         

   

......
......
.

1rcuit.

1e neuron c'

mg

6.2.1.2 A VLSI layout of a s

1gure

F

65

"'1!
1

;EL3; .
4 [I [1513+ ------------ + ———————————— + ———————————— + ............ +— ——————————

 

Figure 6.2.1.3 SPICE simulation results of a single neuron circuit.

 

Figure 6.2.1.4 A die photo of a single neuron circuit.

21323333,. ,3. 9
..amﬁwﬁx .
i "H‘r'j-m I . .

-
‘3nnr-n' '. -—»n'Ir-_ :l'l""‘ *7 > .
= czar-31.3.3. w;....;.....x}.u~. :
.. 5 I 1, -7 ~ 145'

 

Figure 6.2.2.1 A VLSI layout of a single synaptic weight circuit.

67

via direct assignment. Each synaptic weight, i.e., each gate voltage of an interconnect
nMOS transistor, exhibits either logical high or logical low and it can be stored on an
on-chip digital ﬂip-ﬂop. The logical high and the logical low can globally be set at
any analog values. Each synaptic weight can be set at any logical value individually
via direct assignment logic circuitry. Figure 6.2.2.1 depicts the magic layout design of
a synaptic weight circuit in 15411»: x 112nm chip size. Table 6.2.2.1 depicts the

different operating functions and their corresponding control signals.

Table 6.2.2.1 Control signals for the operating functions.

 

 

 

 

 

assign LID ff-enable
Direct assignment 0 1 X
Learning 1 0 1
Initialization l l 1
Running 1 X 0

 

 

 

 

 

 

Figure 6.2.2.2 depicts the SPICE simulation results of a single synaptic weight
circuit for the learning mechanism, while the as_sig£ is set at 5 volts, the data is set at
5 volts, the high is set at 4 volts, and low is set at 2 volts. When the time is at 3, 27,
or 51 us , the VGU is initialized to high because the data is set at 5 volts. When the
time is at (3+30*i+6*j) 11s, for i = 0,1,2 andj = 0,1,2, the V60 is modiﬁed accord-
ing to the the digital learning circuitry and the signals applied at the u,- and 14,-. Fig-
ure 6.2.2.3 depicts the SPICE simulation results of a single synaptic weight circuit for
the direct assignment mechanism, while the £381 is set at 0 volt, the high is set at 4

volts, and low is set at 2 volts. When the time is at 3+6*i 11s, for i = 0, - - - , 6,

68

   

ff-enable

...Iif‘l."+~~—--~——~—--——+ ——————————————— .1---____,,.-_-___1 ---------------

T ime

 

Figure 6.2.2.2 SPICE simulation results of a single synaptic weight circuit for
the learning.

69

data

"$11-34.--

vl3li

 

_I.[I‘:'+--------~-_—-+ .......... .1,
.01'+---—.t—--—+—--—4——--+~—.~'-------+-~-———‘----‘ ————————— 4
. v(27)

 

Figure 6.2.2.3 SPICE simulation results of a single synaptic weight circuit for
the direct assignment.

70

V00 will be set at the high or the low according to the data, if both u,- and 14,- are
simultaneously at logical high. From Figure 6.2.2.2 and Figure 6.2.2.3, the functional-
ity of a single synaptic weight was veriﬁed, before we have the design fabricated. Fig-

ure 6.2.2.4 shows the die photo of a single neuron circuit.

6.3. Chip Experimental Results

The fabricated chips were received on October 19, 1990. Extensive testing of
the 50-neuron chips has successfully substantiated the predicted operation of both the
neural circuit and the on-chip digital learning circuit. A desired equilibrium point can
successfully be stored into the chip by setting the interconnect weights via digital
hardware learning circuitry within 50 us. All initial conditions converge to the steady
states in the order of 20 us .

From experimental measurement using a digital meter, the power consumption
of the neural chip in steady state is less than 1 mWatt with power supplies of 0/5
volts. The neural circuit safely retains the same dynamic properties when the power
supply high level is decreased down to 3 volts to dramatically reduce the power con-

sumptions.

t'drtlvawu 1~| l1

.‘t
.’
I
l

ill

7
5
I.
'.
1

 

Figure 6.2.2.6 A die photo of a single synaptic weight circuit. C

CHAPTER 7

A REAL-TIME APPLICATION
USING THE 50-NEURON CHIP
AS A PATTERN/CHARACTER ASSOCIATOR

Forty nine neurons of the ﬁfty neurons in the 50—neuron chip are employed to
form a 7X7 pixel array to process 2-D images. Thus, each neuron represents one
pixel. A pattern/character is represented as a 7>0 resolution image. If the output vol-
tage of a neuron is higher (lower) than some threshold value, the corresponding pixel

is considered to be one (respectively, zero).

A dedicated interface circuitry and a software environment had been built to
successfully demonstrate the use of the 50-neuron chips. A real-time application has
been performed using the 50-neuron chip with on-chip digital learning as a

pattern/character associator.

7.1. Interfacing and Software Environment

An interface circuitry and a software environment have been designed and
developed for testing and verifying the functionality and the performance of the fabri-
cated chips as well as demonstrating a real-time application using the 50-neuron chip

as a pattern/character associator.

The interface is used to control each neuron’s input and output, the weight of

each interconnect, and a few logical controls of the chip. An Intel 8255 is used as a

71

72

A0

A7

 
 

74HC373

  
 

74HC373

 
    
 

74HC373

74HC373

  
 

74HC373

  
  

    

“ll-5154 ‘ 74HC373

  

74HC373

   

748C373

 
    
 

741-10373
74HC373

  
 

BO
74HC373

  
 

Bl

  
 

32 74HC373

83 741-10373

  
 

74HC373

B4

C6 (X512)

C4 (STE)

(a)

 
   
 

Figure 7.1.1 A block diagram of the interface circuitry for the 50—neuron chip.

73

 

 

1
741.8 154
4X16
Decoder
3
C0 2:
C l 22
BS 23
l a
=- 19 II
=1-

 

 

 

 

A0

A7

 

4

 

@—=~<}—'

 

 

 

 

 

 

 

 

2 —l>°'—‘F-j{' 741.5373 |
w

 

’ —"l>°—£1,” 741.3373 |

g}

 

‘ —‘l>°'—_Ej{' 741.3373 |

 

 

 

 

 

(b)

 

w

 

WW

 

Figure 7.1.1 A block diagram of the interface circuitry for the SO-neuron chip.

74

digital I/O port to communicate between a PC AT and the interface. Three 8-bit D/A
converters (DACOSOO) are used to globally set up the gate voltages of the feedback
nMOS transistors and the gate voltages of the interconnect nMOS transistors. Several
octal latches (74HC373) were employed to build 50 digital output channels and 50
digital input channels. Figure 7.1.1 depicts the block diagram of the interface circui-
try.

Using the new architecture as a pattern/character associator. a software pro-
gram is developed using the C language on a PC AT which provides a user-friendly
environment to easily control and operate the fabricated chip. A desired
pattern/character can directly be speciﬁed from keyboard interactively or from an
external ﬁle and then the program will generate the proper signals via interface circui-
try to have the SO-neuron chip learn and store this speciﬁcally desired pattern/character
in the chip. The current network conﬁguration, the logical values of the gate voltages
of the interconnect MOSFETs, can graphically be displayed on a screen. Initial condi-
tions with various Hamming distances from the desired pattem/character can be tested
via this program. An initial input and a steady state output can be transferred to and
from the SO-neuron chip via this software environment. All experimental data and the

graphics are stored in ﬁles for later use.

7.2. Storing a Single Pattern/Character

Initially, as the power is turned on, the interface sets the gate voltages of the
feedback and the interconnect nMOS transistors to high (i.e., 5 V). In this
conﬁguration, the chip has only two steady states stored in it, namely, the all-high state
(all outputs are near 5 V) and the all-low state (all the outputs are near 0 V); all initial
conditions of the chip now converge to either the all-high of the all-low states. The
network retains these two states for all the subsequent learning conﬁgurations. We

label these two states the no-information states.

75

A pattern/character can successfully be stored into the chip by setting the
interconnect weights via hardware learning circuitry within 50 us. After a desired
character is stored, the complement of this desired character will also be stored in the
network All initial condition images with 18.4% binary distortions from the desired

character can retrieve the desired character in the order of 20 us.

In the particular experiments reported here, we supply the network with the
image of the capital-letter F. The network stores the image of the character success-
fully. We then test some initial conditions with Hamming distances less than 9 from
the character F. All such initial conditions will retrieve the stored character F. Figure
7.2.1 tabulates some of the experiments which were executed. Beginning from the top
of the tabulation in Figure 7.2.1, the ﬁrst (top) row denotes initial images which were
supplied as initial conditions to the network. Each initial image converged to the
image immediately beneath it in Figure 7.2.1. Similarly, the third row of initial
images correspondingly converged to the images on the row beneath it, and so on. It
is clear that all images converge to the image of the desired character F. Figure 7.2.1,
therefore, depicts a set of noisy variations of the character F which the network
tolerates in retrieving the character F. The network, therefore, performs association in
addition to the character recognition.

In this particular experiment, initial-condition images with more Hamming dis-
tance than 9 do not all converge to the desired character F. Figure 7.2.2 tabulates some
of the results where the initial images have Hamming distances 10 from the character
F. In Figure 7.2.2, however, an initial image which loses more than 9 (on) pixels
compared with the image of the character F converges to the all-low state. It is
interesting to note that the initial images that converged to the all-low state may not be

distinguishable by a human observer.

 

aaaaaaaaaaaagagg

 

 

@EEEﬁEEﬁEEEEEEEQ

 

EQEEEE%EEEEEEEEE

 

 

 

 

 

 

. . V
a after 1 pattern has been stored
# of matches = 50, with Hammnig distances =9

F gur . . xperimental results O-neuron c
. co .

i
nal

 

all ' 'ti ns with Hamming distances from the $ch F.
3::
El :E'l :

 

 

gEHHE1=EEHEE
%EEEE%E%EEEL§EE

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

.OOV

E a Vfb = 5. 0W = .
tte as n stored
E] = # of matches =43, with Hammnig distances = 10

 

EQEEEESEEFFEEEFQ

 

 

——

Figure 7.2.2 Experimental results of the SO-neuron chip;
an initial conditions with Hamming distances 10 from the character F.

 

 

 

77

7.3. Storing multiple patterns/characters

The chips also have the capability of learning multiple patterns sequentially.
Figure 7.3.1 tabulates some experimental results after the network stored two images
of the character F and the character T. In this particular experiment, all initial images
with Hamming distance 3 from the the character F (respectively, '1‘) converged to the
image of the character F (respectively, T). We note that the network stored not only
the desired characters but also the intersection images among the desired characters
and their complements. Under this speciﬁc digital hardware learning scheme, the max-
imum number of the intersection images increases exponentially as the number of the

desired characters increases.

 

 

 

 

 

 

 

 

EEEEEEEEEIEEEEEE

00
ms have been stored

ammnig distances = 3

 

 

 

 

 

.a Experimental results of the

all initial conditions

SO-neuron chip;

with Hamming distances 3 from the

 

character F.

 

 

 

.I.‘
II

Ill

 

$$$EI$EEIE%I%$%$*

3
V =
m w
o. m
=
w .m
.m. d
V .We
n
Mm
mm
s
VnH
mmm
5 .1
..Ww.
h
mmw
8..
Vms
em
mm
0.3
.2
Va“...
mﬁo
sans"
__
mm
Vi

 

 

 

 

 

all initial conditions

CHAPTER 8

SUMMARY AND CONCLUSIONS

To make neural computing more powerful in terms of speed, the design and
implementation of compact and electrically programmable artiﬁcial neural networks in
hardware is becoming a necessity. With the massively parallel processing capabilities,
VLSI neural circuits play an important role in the success of future high-performance,

low cost computing machines.

8.1. Summary

In this research, theory and practical design examples for a new feedback
neural network architecture with digital hardware on-chip learning scheme have been

described.

This new neural network architecture is equally motivated from biological
neural nets where neurons have dendro-dendritic connections, i.e., connections among
neurons occurring via dendrites only. The new architecture is realizable via electronic
circuits using a single MOSFET transistor for each dendro-dendritic synapse. The
maximum number of connections is reduced to approximately one half of the max-
imum connections in the architecture of the Hopﬁeld circuit. The dynamic behavior
and characteristics of the new neural network architecture is supported by mathemati-
cal analysis. The new architecture preserves its gradient dynamic properties, including
symmetry of the synaptic weights when they are implemented in the physical world of

79

80

hardware.

Neural properties of a prototype of this new architecture circuits have been
veriﬁed via extensive SPICE simulations and discrete-component experiments. The
simulation and experimental results show that by only adjusting the gate voltages of
the interconnect nMOS transistors, the network can possess all possible number of
stable equilibrium states which in turn correspond to different number of memory or
code data.

A 6—neuron Tiny-chip of the new architecture was successfully designed and
fabricated using 2 pm CMOS n-well technology with a 2.2 x 2.2 m2 chip size as a
prototype chip of the new architecture for developing and assessing various potential
off-chip learning algorithms. A specialized dynamic learning algorithm was success—
fully developed onto software environment with graphics display on a personal com-

puter and tested on the 6—neuron Tiny-chip.

Eventually a SO-neuron CMOS analog chip with on—chip digital learning
scheme was successfully designed and fabricated with a 6.8mm x 4.6mm chip size
having 63,025 transistors via standard 2pm CMOS n-well technology. A desired
equilibrium point can successfully be stored into the chip by setting the interconnect
weights via hardware learning circuitry within 50 ns . In addition, all initial conditions
converge to the steady states in the order of 20 us. From experimental measurement
using a digital meter, the power consumption of the neural chip in steady state is less
than 1 mWatt with power supplies of 0/5 volts. The neural circuit safely retains the
same dynamic properties when the power supply high level is decreased down to 3
volts to dramatically reduce the power dissipation of the analog CMOS VLSI chip.

Dedicated interface circuitries and software environments had been built to
successfully demonstrate the use of the prototype chips of the new neural circuit. A
real-time application has been performed using the SO-neuron chip with on-chip digital

learning as a coprocessor of a personal computer to demonstrate the applicability of

81

the new architecture as a pattern/character associator.

On-chip learning is very important to compensate nonideal characteristics of
VLSI neural chips and to reduce total system learning time. To the best of our
knowledge, this is the only successful and effective (analog) neural chip experiment
with guaranteed learning capability. These implementations and experimental results
have been shown to perform and substantiate the capabilities attributed to artiﬁcial

neural nets by pursuing an approach of analysis followed by direct implementations.

8.2. Conclusions

As a whole, this work lays a solid groundwork for the implementation of
ANN s in both the theoretical and practical aspects. Further studies to improve the per-
formance and to reduce the area of synapse cells and neuron cells are necessary to
integrate a larger neural network into a single VLSI chip. Future research should
emphasize advance development toward system integration for large and practical

application problems such as pattern recognition and image processing.

BIBLIOGRAPHY

[l]

[2]

[3]

[4]

[5]

[6]

[7]

BIBLIOGRAPHY

J. J. Hopﬁeld, "Neural Networks and Physical Systems with Emergent Col-
lective Computational Abilities," Proceeding of National Academy of Science
U.S.A., vol. 79. PP. 2554-2558, April 1982.

J. J. Hopﬁeld, "Neurons with Graded Response Have Collective Computa-
tional Properties like those of Two-state Neurons," Proceeding of National
Academy of Science U.S.A., vol. 81, pp. 3088- 3092, May 1984.

J. J. Hopﬁeld and D. W. Tank, "Neural Computation of Decisions Optimiza-
tion Problems," Biological Cybernetics, vol. 52, pp. 141-152, 1985.

J. J. Hopﬁeld and D. W. Tank, "collective computation with continuous vari-
ables," in Disordered Systems and Biological Organization, E. Bienenstock,
F. Fogelman, and G. Weisbush, eds., Springer-Verlag, 1985.

J. J. Hopﬁeld and D. W. Tank, "Simple Neural Optimization Networks: An
MD Converter, Signal Decision Circuit, and a Linear Programming Circuit,"
IEEE Transactions on Circuits and Systems, vol. CAS-33, no.5, pp. 533-541,
May 1986.

D. W. Tank and J. J. Hopﬁeld, " Collective Computation in Neuronlike Cir-
cuits," Scientiﬁc American, pp. 104-114, December 1987.

J. J. Hopﬁeld, "Artiﬁcial Neural NetWorks," IEEE Circuits and Devices Maga-
zine, pp. 3-10, September 1988.

82

[8]
[9]

[10]

[11]

[12]

[13]

[14]

[15]

[16]

[17]

83

R. Rosenblatt, Principle of Neurodynamics, New York, Spartan Books (1959).

R. O. Duda and P. E. Hart, Pattern Classiﬁcation and Scene Analysis, John
Wiley & Sons, New York.

T. Kohonen, Self-Organization and Associative Memory, Springer-Verlag, 2nd
ed. 1988.

L. O. Chua and L. Yang, "Cellular Neural Networks: Theory," IEEE Transac-
tions on Circuits and Systems, vol. CAS-35, no. 10, pp. 1257-1272, October
1988.

L. O. Chua and L. Yang, "Cellular Neural Networks: Applications," IEEE
Transactions on Circuits and Systems, vol. CAS-35, no. 10, pp. 1273-1290,
October 1988.

Y. Wang and F. M. A. Salam, "Experiments Using CMOS Neural Network
Chips As Pattern/Character Recognizers," 1991 IEEE International Symposium
on Circuits and Systems (ISCAS), Singapore, June 1991.

F. M. A. Salam and Y. Wang, "A Real-Time Experiment Using a SO-Neuron
CMOS analog Silicon Chip with On-Chip Digital Learning," IEEE Transac-
tions on Neural Networks, Vol. 2, no. 4, pp. 461-464, July 1991.

A. Agranat, C. Neugebauer, and A. Yariv, "A CCD Based Neural Network
Integrated Circuit with 64K Analog programmable Synapses," the proceedings
of IJCNN, 1990, pp. II-551-555.

L. J akel, et al. , "Hardware Requirements for Neural-net Optical Character
Recognition," the proceedings of UCNN, 1990, pp. II-855-861.

S. Eberhardt, T. Duong, and A. Thakoor, "Design of parallel hardware neural
network systems from custom analog VLSI ’building block’ chips", the
proceedings of UCNN, 1989, pp. 183-190.

[13]

[19]

[20]

[21]

[22]

[23]

[24]

[25]

[26]

[27]

84

W. Fisher, R. Fujimoto, and R. Smithson, "A programmable Analog Neural
Network Processor," IEEE Trans. on Neural Networks, vol. 2, No. 2, pp.
222-229, March 1991.

C. Mead, X. Arreguit, and J. Lazzaro, "Analog VLSI Model of Binaural Hear-
ing," IEEE Trans. on Neural Networks, vol. 2, No. 2, pp. 230-236, March
1991.

M. Holler, S. Tam, H. Castro, and Ronald Benson, " An Electrically Trainable
Artiﬁcial Neural Network (ETANN) with 10240 Floating Gate Synapses," the
proceedings of IJCNN, 1990, pp. II-191-196.

W. Hubbard et. al., "Electronic Neural Networks," AIP Neural Networks for
computing, pp. 227-234, 1986.,

M. A. Sivilotti et. al., "VLSI Architectures for Implementation of Neural Net-
works," AIP Conf. Proc. 151, 408, 1986.

HP. Graf and P. Vegvar, "A CMOS Implementation of a Neural Network
Model," Proc. Stanford Conf. Advanced Research in VLSI, MIT Press, pp.
351-367, 1987.

J. P. Sage, et. al., "An Artiﬁcial Neural Network Integrated Circuit Base on
MNOS/CCD Principle" AIP Neural Networks for computing, pp. 381-385,
1986.

D. E. Van Den Bout and T. K. Miller III, "A digital architecture employing
stochasticism for the simulation of Hopﬁeld neural nets," IEE Trans. on Cir-

cuit and Systems, vol. 36, no. 5, pp. 732-738, May 1989.
C. Mead, Analog VLSI and Neural Systems, Addison Wesley, 1989.

D. Psaltis and N. Farhat, "Optical information processing based on an
associative-memory model of neural nets with thresholding and feedback,"
Opt. Let. 10, 98 (1985).

[28]

[29]

[30]

[31]

[32]

[33]

[34]

[35]

[36]

[37]

85

F. M. A. Salam and Y. Wang, "A new architecture for 2-D array neural nets
with an electro-optical implementation", SPIE Vol. 977, Real Time Signal
Processing XI (1988).

F. M. A. Salam and Y. Wang, "A formulation for 2-D Array Neural nets for
Electra-Optical Implementation", 1989 IEEE International Symposium on Cir-
cuits and Systems (ISCAS), Portland, Oregon, CA, May 9-11, 1989, pp. 478-
481.

F. M. A. Salam and Y. Wang, "A Learning Algorithm for Feedback Neural
Network Chips," 1991 IEEE International Symposium on Circuits and Systems
(ISCAS), Singapore, June 1991.

F. M. A. Salam and S. Bai, "A New Feedback Neural Network with Super-
vised Learning," IEEE Transactions on Neural Networks, January 1991.

Fathi M. A. Salam, S. Bai, and J. Hou, "Dynamics of Feedback Neural Nets
with Unsupervised Learning," IEEE International Joint Conference on Neural
Networks, San Diego, California, vol. 11, June 17-21, 1990.

F. M. A. Salam and M. R. Choi, "An All-MOS Analog Feedforward Neural
Circuit With Learning", 1990 IEEE International Symposium on Circuits and
Systems (ISCAS), May 1990, pp. 2508-2511.

D. Rumelhart, G. Hinton, and G. Williams, "Learning Internal Representations
by Error Propagation," in Parallel Distributed Processing, vol. 1, eds. D.
Rumelhart and J. McCleland, MIT Press.

J. S. Denker, "Neural Network Models of Learning and Adaptation," Physica
22D, 1986. PP. 216-232.

D. O. Hebb, The Organization of Behavior, Wiley, New York, 1949.

T. Kohonen and M. Ruohonen, IEEE Transactions on Computer 022 pp. 701 ,
197 3.

[38]

[39]

[40]

[41]

[42]

[43]

[44]

[45]

[46]

[47]

[48]

86

R. P. Lippman, "An Introduction to Computing with Neural Nets, " IEEE
ASSP, pp. 4-22, April 1987.

F. M. A. Salam, "A Model of Neural circuits for Programmable VLSI Imple-
mentation of the Synaptic Weights for Feedback Neural Nets," 1989 IEEE
International Symposium on Circuits and Systems (ISCAS), Portland, Oregon,
May 1989, pp. 849-851.

F. M. A. Salam, "New Artiﬁcial Neural Models: Basic Theory and Charac-
teristics" 1990 IEEE International Symposium on Circuits and Systems

(ISCAS), New Orleans, Louisiana, May 1990, pp. 200-203.

W. S. McCulloch and Pitts, "A Logical Calculus of the Ideas Imminent in
Nervous Activity," Bulletin of Mathematical Biophysics, 5, 115-133, 1943.

R. Rosenblatt, Principle of Neurodynamics, New York, Spartan Books 1959.
B. Widrow and M. E. Hoff, "Adaptive Switching Circuit," 1960 IRE WES-
CON Conv. Record, Part 4, 96-104, August 1960.

M. A. Cohen and S. Grossberg, "Absolute Stability of Global Pattern Forma-
tion and Parallel Memory Storage by Competitive Neural Networks," IEEE
Transaction on Systems Man, and Cybernetics, Vol. SMC-13, No. 5,
September/October 1983, pp. 815-826.

S. Grossberg, The Adaptive Brain, 1!: Vision, Speech, Language, and Motor
Control, North-Holland, Amsterdam, 1987.

S. Grossberg, Eds., Neural Networks and Natural Intelligence, MIT Press,

D. E. Rumelhart, J. L. McClelland, and the PDP Research Group Eds., "Paral-
lel distributed processing-Explorations in the microstructure of cognition," vol.
1, Foundations. Cambridge, MA: MIT Press, 1986.

T. H. Bullock, R. Orkand and A. Grinnel, Introduction to Nervous Systems,

W.H. Freeman, San Francisco, 1977.

[49]

[50]

[51]

[52]

[53]

[54]

[55]

[56]

[57]

87

L. Stryer, Biochemistry, W. H. Freeman and Company, 2nd ed., 1981.

G. D. Pappas and D. P. Purpura, eds., Structure and function of Synapses,
Raven Press, NY, 1972.

S. W. Kufﬂer, J. G. Nichols and A. R. Martin, From Neuron to Brain, 2nd.
ed. 1984.

L. W. Massengill, "A Dynamic CMOS Multiplier for Analog VLSI Based on
Exponential Pulse-Decay Modulation," IEEE Journal of Solid-State Circuits,
vol. 26, no. 3, pp. 268-276, March 1991.

F. M. A. Salam, N. Khachab, M. Ismail, and Y. Wang, "An Analog MOS

Implementation of the Synaptic Weights for Artiﬁcial Neural Nets," Analog
Integrated Circuits And Signal Processing, an international journal, Kluwer

" Academic Publishers, Vol. 2, 1991.

P. W. Hollis and J. J. Paulos, "Artiﬁcial Neural Networks Using MOS Analog
Multipliers," IEEE Journal of Solid-State Circuits, vol. 25, no. 3, pp. 849-855,
June 1990.

F. Kub, K. Moon, I Mack, and F. M. Long, "Programmable Analog Vector-
Matrix Multipliers," IEEE Journal of Solid-State Circuits, vol. 25, no. 1, pp.
207-214, February 1990.

F. M. A. Salam, N. Khachab, M. Ismail, and Y. Wang, " An Analog MOS
Implementation of the Synaptic Weights for Feedback Neural Nets," 1989

IEEE International Symposium on Circuits and Systems (ISCAS), Portland,
Oregon, CA, May 9-11, 1989.

Y. Wang and F. M. A. Salam, "Design of Neural Network Systems from Cus-
tom Analog VLSI Chips," 1990 IEEE International Symposium on Circuits
and Systems (ISCAS), New Orleans, Louisiana, May 1990, pp. 240-243.

[58]

[59]

[60]

[61]

[62]

[63]

[64]

[65]

[66]

[67]

88

F. M. A. Salam, Y. Wang, and R. Y. Choi, "On The Analysis and Design of
Neural Nets," IEEE Transactions on Circuits and Systems, vol. GAS-38, no. 2,
pp. 196-201, February 1991.

C. Chin, C.-Y. Maa, and M. A. Shanblatt, "Energy Function Analysis of
Dynamic Programming Neural Networks," IEEE Transactions on Neural Net-
works, Vol. 2, no. 4, pp. 418-426, July 1991.

M. P. Kennedy and L. O. Chua, "Neural Networks for Nonlinear Program-

ming," IEEE Transactions on Circuits and Systems, vol. CAS-35, no. 5, pp.
554-562, May 1988.

M. A. Mahowald and C. Mead, "The Silicon Retina," Scientiﬁc American, pp.
76-82, May 1991. '

F. C. Hoppensteadt, an introduction to the mathematics of neurons, Cam-

bridge University Press, 1986.
C. Tomovich, MOSIS User Manual, Release 3.1.

F. M. A. Salam and Y. Wang, "Neural Circuits for Programmable Analog
MOS VLSI Implementation", Proc. of 32nd Midwest Symposium on Circuits
and Systems, Champaign, Illinois, August, 1989.

F. M. A. Salam and Y. Wang, "Some Properties of Dynamic Feedback Neural
Nets," in the session on Neural Networks and Control Systems, the 27th IEEE
Conference on Decision and Control, December 1988, pp. 337-342.

F. M. A. Salam, "Global convergence properties of a new feedback model for
mum-engineering," Memorandum No. MSU/EE/S 89/01, Department of
Electrical Engineering, Michigan State University, East Lansing, MI 48824-
1226, 25 January 1989.

F. M. A. Salam, R. Y. Choi, Y. Wang, and B. Javidi, "On The Design of

Neural Nets," 25th Annual Allerton Conference on Communication, Control,

[68]

[69]

[70]

[71]

[72]

[73]

[74]

[75]

[76]

89

and Computing, Sep. 30 - Oct. 2, 1987.

F. M. A. Salam, R. Y. Choi, Y. Wang, "An Analog MOS Implementation of
the Synaptic Weights for Feedback/Feedforward Neural Nets," Proc. of 32nd
Midwest Symposium on Circuits and Systems, Champaign, Illinois, August,
1989.

B. W. Lee, B. J. Sheu, and H. Yang, "Analog Floating-Gate Synapses for

General-Purpose VLSI Neural Computation," IEEE Transactions on Circuits
and Systems, vol. GAS-38, no. 6, pp. 654-658, June 1991.

R. Shimabukkuro et. al., "Dual-Polarity Nonvolatile MOS Analog Memory
Cell for Neural -type Circuitry," Electronic Lett., Vol. 24, No. 19, pp. 1231-
1232, 1988.

T. Borgstrom et. al., "A Neural Network Integrated Circuit Utilizing Pro-
grammable Threshold Voltage Devices," IEEE Proc. of ISCAS’89, pp. 1227-
12300, 1989.

R. Shimabukkuro et. al., "Circuitry for Artiﬁcial Neural Networks with Non-
Volatile Analog Memory," IEEE Proc. of ISCAS’89, pp. 1217-1220, 1989.

M. White et. al., "Electrical Modiﬁable Nonvolatile Synapses for Neural Net-
work," IEEE Proc. of ISCAS’89, pp. 1213-1216, 1989.

D. Schwartz et. al., "A Programmable Analog Neural Network Chip," IEEE J.
of SSC, Vol. 24, No. 2, pp. 313-319, Feb. 1989.

Y. Tsividis et. al., "Analog Circuits for Variable-Synapse Electronic Neural
Networks," Electron. Lett., Vol. 23, No. 24, pp. 1313-1314, Nov. 1987.

B. Lee, J. Lee, and B. Sheu, "VLSI Image Processors Using Analog Pro-
grammable Synapses and Neurons," The proceedings of International Joint

Conference on Neural Networks, San Diego, California, June 1990, pp. II
575-580.

[77]

[78]

[79]

[80]

[81]

[82]

[83]

[34]

[85]

90

F.M.A. Salam and S. Bai, "Complicated Dynamics of a Prototype
Continuous-Time Adaptive Control System," IEEE Trans. on Circuits and Sys-
tems, Vol. CAS-35, July 1, 1988.

Y. Wang and F. M. A. Salam, "VLSI-design and Testing of an Analog Pro-
grammable feedback neural circuit," Memorandum No. MSU/EE/S 89/09,
Department of Electrical Engineering, Michigan State University, East Lans-
ing, MI 48824-1226, 29 November 1989.

F. M. A. Salam and Y. Wang, "Simulation , experiment, and VLSI-design of
an analog programmable feedback neural, circuit," Memorandum No.
MSU/EE/S 89/02, Department of Electrical Engineering, Michigan State
University, East Lansing, MI 48824—1226, 2 February 1989.

Atul Dhablania, "Interfacing and Computer Simulations for the 6-neuron Tiny
Chip," Project Report of EE801-2, Department of Electrical Engineering,
Michigan State University, East Lansing, MI 48824-1226, December 1989.

R. Beale and T. Jackson, Neural Computing, Adam Hilger 1990.

Y. Arima et. al., " A Self-Learning Neural Network Chip with 125 Neurons
and 10K Self-Organization Synapses," IEEE Journal of Solid-State Circuits,
Vol. 26, No. 4, April, 1991.

K. Mashiko et. al., "Silicon Implementation of Self-Learning Neural Net-
works," 1991 IEEE International Symposium on Circuits and Systems
(ISCAS), Singapore, June 1991.

G. Shepherd, The Synaptic Organization of the Brain, 2nd ed., New York:
Oxford University Press, 1979.

H. Graf, et al. , "A Neural-Net Board System for Machine Vision Applica-
tions," the proceedings of UCNN, 1991, pp. I-481-486.

[86]

[87]

[88]

[89]

[90]

[91]

[92]

[93]

91

B. Boser, et al. , "An Analog Neural Network Processor and its Application to
High-Speed Character Recognition," the proceedings of UCNN, 1991, pp. 1-
415-420.

A. Agranat, C. Neugebauer, R. Nelson, and A. Yariv, "The CCD Neural Pro-
cessor: A Neural Integrated Circuit with 65,536 Programmable Analog
Synapses," IEEE Transactions on Circuits and Systems, Vol. 37, pp. 1073-
1075, 1990.

C. Neugebauer, and A. Yariv, "A Parallel Analog CCD/CMOS Neural Net-
work IC," the proceedings of UCNN, 1991, pp. 1-447-451.

H. Eguchi, et al. , "Neural Network LSI Chip with On-Chip Learning," the
proceedings of UCNN, 1991, pp. I-453-456.

F. Kub, K. Moon, and J. Modolo, "Analog Programmable Chips for Imple-
menting ANNs Using Capacitive Weight Storage," the proceedings of UCNN,
1991, pp. I-487-492.

P. Muller, et al. , "A General Purpose Analog Neural Computer," the proceed-
ings of UCNN, 1989. PP. II- 177-182. A

P. Muller, et al. , "Design and Performance of A Prototype General Purpose
Analog Neural Computer," the proceedings of IJCNN, 1991, pp. I-463-468.

C. Schneider and H. Card, "CMOS Implementation of Analog Hebbian Synap-
tic Learning Circuit," the proceedings of UCNN, 1991, pp. I-437-442.

"lllllllllll'lllll