 

 

 

 

 

 

 

 

3:...

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

vrrvi..x;lﬁry
..,¢:.11..ra {.2}
ritlrfuirl
. i
y p‘ 3...}.
:1... 2. <3, :2...

a:
.54....1153355
fir-.1594.»
. \:ra . . #2.; .Zi

. 3v.>_r;1 1:14....

.1:r,(:!1; y. r'l.:{l!alia 1 Ori...’i
a ivllxvaulll..rlh.~t{rtc

L1}. .4319 .(...O(frv!v.
.41.!1r! .1!;)\.lk ‘ , J-Ialux‘krittrlrxl‘f

pla .5553‘P11i17tf.
,7!5.$In.f..rr§.flu.1lr.
I’ rllririI/fff

- .
.2, 38...}: it: rivolrr‘.l:fo~.'f‘t I: f

Cu . ‘flftffflrxgii 1.5!!‘b‘1f1-3

. It’s) )(JIIP,rr?3\I1 A ll.

 

 

 

‘3. ¢ , 1....

y .!,.7.;fl.u (5)

 

 

THESIS

CHIGA

I IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII

3009 1 O 9996

III

 

 

 

 

 

This is to certify that the

dissertation entitled

Architecture and Statistical Model of a Pulse—Mode
Digital Multilayer Neural Network

presented by

Young-Chul Kim

has been accepted towards fulﬁllment
of the requirements for

Ph.D. degree in Electrical
Engineering

 

 

JJt/gma/M

IMajor professor

 

Date F26. Off #293

MSU is an Affirmative Action/Eq ual Opportunity Institution 0-12771

 

 

I LIBRARY
Michigan State
3 University

 

 

PLACE IN RETURN BOX to remove this ch'eckout from your record.
TO AVOID FINES return on or before date due.

DATE DUE DATE DUE DATE DUE

Iii-12.} ‘

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

MSU Is An Affirmative Action/EquaI'Opponunity Institution
c:\cltc\datedue.pm3—pi1

 

 

ARCHITECTURE AND STATISTICAL
MODEL OF A PULSE-MODE DIGITAL

MULTILAYER NEURAL NETWORK

By

Young- Chul K im

A DISSERTATION

Submitted to
Michigan State University
in partial fulﬁllment of the requirements

for the degree of

DOCTOR OF PHILOSOPHY

Department of Electrical Engineering

1993

 

 

ABSTRACT

ARCHITECTURE AND STATISTICAL
MODEL OF A PULSE—MODE DIGITAL

MULTILAYER NEURAL NETWORK

By

Young- Chul Kim

A new architecture for a pulse-mode digital neural network is presented. Algebraic
neural operations are replaced by stochastic processes using pseudo—random pulse se-
quences. Synaptic weights and neuron states are represented as probabilities and
estimated as average rates of pulse occurrences in corresponding pulse sequences. A
statistical model of error (or noise) is developed to estimate relative accuracy associ-
ated with stochastic computing in terms of a mean and a variance.

The stochastic computing model translates into simple logic gates as basic com~
puting elements leading to a high neuron-density on a chip. Furthermore, the use
of simple logic gates for neural operations, the pulse-mode signal representation, and
the modular design techniques lead to a massively parallel yet compact and ﬂexible
network architecture well-suited for VLSI implementation. Any size feed-forward net-
work can be conﬁgured using the modules. Processing speed is independent of the

network size.

 

Multilayer feed-forward networks are modeled and applied to pattern classiﬁca-
tion problems such as encoding and character recognition. The architecture and all
digital sub-components in the proposed neural network are modeled and simulated
in VHDL. Computational accuracy is analyzed and the network performance is eval-
uated in terms of a correct classiﬁcation rate. The simulation experiments in these
applications show the network performance is competitive with that of determinis-
tic DMNN simulations and ordinary back—propagation networks while retaining the

desirable properties of high speed and high density on a chip.

 

 

Copyright by

Young-Chul Kim
1993

 

 

 

 

To my parents and my wife

 

 

ACKNOWLEDGEMENTS

I would like to thank my major advisor, Dr. Michael A. Shanblatt, for his guid-
ance and encouragement throughout the years of this research.

I also want to thank all the members of my Ph.D guidance committee, Dr. P.
David Fisher, Dr. Chin-Long Wey, Dr. Moon-Jung Chung, and Dr. Jacob Plotkin,
for their valuable comments and suggestions.

Finally, I wish to dedicate this dissertation to my parents for their love, under-
standing, and support, my wife, Gyea—Sook Kim, for her love, patience, encourage-
ment, and my lovely children, Jong-Seok and So-Youn.

vi

 

TABLE OF CONTENTS

LIST OF TABLES
LIST OF FIGURES

1 . Introduction

1.1 Overview ..................................
1.2 Problem Statement ............................
1.3 Research Tasks ..............................
1.4 Organization of the Dissertation .....................

2. Background

2.1 Artiﬁcial Neural Networks ........................
2.1.1 Biological/ Artiﬁcial Neurons ...................
2.1.2 Feedback Model ..........................
2.1.3 Feedforward Model ........................
2.1.4 Recurrent Model .........................

2.2 Artiﬁcial Neural Network Implementations ...............
2.2.1 Analog and Hybrid Implementations ..............
2.2.2 Digital Implementations .....................

2.3 Pattern Recognition and Neural Networks ...............
2.3.1 Statistical Approach .......................

2.3.2 Structural Approach .......................
2.3.3 Neural Network Approach ....................
2.4 Behavioral Modeling with VHDL ....................
2.4.1 Behavioral Modeling .......................
2.4.2 VHDL Characteristics ......................

3. Stochastic Computing in Neural Networks
3.1 Introduction ................................
3.2 Generating Probability ..........................

3.2.1 Pseudo-Random Pulse Sequences ................

vii

xi

WOVAIQH

10
11
11
14
19
24
26
27
29
33
33
35
35
36
37
38

42
43
44
44

 

3.2.2 Generating Probability ............... ‘ .......
3.3 Distribution of Estimated Generating Probability ...........
3.3.1 Factorial Moment Generating Function .............
3.3.2 Binomial Distribution Model ...................
3.3.3 New Distribution Model .....................
3.4 Stochastic Computing in ANNs .....................
3.4.1 Basic Stochastic Computations .................
3.4.2 Stochastic Computing in the DMNN ..............

3.5 Back-Propagation in the DMNN .....................

. Pulse-mode Digital Multilayer Neural Networks

4.1 Basic Computing Elements ........................
4.1.1 Random Pulse Generator .....................
4.1.2 Synaptic Element .........................
4.1.3 Input Neuron Body Element ...................

4.1.4 Regular Neuron Body Element .................
4.2 Modular Architecture ...........................
4.3 DMNN Coprocessor ............................
4.4 Behavioral model of a DMNN Coprocessor ...............
4.4.1 Introduction ............................
4.4.2 Design Methodology .......................
4.4.3 Coprocessor Control in VHDL ..................
4.4.4 DMNN Model in VHDL .....................
4.5 Hardware Complexity ...........................

. Analysis of the DMNN

5.1 Statistical Models .............................
5.1.1 Synaptic Multiplication ......................
5.1.2 Two-input Logical OR ......................

5.2 Effects of Random Noise in Hidden Layers ...............
5.2.1 First Hidden Layer ........................
5.2.2 Kth Hidden Layer .........................
5.2.3 Neural Activation .........................

5.3 Network Performance Model .......................

5.4 Simulations ................................

. DMNN Application: Pattern Classiﬁcation

6.1 Introduction ................................

viii

47
48
48
49
51
53
53
55
58

63
64
64
65
66
67
69
72
73
73
74
76
78
79

81
81
82
82
88
88
90
93
95
97

102
102

 

 

6.2 Methodology ............................... 104

6.2.1 Training and Classiﬁcation .................... 105

6.3 Benchmark Problems ........................... 107

6.3.1 DMNN XOR Problem Solver .................. 107

6.3.2 DMNN Encoder .......................... 110

6.4 DMNN Character Recognizer ...................... 111

6.4.1 Data Set .............................. 111

6.4.2 Experimental Results and Network Performance ........ 111

6.5 Summary ................................. 118

7. Conclusion 119

7.1 Summary ................................. 119

7.2 Contributions ............................... 121

7.3 Future Research .............................. 122

APPENDICES 124
A Derivation of the rth Factorial Moment of a Hypergeometric Random

Variable X ................................ 124

B Program for DMNN Back-Propagation Training ............ 126

C VHDL Code and Corresponding Schematics .............. 130

D Input Data for DMNN Binary Classiﬁers ................ 143

BIBLIOGRAPHY 153

ix

 

3.1

4.1
4.2
4.3

5.1

5.2

5.3

6.1
6.2
6.3
6.4
6.5
6.6

LIST OF TABLES

Number of distinct PN sequences with the maximal length period. . . 46
Chip area required by basic digital components. ............ 80
Chip area required by DMNN elements .................. 80
Chip area required by two example networks. ............. 80

Standard deviations of 13,-(‘on’) and 13,-(‘ofi") when no 2 25, 17.1 = 5,

n2 = 5, and (a) N = 127; (b) N = 255; (c) N = 511 ........... 99
Standard deviations of 13,-(‘on’) and 13,-(‘off’) when no = 25, n1 2

10,n2 = 5, 713 = 5, and (a) N =127; (b) N = 255; (c) N = 511. . . . 100
Pm, obtained from equation 5.13 and correct classiﬁcation rates from

VHDL simulations. ............................ 101
Representation of the XOR problem in a DMNN. ........... 107
Actual outputs of an n-bit two-layer DMNN for solving XOR problem. 108
Representation of the 8-to—3 encoding problem in a DMNN. ..... 109
Average number of iterations required for training in experiment 1. . 114
Performance of the DMN N 5-digit recognizer. ............. 115
Performance of the DMNN 10-digit recognizer ............. 117

 

 

2.1
2.2
2.3
2.4
2.5
2.6
2.7
2.8
2.9
2.10
2.11
2.12
2.13

3.1

3.2
3.3

3.4

4.1

4.2
4.3
4.4
4.5

LIST OF FIGURES

A biological neuron ............................. 11
An electrical synapse ............................ 12
A simpliﬁed artiﬁcial neuron. ...................... 13
A sigmoid threshold function. ...................... 14
The Hopﬁeld network model ........................ 15
The Kennedy-Chua network model .................... 18
A simple perceptron. ........................... 19
A multilayer perceptron. ......................... 21
A Boltzmann machine consisting of visible and hidden units ...... 24
The structure of a processing element in [64] ............... 31
A typical character recognition system .................. 33
Entity declaration in VHDL ........................ 38
The ﬂow of design data in VHDL design process. ........... 41

(a) The block diagram of a LFSR. Examples of LFSRS with a maximal
length period where (b) c1, C2, . . . , c7 = 1000001 and (c) c1, c2, . . . , c7 =
0101011 ................................... 45
A random pulse generator for fractional number a: ........... 47
Duality between Boolean operations and numerical operations, where
the sampling clock period = 20 is assumed and the number of ‘1’ pulses
generated during the period in 2:0,) is 203: for a: ............ 54
Stochastic computations in the DMNN (a) synaptic multiplication; (b)

logical OR; (c) neural activation. .................... 57

(a) A maximum length 8-order LFSR where f (:c) = $2 EB :53 EB 2134 GB 233;

(b) a random pulse generator for v; .................... 64
(a) A synaptic element (SYN); (b) a block diagram of a SYN. . . . . 66
(a) An input neuron body (INB); (b) a block diagram of INB. . . . . 67
(a) A regular neuron body element (RNB); (b) a block diagram of RNB. 68
An input layer module (ILM). ...................... 69

xi

 

4.6
4.7
4.8
4.9
4.10
4.11
4.12

5.1

5.2

5.3

5.4

6.1
6.2
6.3

6.4

CI
C2
C3

A synaptic array module (SAM). .................... 70

A regular neuron array module (RNAM) ................. 70
The general architecture of the DMNN .................. 71
A DMNN coprocessor. .......................... 72
The design hierarchy of a DMNN coprocessor. ............. 75
VHDL code implementing the DMNN coprocessor. .......... 77
VHDL code implementing the DMNN. ................. 78

2-input logical OR where the dotted lines illustrate the deterministic
nature of the output sequence net;(,,) ................... 83
one“ obtained from equation 5.5 when (a) N = 127; (d) N = 255,

and one“. obtained from actual simulations when (b,c) N = 127; (e,f)

N = 255 ................................... 85
(a) K 5K 5‘ with various network conﬁgurations when net.- = 0.55, no =
36, and k > 1; (b) standard deviation of mat; with respect to net;. . . 92

The distribution of 13,- in the hidden layer (0) and the output layer (+)

for 8825 tests compared to a binomial distribution when v,- = (a) 0.45

or (b) 0.54, k = 2, no = 25, n1 = 5, and 71.; = 5. ............ 94
An example DMN N for solving the XOR problem ............ 108
An example DMNN for solving the 8-to—3 encoding problem ...... 110
For 5-digit classiﬁcation in experiment 1: (a) pixel images of a typical

data set; (b) Hamming distances between two digits. ......... 112
For IO-digit classiﬁcation in experiment 2: (a) pixel images of a typical

data set; (b) Hamming distances between two digits. ......... 113
An n-bit register with parallel load .................... 140
An n-bit magnitude comparator. .................... 141
An n-bit up-counter. ........................... 142

xii

 

 

CHAPTER 1

Introduction

 

Artiﬁcial neural networks (ANN) present a practical approach to solving computa-
tionally intensive and (or) ill-deﬁned problems such as pattern recognition, optimiza-
tion, adaptive control, associative memory, and some complex information processing
tasks. Dedicated VLSI implementation is crucial to building fast ANNs fully utilizing
the parallelism embedded in ANN computations. This dissertation presents a new
architecture for a digital feedforward neural network using stochastic computing tech-
niques. Random noise effects in this architecture are also presented. The applicability
of the network is demonstrated using pattern classiﬁcation examples. This includes
the network architecture, analysis, modeling, simulation, and applications. This chap-
ter begins with a brief overview of the ANN implementation models. The problem to
be solved is then deﬁned, followed by the research tasks. Finally, the organization of

this dissertation is outlined.

1 . 1 Overview

An artiﬁcial neural network is a highly interconnected array of simple computing
elements inspired by the computational strengths of biological neural systems. The
structure of individual nerve cells, called neurons, in biological neural systems is well
understood. The neuron is specialized to conduct electrochemical impulses from or to
sensory organs and other neurons. Its function is accomplished by means of hairlike
nerve ﬁbers. However, it is not yet well known how this neural network with its mas-
sive parallel interconnections functions as memory and manipulates complex human
behaviors. Over the last decade many researchers from the ﬁelds of physics, mathe-
matics, computer science, and engineering have provided useful theoretical analyses
for various models of ANNs [1—11]. Neural network topologies and some design pro-
cedures have been proposed and many of these ANN models have been proven to
be superior to conventional digital computers in areas such as pattern recognition,
combinatorial optimization, associative memory, and human information processing
tasks.

It is now widely believed that the massive parallelism and computational power of
the human brain results from the global and complex interconnections among a large
number of neurons rather than from the complexity of individual neurons. One of
the major goals in the ﬁeld of ANN implementation is to produce dedicated hardware
that mimics those dense interconnections among a large number of neural elements.
Most of the current ANN models, however, rely on computer simulations. With the
help of the current advancements in integrated electronics, optical, and electro—optical
technologies, dedicated hardware implementation of ANNs is now progressing [12-18].
To date, many analog and hybrid ANNs have been built using CMOS [12,15-17] or

CCD technology [14]. Most of these are analog implementations of simple feedback

 

 

 

 

 

 

or feedforward neural networks. Analog implementation offers high-speed with low
hardware cost. The primary disadvantages of analog processing are the inaccuracy of
analog computations and the low design ﬂexibility due to the physical constraints of
analog electronic devices.

Digital ANN implementation can take advantage of some of the beneﬁts of current
VLSI technology such as well-understood and advanced design techniques and tools.
Several digital neural networks based on custom VLSI design have been developed
where a neuron is a processing element consisting of computing units, registers, and a
loop—up table (or memory) [19—23]. This approach has an increased area requirement
and the level of parallelism decreases signiﬁcantly due to the communication over-
head. Recently, however, a new digital approach has been introduced to reduce the
hardware requirement and to increase the level of parallelism. In this new approach,
a synaptic multiplication and (or) a neuron activation function is implemented with
simple logic gates using stochastic computing techniques [28-32].

In this dissertation, a set of fundamental research tasks are described which are
aimed toward developing an efﬁcient architecture and statistical model of a pulse-
mode Digital Multilayer Neural Network (DMNN) based on stochastic computing. A
statistical model is developed by which the accuracy of stochastic computing in the
DMNN is analyzed. The operational characteristics and performance of the DMNN
are quantiﬁed. The applicability of the developed network is demonstrated using
benchmark comparisons and example character recognition problems. The results of
this research contribute to the establishment of a pulse-mode DMNN which has a

compact, ﬂexible, and expandable structure.

 

 

 

1 .2 Problem Statement

Many current ANN models rely on software simulations using serial or parallel
digital computers. The speed of all software simulators, even those run on parallel
machines, is far from equaling that of specialized VLSI ANNs. This is due mainly
to the sequential nature of control flow and the communication overhead in digital
computers. Some VLSI analog or hybrid ANN implementations have been built us-
ing matrices of ﬁxed or variable resistors and nonlinear ampliﬁers [12,15—17,24,25,56].
Analog implementations of ANNs have the potential for high density; however, with
current VLSI technology, it is very difﬁcult to build large (or multichip) analog ANNs.
This is mainly due to the inaccuracy of analog elements, the unavailability of reliable
permanent analog storage devices, and design parameter variations such as noise,
temperature, and high parasitic capacitances on external I/O pins. Difﬁculties in
VLSI analog implementation of AN Ns limit their density on a chip and constrained
their applications, in turn, leading to a limitation in solving real engineering prob-
lems.

A digital approach is a viable alternative alleviating some of the above drawbacks
to analog implementation. Digital implementation can take advantage of some of
the beneﬁts of current VLSI technology such as well-understood and advanced de—
sign techniques. Nevertheless, dedicated VLSI digital implementation has been less
developed because a conventional digital approach to ANN implementation has an
increased area requirement and complex connectivity. In order to build a large digital
neural network, a space-efﬁcient network architecture must be developed.

Some digital ANN architectures using stochastic computing techniques show the
possibility of the low-cost and high-speed digital ANN implementation [28-32]. In

these architectures, algebraic operations are replaced by random processes using ran-

 

 

 

 

 

dom pulse sequences. Simple logic gates combined with some other simplistic com-
ponents perform multiplications and nonlinear transformation of signals. In this
approach, the network performs pseudo-analog computations with operands ranging
from 0.0 to 1.0. An operand :c in the pulse-mode representation is the probability
of pulse occurrence in the corresponding binary pseudo-random pulse sequence 3(a)
generated at each clock. However, the overall feedforward network architecture which
is programmable and expandable to any size has not yet been established. The math-
ematical model of the pulse-mode digital neural network also must be developed to
estimate the relative accuracy of stochastic computations and to anticipate the net-
work performance. Furthermore, the applicability of the developed neural network

architecture must be veriﬁed using real—world application examples.

1 .3 Research Tasks

The tasks of this research are to (1) develop a pulse-mode digital neuron ar-
chitecture and the corresponding statistical model; (2) develop an efﬁcient DMN N
architecture and the statistical model of error (or noise) in the DMNN and analyze
the accuracy of stochastic computations utilized in the DMNN; (3) Formulate the
framework of VHDL modeling techniques for the DMNN and simulate the DMNN in
VHDL; and (4) apply pattern classiﬁcation problems to the DMNN and evaluate the
network performance and compare the performance of the DMN N classiﬁer with the
results from other deterministic feedforward neural networks.

To develop a pulse-mode digital neuron model, the ﬁrst step is to investigate var-
ious stochastic computing techniques using similarities between boolean algebra and

probability algebra. The study is concentrated on developing a digital neuron model

 

 

 

in which a non-linear transfer (sigmoid) function is embedded, which is essential to
ANN models. Also various digital neuron architectures are studied, including those
published recently. Simultaneously, all necessary components are developed in such a
way that each of them can contribute to a simple and regular neuron architecture. A
statistical model of the neuron is developed. The computing accuracy of a synaptic
multiplication and a neuron activation is estimated in terms of means and variances.
A regular neuron architecture is sought in such a way that it leads to an expandable
network architecture.

The second task is the development of a DMN N architecture and an analysis of
the network. Existing feedforward network models with advanced architectures are
explored. A ﬂexible and modular architecture for the DMNN is sought such that the
network can be programmed for different network conﬁgurations by simply connecting
basic modules. The effective network structure to minimize the correlation between
multiple pseudo—random pulse sequences is sought. The number of clock cycles per
sampling period for the pulse-code representation of signals is determined in such a
way that the required accuracy for a particular application problem is satisﬁed. A
variable register length is one of the design issues for the DMNN. This decision may
be made using knowledge gleaned from simulation results of actual problems. Simul-
taneously, network analysis is performed based on the statistical models to estimate
the differences between the results obtained by the DMNN and those obtained by the
deterministic calculation. At this stage, some assumptions are made for the analy-
sis on distributions of synaptic weights and neuron activations because they depend
highly on network architectures and application problems.

The third task involves VHDL modeling and simulation which demonstrate efﬁ~
cient behavioral modeling techniques for the DMNN. First of all, the clever use of
VHDL semantics is necessary to get a precise model. Detailed investigations are un-

dertaken on process statements, functions, and delay characteristics. A logic block

 

 

 

can be modeled using process statements and accompanying wait statements for the
ﬂow control in a VHDL description. In VHDL, a function subprogram deﬁnes an
algorithm for computing values or representing the behavior of a hardware model.
One of the useful functions in modeling digital neural networks is the bus resolution
function which deﬁnes the resolution of output values for a common output signal.
The delay model in VHDL must provide an accurate view of the timing associated
with the logic gate. In addition, an effective naming convention is considered in order
to develop VHDL models conveniently and to document them properly.

For the last task, some testbench problems and character recognition problems are
applied to the developed DMNN. This demonstrates the applicability of the DMNN.
Traditional pattern recognition systems rely on programmable algorithms based on
statistical or syntactical approaches. They perform a mapping from the observation
space to the interpretation space by extracting features from observed data and clas-
sifying the collected features into certain categories. The developed DMNN should
self-organize the complex mapping required to solve the problem and provide a fast
classiﬁcation rate. The back-propagation algorithm for the DMNN is programmed
in C and the DMNN will be modeled in VHDL. The network is trained on a host
computer. After the training, the network conﬁguration is determined and the clas-
siﬁcation of test patterns is performed by the DMNN. Testbench problems are tested
on the DMNN at ﬁrst. These problems include “exclusive OR” and “encoding” prob-
lems. The experimental results show the strength as well as the limitations of the
DMNN. The performance measures include the number of classiﬁcations per second
and the correct classiﬁcation rate. Next, character classiﬁcation problems are applied
to demonstrate its applicability to real-world problems. The experimental results are
compared with those of other approaches. For a particular problem, the proper rep-

resentation for input and output patterns, and the best choice of a register length

 

 

 

in the DMNN is determined. As a result, a design procedure for a DMNN binary

classiﬁer is proposed.

1.4 Organization of the Dissertation

The remainder of the dissertation is organized as follows. Chapter 2 contains
the background discussion of related topics. It begins with a discussion of various
artiﬁcial neural network models, and it brieﬂy describes the existing hardware im-
plementations of ANNs. This is followed by a discussion of traditional and ANN
approaches for solving pattern classiﬁcation problems. It ends with the brief discus-
sion of VHDL characteristics and behavioral modeling techniques.

Chapter 3 presents the fundamentals of stochastic computing techniques. The
techniques for generating random pulse sequences using Linear Feedback Shift Reg-
isters (LFSRs) and the randomness properties of the pulse sequences are presented.
Synaptic weights and neuron activations are represented as generating probabilities
with the pulse sequences. The statistical model of the generating probability is de—
veloped in terms of mean and variance. Then stochastic computing techniques to
perform a synaptic multiplication and a signal intergration are discussed.

Chapter 4 proposes an architecture for the DMNN. The DMNN consists of synap—
tic elements, neuron body elements, and necessary connections. To develop an overall
network architecture, modular design techniques are used. The DMNN is trained with
the Back-Propagation (BP) learning rule suitable for the pulse-mode feed-forward
neural network. A generic architecture of the DMN N coprocessor which can be at—
tached to a host computer is proposed. The DMNN coprocessor is composed of the
DMN N, a control unit, a memory unit, and some digital components. The network

conﬁguration for solving a particular problem is determined during the training ses-

 

sion. Once the training is completed, the determined synaptic weights and network
conﬁguration are loaded into the memory in the DMNN coprocessor from a host com-
puter. Then, the programmed or hardwired control unit can be used to control the
operations of the coprocessor during the classiﬁcation session.

Random noises (errors) are involved in the stochastic computations of network
operations. In Chapter 5, the statistical models of the network operations performed
using stochastic computing techniques are presented. The relationship between the
computing accuracy and the register length (or the sampling period), and the relation-
ship between the computing accuracy and the network architecture will be discovered.
The overall random noise effects on hidden and output layers are analyzed. The va-
lidity of the developed models and analysis results is justiﬁed by simulations.

In Chapter 6, the DMNN coprocessor is modeled and simulated in VHDL. Some
testbench problems and character classiﬁcation problems are applied to the coproces-
sor. A design procedure for solving binary classiﬁcation problems with the DMNN
coprocessor is proposed. Testbench problems are tested to see the applicability of the
DMNN to binary classiﬁcation problems. Network performance of DMNN character
classiﬁers is evaluated in terms of successful classiﬁcation rates. The network perfor-
mance is compared with that of deterministic DMNN simulations or other ordinary
back-propagation networks.

Finally, Chapter 7 contains the conclusions, contributions, and future direction of

this work.

CHAPTER 2

Background

 

Many artiﬁcial neural network models have been developed based on current knowl-
edge of biological neurons and with the help of available analytic methods for linear or
nonlinear dynamic systems. Network topology, computational characteristics of neu-
ron elements, and learning rules play key roles in specifying artiﬁcial neural networks.
In the ﬁrst section, feedback , feedforward, and recurrent models are discussed with
learning rules associated with the network models. In recent years, many software
and hardware implementations of these models have been developed. Among them,
some software simulators, analog, and digital electronic ANNs are discussed in the
next section, followed by related issues. Pattern classiﬁcation is one of the major
applications for feedforward ANNs. Traditional and neural network approaches used
for pattern classiﬁcation are presented. This chapter concludes with a brief discussion
of behavioral modeling and VHSIC {Very High Speed Intergrated Circuit) Hardware
Description Language (VHDL).

10

 

 

11

2.1 Artiﬁcial Neural Networks

In this section, a brief review of biological and artiﬁcial neurons is provided. Next,
typical feedback network models such as the Hopﬁeld model and the Kennedy—Chua
model are discussed in relation to ANNs. This is followed by a discussion of feedfor—

ward network models and the Boltzmann machine as a recurrent network model.

2.1.1 Biological/ Artiﬁcial Neurons

2.1.1.1 Biological Neuron

The biological nervous system consists of two principal classes of cells, the neurons
and the neuroglia. The neuroglia are cells that ﬁll the spaces between the neurons
[33]. The neuron is a fundamental processing unit of all nervous systems. Most neu-
rons contain four distinct regions which carry out the specialized functions of the cell:

the cell body, the dendrites, the axon, and the synapse (Figure 2.1).

“UCICUS axon hillock synapse
axon /
\ é ’
K W -». ‘
’ 7" -

11 b d
dendrite ce 0 y

Figure 2.1. A biological neuron.

 

 

Axons are specialized for carrying information toward other cells without reducing
the magnitude of signals. Action potentials originate at the axon hillock and travel
to synapses, from which point signals are passed to other cells. Dendrites receive
signals from sensory organs or from the axons of other neurons, convert these signals
into electrical impulses, and transmit them to the cell body. The cell body receives
signals independently. If the electrical impulses are greater than a certain threshold,
action potentials are generated and are actively conducted down the axon. The action
potentials are pulse streams with a pulse-width of about I msec.

Synapses generally pass signals to other cells in only one direction; an axon ter-
minal from a presynaptic cell sends chemical or electrical signals through a synaptic
gap. The signals are collected by a postsynaptic cell. Two types of synapses exist in
biological neural systems: electrical and chemical. They differ in both structure and
function. Cells communicating by electrical synapses are connected by gap junctions
(Figure 2.2). This allows an electrical pulse to pass from the presynaptic cell to the
postsynaptic cell. In chemical synapses, chemical substances, called neurotransmit-
ters, are involved in passing the signals [33]. An action potential is generated in the

postsynaptic cell.

presynaptic cell

/ plasma membrane

    
  

. . axon
gap junction
connection

Postsynaptic cel

 

Figure 2.2. An electrical synapse.

 

13

Two types of signals occur in synapses: excitatory and inhibitory. With an ex-
citatory synapse, the signal from the presynaptic cell causes a change in the plasma
membrane of the postsynaptic cell that tends to induce an action potential. How-
ever, with an inhibitory synapse a nerve impulse in a presynaptic neuron affects the
electrical properties of the postsynaptic membrane in such a way as to prevent the
generation of an action potential. Excitatory and inhibitory stimuli often affect a

single neuron in combination.

2. 1.1.2 Artiﬁcial Neuron

An artiﬁcial neuron can be considered as a simple processing element which sums
the weighted inputs and passes the result through a threshold or activation function.
Figure 2.3 shows this simpliﬁed neuron.

The input signals, which come from either sensors or outputs of other neurons,
form the input vector, X = (r1,---,:r,-, - - - ,xn). The weights associated with each
input form the weight vector, W.- = (wil, - - - , wij, - - - , w...) for the ith neuron, where
wij represents the connection strength between the ith and jth neurons. A threshold

function can be modeled by associating a threshold 0,- in each neuron.

  
  

Non-linear activation
function

 

Yi

 

 

 

 

 

 

 

 

 

Figure 2.3. A simpliﬁed artiﬁcial neuron.

 

 

 

14

The output of the ith neuron, y,, is then given by
Eli :f(X'I/I/i“6i) (2.1)

where f() is the threshold function. The most pervasive threshold function is the
sigmoid function because it is a bounded, monotonic, non-decreasing function that
provides a graded, nonlinear response, most resembling a biological neuron. The

sigmoid function is shown in Figure 2.4.

y=1/(1+ exp(-x))

/

‘!

 

Figure 2.4. A sigmoid threshold function.

2.1 .2 Feedback Model

Two feedback ANN models are reviewed: the Hopfield model and the Kennedy-
Chua model. In feedback neural networks, neural elements are connected to one an-
other by feedback paths from outputs to inputs of neural elements. Continuous-valued
neural elements are normally implemented as electrical circuits, and the network dy-
namics are described by differential equations. A key issue of these networks is to

deﬁne an energy function which always decreases during the dynamical evolution.

2.1.2.1 The Hopﬁeld Model

The Hopﬁeld model is a one-layer feedback network which consists of intercon-
nected nonlinear analog neurons. Many implementations have been built based on
this model. The general structure of this network is shown in Figure 2.5. In this
model, each neuron is an ampliﬁer with a capacitor C.- and a register p,- at the input
node. The output of neuron j, 1),, is connected to the input of neuron i, u,-, via a

conductance wij.

   

Weight
connecuon

Outputs

V1 V2 V3 V4

Figure 2.5. Hopfield network model.

 

16

The dynamics of an interacting system of n neurons can be described by the

nonlinear differential equation

u-
— -- w, v — -—i + 1; (2-2)
,2; J ’ R.-
where
= — 1+ wt 9
a 5: .

Pi j=l
I,- is an external input current, v,- = f,(u,-), and f,- is a sigmoid function. 12,-0.- forms
the time constant of neuron i for charging and discharging and ui/R; is the leakage

current. The energy function deﬁned by integral of equation 2.2 is

2

i=1 j

wwv. _;1... + 2‘12. f" We.) d5.- (23)

1 i=1

mIH

TI. TI

du _ _m
for Ci—(Ri — avi-

If 11);,- = 11),; for all i and j, the time derivative of the energy function is

—=—§é~ Lit) %i> (24>

Since f (u,) 18 monotonically 1ncreasing,—— d—E < 0 for all t. As a result, the value of the

energy function is strictly decreasing and becomes zero only at the equilibrium point

where%—— =—C.-dj5,* = 0 for all i.

Equations 2.2 and 2.3 deﬁne a gradient system and thus guarantee convergence.
The Hopﬁeld model has been applied to combinatorial optimization problems where it
has been observed that the network model converges to a good solution in a few time
constants [6, 10]. The objective function of the combinatorial problem is mapped

to the computational energy function through the adjustments of the connectivity

strengths My. Local minima of the energy function correspond to solutions to the

17

problem. When the Hopﬁeld network is used as an associative memory, solutions for
this network model may be memory patterns stored in the network. Approximately
0.15n memory patterns are simultaneously stored before the patterns become too

close to each other and tend to merge [4].

2.1.2.2 The Kennedy—Chua Model

A canonical circuit model with feedback was proposed for solving both linear and
nonlinear programming problems by Kennedy and Chua [35, 36]. This model uses
integrators as neuron elements. The structural parameters of the networks correspond
to the coefficients of the objective function and constraints descriptions. Figure 2.6
shows an architecture of the model, where p-cells are constraint ampliﬁers, f-cells are
integrators, and V is the node voltages v1, v2, - - - , vn. The network dynamics can be
described by

@5591 "‘ . . in
C. d, — sufgp’IgJII/IIav. (2.5)

where C; is capacitance, v,- is the voltage of node i, f (v) is the objective function,

and g(V) are constraints. The corresponding energy function is
m 9.1V)
E(V) = f(V) + Z l. mods. (2.6)
i=1

Since if- S 0 for all t, E(V) is a Lyapunov function ensuring the system convergence to
a stable equilibrium point without oscillation [36]. This model requires more hardware
to form the integrator than the Hopﬁeld model does, but it is superior to the Hopﬁeld
model in solving linear programming problems for which the Kennedy-Chua model

guarantees a stable equilibrium point while the Hopﬁeld model does not [79].

 

 

 

 

 

 

 

 

BI

 

 

 

221-2 a

 

 

 

 

 

 

 

 

 

 

 

 

-:

2;

 

 

 

 

t

2 t7? “ex-2

2; in 322%

Figure 2.6. The Kennedy-Chua network model.

2.1 .3 Feedforward Model

The Hopﬁeld and Kennedy-Chua models are examples of one-layer feedback struc-
tures. The interconnection structures of biological neurons are often organized into
multiple layers of cellsi[7, 33]. Layered feedforward networks were ﬁrst studied in de-
tail by Rosenblatt and his colleagues in the early 1960’s [42]. Since then, feedforward
multilayered structures and learning algorithms for training have been developed.
The networks are trained with a set of input—target pairs as examples and can suc-
cessfully generalize what has been learned. Feedforward networks have been applied

to pattern recognition [37, 38, 49], robotics [39], and control problems [40, 41].

2.1.3.1 Simple Perceptrons

A simple perceptron is a single layered feedforward neural network, consisting of n

inputs and an output layer. Figure 2.7 illustrates an example of a simple perceptron.

Y1 3’2 ‘ ‘ ' .Ym-l I’m

WI] ..

 

 

 

Figure 2.7. A simple perceptron.

20

:3? is the ith element of the input pattern and yf‘ is the output of neuron i when
pattern p is presented to the network. w;,- is the connection weight between neuron
i and the jth element of the input pattern. If the number of patterns is p such that
,u = 1, 2, - - - ,p, the output in the output layer can be described by

yi = wijxj +6i)

if]: II M:

where :58 = 1 for all ,u, wio = 0.- is a bias, and f () is the continuous sigmoid function.
When tf‘ is the desired output of neuron i for input pattern ,u, the cost function,

which measures the system’s performance, is deﬁned by

1” u
E 2 EEE
= ézzwyr‘t
= $23: vow-Mr. (27>

The connection weights, w;,-, are changed by the gradient descent algorithm.

8E

ang

= n for — you; waxy). (2.8)

11:1

 

Awe = -77

The condition for the existence of a solution in the simple perceptron is the linear in-
dependence of the input patterns [43]. The simple perceptron can not solve problems
in which input patterns are not linearly independent, and may offer alternate par-
tial solutions [43]. However, multilayer feedforward neural networks with nonlinear

neuron elements can overcome this limitation.

21

2.1.3.2 Multilayer Perceptrons

A multilayer perceptron (or feedforward neural network) consists of an input layer,
an output layer, and one or more hidden layers in between. Figure 2.8 shows the
generic structure of a multilayer neural network. y,- is the output of neuron i and w,,-
are connection strengths between neuron pairs. Outputs of any layer are weighted
and summed as an input to a neuron in the next layer. An external input is applied

to the input layer.

yi yn
0 0
layer

X

l

I

I
.33

I

I

I
H

3

Figure 2.8. Multilayered feed-forward neural network.

 

_._.._.____..._.._.._-.-.,__.... - . _ _ .

22
Given pattern p, where u = 1,2, ~ - - ,p, a net input netf‘ in neuron i in any layer is
netf‘ = 2 way; (2.9)
i

where y;‘ is the output of neuron j in the previous layer when pattern [1 is presented.

yo‘ = 1 is often used. Thus, neuron i produces output

it: f(net“) - f(Zw..y;-‘) (2.10)

where f () is a differentiable sigmoid function. For a given input pattern, the output

of the output layer is compared to the target pattern and the connection weights

 

between layers are modiﬁed in a backward direction according to the error. This is

known as back-propagation learning. Given pattern p, the error measure is

1

E" = 5:0? — yf‘)2 Q“)

where tf‘ and yf‘ are the desired output and actual output for the ith output neuron,

respectively, when pattern ,a is presented. The back-propagation rule states that
we“) = w.-,-(k — 1) + EMMA“ (2-12)
it

For the output-to—hidden layer connections, the gradient descent rules gives

6E“
310.5
8E“ Bnetf‘
“"W awo-
_176E“ 3y” anet"
ﬂay,” Bneti‘ (9ng
= 775.9% (2-13)

Auwidk) = -77

 

 

 

 

 

23

where 6r = (t? — yomnett).
In the hidden-to-hidden (or input) layer connections, prg, for the connection be-
tween neuron i in the hidden layer and neuron j in the lower layer can be obtained

by using the chain rule.

AuwijUC) = ”77

 

6ng
_776E“ 6y,”
33/.“ 5‘ij
3E“ anetf 8y?
— _T’;3netfcl By,” 810,-,-

 

 

where 65‘ = f,’ (net?) 2,, 6;:wk; and k denotes neurons in the upper layer.

The overall measure of the error is therefore

E = E E“. (2.15)

Thus, the back-propagation rule for any layer has the form

P 6E“

”=1 awb‘

 

Awu = ‘7]

P

u=1

Some variations of the ordinary back-propagation algorithm have been suggested
in order to help the networks learn faster or escape local minima [45-47]. Multilayer
feedforward networks trained by these back-propagation algorithms have been used

to solve pattern classiﬁcation problems [45, 48-50].

 

24

2.1.4 Recurrent Model

Recurrent networks allow connections in both directions between a pair of layers,
and within a layer to itself. The Boltzmann machine is a well-known recurrent network

with symmetric connections [51, 52].

2.1.4.1 Boltzmann Machine

The Boltzmann machine consists of visible and hidden units where the visible
units can be divided into input and output units. Figure 2.9 illustrates the structure
of the Boltzmann machine. The units are stochastic and take output value v,- = +1

with probability f(h,) and value v,- = —l with probability 1 — f(h,), where

h.- = Z wuvj
,-

and

_ 1
" 1 + e—2Bh'

f(h)

 

Output pnits
Hidden Vi§ib1e
units units

 

 

_/
’4’
. .

x“ .
Input units

 

Figure 2.9. A Boltzmann machine consisting of visible and hidden units.

25

Here ,8 = % where T is pseudo—temperature. If w;,- = w,,- for all i and j, the energy

function

H{v.-} = —% Z ngjvgvj (2.17)

has a minimum at a stable state characterized by v,- = sgn(h,-) where sgn(h,—) = +1
if h,- 2 0, otherwise sgn(h,-) = —1.
The probability of ﬁnding the system in a particular state {vi}, after equilibrium

is reached, is given by the Boltzmann-Gibbs distribution

_ {o.
P{v.~} = eﬂzrl

where Z is a normalized constant.
Boltzmann learning adjusts the connections ng such that the states of the visible
units, a, have a desired probability distribution. Let ﬂ be the states of the hidden

units. The probability Por of ﬁnding the visible units in state a irrespective of ,6 is

Pa = Zap
[3

= Ze’ﬂlﬂg (2.18)
B

where
1
Hag = —5 Z Z wijvflﬁ'US-yﬁ.
1' j

The relative entrophy between actual probability PO, and desired probabilities RC, is

R

7):. (2.19)

E: ZRalog

 

26

E _>_ 0 and E = 0 if R, = R... for all a. The gradient descent rule gives

 

Awe = ~77

 

= ”ﬁzzaapmavrﬁvfﬁ— < 5.5,- >] (2.20)
a ﬁ

where the correlations < 5.5,- > are measured by taking a time average of Sgsj and the
system must reach an equilibrium state for each a. A simulated annealing procedure is
used to rapidly achieve a global minimum. Disadvantages of the Boltzmann machines
are that learning requires an extremely long convergence time even with simulated
annealing and its hardware implementation is impractical.

Boltzmann machines have been applied to various problems: statistical pattern
recognition [7], constraint satisfaction problems [51], and combinational optimization

[53].

2.2 Artiﬁcial Neural Network Implementations

Many current ANN models rely on software simulations run on serial or parallel
digital computers. The speed of software simulation even on a parallel machine is
far from equaling that of specialized hardware AN Ns mainly because of programming
and communication overhead. To date, a number of ANN hardware prototypes have
been built using electronic, optical, and opto—electronic technologies. Electronic ANN
hardware implementations, software simulators run on digital computers, and related
issues are discussed. ANN implementations can be divided into three categories based
on the method used to express the values within the network: analog, digital, and

hybrid.

 

 

 

 

27

2.2.1 Analog and Hybrid Implementations

Analog computation performed in analog or hybrid electronic hardware uses some
fundamental physical principles such as the linear attenuation of voltage by an elec-
trical resistor and the nonlinear transfer characteristics of an ampliﬁer. In a simple
analog neural network, the interconnections are simple ﬁxed value resistors (see Figure

2.5). The output voltage of neuron i is given by
vi = f(Z w;,-v,-)
i=0

where 10,-, is the conductance of the resistor between neuron i and neuron j and f () is
the transfer function of the ampliﬁer. Neural networks with ﬁxed value resistors can
be used when the network function is known in advance and weight changes are not
needed. This type of network with 256 neurons was designed on a single chip using
standard CMOS technology by J ackel, et al. [15]. This circuit was not programmable
due to the ﬁxed synaptic weights.

AN Ns can be programmed by storing synaptic weights in memory. A static mem-
ory cell has been used as storage for a weight bit where the neurons and synapses were
binary units. Multiplication was performed by a logical XOR gate [16]. For many
applications, a higher resolution for weight values is required. One way of storing
analog weights is to use a capacitor [55, 56]. A weight can be stored as the voltage
difference between two capacitors; the voltage difference is multiplied by the input
voltage in the circuits. The main disadvantage of this dynamic storage technique is
that it requires refresh circuitry to overcome the charge leakage on the capacitance.

An alternate way is to store weights digitally. In this case, a digital-to-analog
(D / A) converter is required at each connection to perform an analog multiplication

of the stored weight with the input signal. A matrix with 1024 multiplying D / A con-

 

 

 

 

28

verters was built using CMOS technology, where a weight was represented in four-bit
magnitude plus a sign bit [57].

A ﬂoating gate ﬁeld effect transistor (F ET) was used as a device to combine the
weight storage and the multiplication, where the weight was determined by the charge
stored in the ﬂoating gate. However, the weight range and polarity difﬁculties were
signiﬁcant limitations [58]. To overcome these difﬁculties, a Gilbert multiplier [59]
was used to carry out the weight multiplication while a ﬂoating gate PET was used
simply for weight storage [60]. Sage and his associates designed an ANN chip based
on Metal Nitride Oxide Semiconductor (MNOS) ﬂoating gate transistor technology
and Charge Coupled Device (CCD) technology [14]. Analog weights were stored in
MNOS ﬂoating gate transistors. Charge packages instead of currents were added to
compute the sum of products. This circuit implemented a simple Hopﬁeld-type neu-
ral network by operating with binary inputs and analog weights.

In analog computation, available mathematical functions are limited because those
functions are found in some physical principles of devices. When a complex transfer
function is required, it is difﬁcult to implement correctly using analog hardware alone.
In this case, hybrid ANN hardware is more appropriate where the sum of products
is carried out with analog components, digitized for the transfer function processing,
and then converted back to analog [24].

The potential advantage of analog computation is that operations in the network
can be performed using inexpensive hardware. However, analog computation results
in low accuracy and limited dynamic range due to physical constraints, such as ther-
mal and quantum noise of analog components. In addition, design ﬂexibility in analog
implementation is strictly constrained because only mathematical functions resulting

from physical principles are available for use.

 

 

 

29

2.2.2 Digital Implementations

In this section, the two mainstream approaches to digital implementation of AN N s
are discussed: software simulations on general-purpose or special-purpose computers

and dedicated VLSI implementation.

2.2.2.1 Software Simulators

ANN simulations on digital computers can be divided into two categories: ANN
simulations on general-purpose parallel computers and ANN simulations on special-
purpose processors.

Many general-purpose parallel machines, consisting of a large number of process-
ing elements, are currently used for ANN simulation. Processing elements, cooperated
on the same task, communicate through a single high speed data path between pro-
cessing elements. A neural network and data are partitioned into different processing
elements. Each processing element may have a dedicated memory to store data as-
signed. For example, the Warp machine, which was a systolic array of 10 processing
elements, was used to implement a back—propagation network [61]. Each processing
element contained an adder, a multiplier, and an ALU. The 39 Mbyte cluster memory
was used to store weights and 17 million weight updates per second was achieved.
Forrest, et al. used a Distributed Array Processor (DAP) consisting of 4096 proces-
sors to implement a Hopﬁeld network [62]. The DAP was able to perform 25 million
additions per second. The use of general-purpose parallel machines for ANN sim-
ulations can be justiﬁed for the problems to be completed in a feasible amount of
simulation time. However, large-size ANNs often require faster simulations.

Special purpose processors, which are designed for ANN simulations and attached

 

30

as coprocessors to a host computer, are often called neurocomputers. A user program
run on the host computer calls a special subroutine, and controls the neurocomputer
whenever needed. Three methods for attaching a neurocomputer to a host computer
have been deﬁned [58]. The ﬁrst method is to install the neurocomputer as a memory-
mapped device on the host computer. In this method, the neurocomputer shares the
memory space of the host computer. Data transfers between the host computer and
the neurocomputer are controlled by the central processing unit in the host with ad-
dresses in the memory space. The second method is to attach the neurocomputer as
a peripheral device using a standard peripheral interface . The neurocomputer can
be ported from one type of host computer to another relatively easily. The ﬁrst and
second methods have high bus loading problems on the host computer. In addition,
the second method suffers from the reduced bandwidth of the peripheral interface.
Thus, these two approaches are appropriate for small computers. The third approach
is to attach the neurocomputer as a coprocessor to a host computer via a local area
network (LAN). This method has the advantage that the neurocomputer can access
memory servers and other outboard devices on a high—bandwidth LAN.

Several manufacturers, such as TRW, Science Applications International Corpo-
ration, and Hecht-Nielsen N eurocomputers, developed neurocomputers. For example,
Mark III and Mark IV neurocomputers were developed by TRW [63]. The Mark III
(Mark IV) machine consisted of many Motorola 68010 (68020) based single board
computers mounted on a broadcast bus backplane. These systems used the Artiﬁcial
Neural System Environment (ANSE) developed at TRW for specifying the neural
network to be implemented. A neural network was called on the Mark III (IV) from
user software on the DEC Micro VAX through an user interface. The Mark IV had
an ultra high-speed graphics display facility for monitoring the activity of the neural
network. The Mark III and Mark IV systems were able to process up to 450,000 and

5,000,000 interconnections per second, respectively.

31

2.2.2.2 Dedicated Hardware Implementations

In order to fully utilize the parallelism embedded in ANN computations, the de-
sign of dedicated VLSI ANN digital systems is desired. A three-layer feedforward
ANN was designed to classify handwritten numbers [20]. The network consisted of
50 neurons and 6688 ﬁxed interconnections using a 2—micron CMOS process. The
resulting VLSI layout was 7.9 x 9.2 mm) in size. This design is quite compact, but
its ﬂexibility was so low due to the ﬁxed synaptic weights. Suzuki and Atlas mapped
an ANN to an array of custom processors [64, 65]. Figure 2.10 shows the structure
of the proposed processing element, where the blocks represent special operations for
the network update. A weight matrix W and a threshold vector 0 are stored in the

product—sum unit (PSU). The arithmetic unit (AU) performs operations required for

back-propagation.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

m LL] 5 ,
PSU
I DF DO I
NI“ AU "

 

 

 

 

 

 

 

 

 

 

 

A
‘

 

 

Figure 2.10. The structure of a processing element in [64].

 

 

 

32

The derivative of a nonlinear function (DF), desired outputs (DO), and a learning
rate (77) are stored in the memory of the AU. Neural activations (X) and the error
value (6) are accessed by both the PSU and the AU. This ANN hardware has a high
design ﬂexibility, but hardware requirements for this design are large.

As indicated in the above two examples, dedicated digital ANN implementations
can facilitate high parallelism, but it is difﬁcult to simultaneously achieve the desired
high design ﬂexibility and high density on a chip.

A new digital approach - digital ANNs using stochastic computing techniques -
replaces algebraic operations in ANNs by stochastic processes using pseuddrandom
pulse sequences [28, 31, 32]. Simple logic gates combined with other digital compo-
nents perform multiplications and nonlinear transformation of signals.

In this new approach, the values for synaptic weights and input operands are
normalized after a network has been trained [28, 32] or all operands are restricted
to the range between 0.0 and 1.0 both for training and testing [66]. An operand :1:
in the pulse-mode representation is the probability of pulse occurrence in the corre-
sponding binary sequence :r(,,) at each clock. 5: is the estimate of .1: taken over ﬁnite
clock periods N. Stochastic computations using random pulse sequences inherently
utilize concurrent processing in all synaptic and neuron elements. Furthermore, the
use of simple logic gates as computing elements allows a high neuron-density on a
chip and a relatively compact network architecture. High design ﬂexibility can also be
achieved by making the network programmable. However, network speed depends on
the length of a sampling clock period. The sampling clock period is the time required
to estimate the computation results. A longer sampling clock period yields more ac-
curate computations. Thus, there exists a trade-off between speed and accuracy in
this approach. Details on the network architecture, analysis, and performance will be

discussed in following chapters.

 

:33

2.3 Pattern Recognition and Neural Networks

Pattern recognition is concerned with classification or description of complex pat-
terns by means of some measured prOperties. A pattern recognition system requires
data acquisition, data representation, and data classification.

The design of a pattern recognition system involves the following three steps: (1)
data acquisition, (2) preprocessing, and (3) decision making [68]. A typical character

recognition system is illustrated in Figure 2.11.

 

 

 

Digitized . Size normalization, Matching Identity of
character matnx Noise cleaning character
i. Pre- __i_> Feature __i__> Decision _I_>
processor extractor maker

 

 

 

 

 

 

 

 

 

Figure 2.1 1. A typical character recognition system.

The ﬁrst stage involves image processing, the last two stages deal with the pattern
recognition. Mask (joint occurrences of black and white pixels), strokes and bays in
various directions, the location of end points, and the intersection of line segments
and loops, are all popular features for character recognition. Most pattern recognition
systems utilize one of the following three approaches: statistical, structural, or neural

network.

2.3.1 Statistical Approach

In the statistical approach, a pattern is represented in terms of N features. Each

34

pattern can be viewed as a point in the N-dimensional space. If the choice of features
is good, then pattern vectors belonging to diﬂerent classes will occupy different regions
of this feature space. The objective in this approach is to establish decision boundaries
in the feature space to separate patterns belonging to different classes.

Assume that a given sample pattern belongs to one of M classes c1,c2, - . - ,cM
based on its feature vector x = ($1,232, - - - ,xN) and that x has a class-conditional
density p(x|c,-). Bayes decision rule states that a pattern with x as its feature vector

is assigned to class c,- if

p(CIIX) Z p(CjIX) for all i7“
where p(c,-|x) is the posteriori density for class q, deﬁned as

P(XIC:‘)P(C£)
iii P(XIC:‘)P(C£)

 

P(C:'IX) = E

where p(c,-) is a priori probability density for class c,-. If p(c,-) = l/M, then the
Bayes decision rule is identical to the maximum-likelihood decision rule. The decision

boundary between pattern class c.- and c,- is deﬁned by
P(CiIx) - P(CjIX) = 0-
If class-conditional densities are multivariate Gaussian, then
P(XIC:') = NW, 1),

where p,- is the mean vector for class c; and I denotes the identity covariance matrix.

35

If p(c,-) for all i are equal, then

IX - it-II2

p(c.-|x) = —'———,—*-.
where [I - M denotes the Euclidean norm. As a result, a pattern x is assigned to the
class of the closest mean vector. If the class-conditional densities are known, Bayes
decision rule can be used to design a classiﬁer. If they are not known, they must be

estimated by training with sample patterns.

2.3.2 Structural Approach

When the number of features required to establish a reasonable decision boundary
is very large, it is more appropriate to View a pattern as being composed of simple sub-
patterns. In the structural approach, a complex pattern is represented in terms of the
interrelationships among the simplest subpatterns, called primitives. This paradigm
has been used in situations where the patterns have a deﬁnite structure which can be
captured in terms of a set of rules.

The primitives or grammatical rules must be inferred from the available samples.
In this approach, the difﬁculty resides in segmentation or reliable extraction of the

primitives from a finite number of pattern samples.

2.3.3 Neural Network Approach

The neural network approach is based on the notion that a network of simple

processing elements arranged in a manner similar to a biological neural system might

36

be able to self-organize itself to recognize and classify patterns. The Perceptron is
considered as the ﬁrst signiﬁcant development of such in the early 19603 [2]. The basis
for the inherent power of Perceptron devices was well understood. However, at that
time, no method was known for training multilayer Perceptron devices and the cost
for full implementation of those devices was extremely high. VLSI technology has
advanced and the price of processors has dropped tremendously. More signiﬁcantly,
the generalized delta rule developed in 1986 by Rumelhart, et al. provides a practical
way for training the multilayer Perceptrons [8]. Today, perceptron-like models trained
by the generalized delta rule are being applied to pattern recognition.

In pattern recognition systems using the neural network approach, all stages or
some of stages in Figure 2.11 can be combined into one neural network. The net-
work learns the mapping from the observation space to the interpretation space by
a training algorithm. In this approach, human interactions involved in statistical or
structural pattern recognition systems are minimized. Most recognition processes are

performed in an autonomous manner.

2.4 Behavioral Modeling with VHDL

In the design of large systems like ANN 3, use of Design Automation (DA) becomes
necessary. The simulation and veriﬁcation of a design using a behavioral description
language at an early stage of the design process also becomes more important as the
complexity of systems continues to grow. VHDL is a typical behavioral description
language which is semantically oriented for digital systems. Digital ANNs can be

modeled and simulated using VHDL.

37

2.4.1 Behavioral Modeling

A promising approach for implementing artiﬁcial neural networks is the fabrica—
tion of special-purpose VLSI chips. Traditionally designers start with a gate—level or
a circuit-level schematic. However, as systems become more complex, a top-down
design approach is needed in order to manage complexity and to reduce the design
time and development costs. Test and modiﬁcation of an original design can be done
in an early stage of the design process. T op-down design starts with a high-level spec-
iﬁcation which is decomposed into lower level speciﬁcations in a hierarchical fashion.
Designers look at the system at an abstract level in a high—level speciﬁcation. Hard-
ware Description Languages (HDLs) are crucial to the high-level design [69-72].

VHDL is a typical HDL that can be used to express the function and logical
organization of circuits, ranging from simple logic gates to complex digital systems
[73-77]. VHDL is fast becoming an industry standard. The US. government made it
a standard language, requiring the use of VHDL as the design and description mech-
anism in Department of Defense (DoD) hardware designs. Compilers, translating the
structural design in VHDL to an intermediate format such as Caltech Intermediate
Format (CIF), are being produced by many CAD vendors.

In VHDL one can model the behavior of systems and simulate them to verify
the design. Modeling involves specifying the inputs and outputs of a device, and
describing its behavior and/or structure. For example, when an ANN is modeled in
VHDL, its behavior may be described by a set of static or dynamic equations by
using function statements. Structure is described by interconnections of the subcom-
ponents (synapses and neurons). An efﬁcient and precise modeling of VLSI ANNs
is facilitated by analysis of VHDL semantics, including a detailed investigation of

process statements, functions, and delay characteristics.

38

2.4.2 VHDL Characteristics

The primary element in VHDL is a design entity which can represent portions of a
hardware design ranging from simple logic gates to complex digital systems. A design
entity consists of two different types of descriptions: the entity declaration and one
or more architectural bodies. The entity declaration defines the interface between the

entity and the outside world. Figure 2.12 illustrates an example entity declaration.

 

entity [COUNTER is . ,

« generic .(timegdelayz‘time: 10 ns);
p. port (elk, reset: in it;
' sum: bufferinteger);

end COUNTER

 

 

 

Figure 2.12. Entity declaration in VHDL.

The ports are the signals through which the design entity communicates with other
modules. Their declaration can be any predeﬁned or user-deﬁned type. The port and
local item deﬁned in the entity declaration are made available to architectural bodies
associated with this entity. A set of parameters, called generics, provides a channel
for static information to be communicated to a design entity from its environment.
Generics can be used to specify timing characteristics, the bit size of ports, or other
descriptive characteristics of a design such as temperature, capacitance, location, etc.

An architecture body supports three implementation styles of a design entity: be—
havioral, structural, and data-ﬂow. The behavioral body describes the system model

in sequential program statements just like programs written in a high-level program-

 

39

ming language. The structural body describes a design entity purely in terms of its
subcomponents and their interconnections. Finally, the data-ﬂow body decomposes
the architecture into a set of concurrent register assignments under the control of
gating signals. Data-ﬂow style emphasizes the ﬂow of information between memory
and gating elements. All three styles may be intermixed in an architectural body.

A VHDL design entity is a template to be used in creating speciﬁc instances of a
component via the component instantiation statement. A component may represent
a structural partitioning of the design or a functional decomposition of a large system.
Because this feature essentially isolates one level of design from another, two differ-
ent design methodologies can be accommodated: top-down approach and bottom-up
approach. In the former approach, the architectural body can be written in terms
of abstract lower—level components. Such components must be fully described with a
variety of design entities later in the code. In the latter approach, the local compo-
nent declaration speciﬁes the portion of the interface from an existing design entity
that resides in the design library.

Designers may specify the behavior of a subsystem and leave the implementation
details of structural design to others. Thus, VHDL designers can model simply the
function of the system independent of any implementation technology.

A VHDL description is evaluated when an event occurs at one of the component’s
inputs. The evaluation yields a new set of projected values for the outputs of the com-
ponent. This effect may, in turn, causes additional changes. Independent sequences
of events can occur simultaneously. The event-driven semantics of VHDL are based
on the assumptions that all signals in a design propagate in well-deﬁned directions
and that signal propagation always includes a delay.

A typical signal assignment statement consists of a driver and a target. A driver
is a source of the value for a signal. A signal may have multiple sources. If a signal

has more than one source, then all sources can participate in the calculation of the

 

40

value. Such a signal must be a resolved signal, and the resolution function calculates
one effective value from an array of values. The target of a signal assignment is the
signal on the left hand side of the assignment operator. The simulator creates a driver
for each element of a target of every concurrent signal assignment [74].

Timing is one of the most important aspects of a VHDL model. The representation
of time in VHDL has both a macrotime scale and a microtime scale. The macrotime
scale represents real time (nanoseconds, microseconds, etc.) which is measured in
discrete units. The microtime scale represents a unit delay which is essentially not
measurable. Any number of micro-units of time may exist between any two macro-
units of time. With two time scales, designers can perform unit-delay or real-time
simulations [73].

There are two kinds of statements in VHDL: sequential and concurrent. Sequen-
tial statements are used to deﬁne algorithms for the execution of a subprogram or
process. They are executed one at a time. Concurrent statements are executed in an
asynchronous pseudo-parallel fashion. They are used to deﬁne interconnected blocks
and processes that jointly describe the overall behavior or structure of a design.

Figure 2.13 shows the ﬂow of data in the design process under a VHDL hardware
support environment including an analyzer, a proﬁler, and a simulator. The design
library contains intermediate representations of VHDL descriptions. The library unit
resulting from the analysis of a design unit is placed into a working library. Only one
library may be the working library during the analysis of any given design unit [74].

The analyzer accepts a VHDL source code, translates it into the intermediate
form, and stores it in the design library. It checks the syntax and semantic rules
of the language. The proﬁler pulls all necessary design entity interfaces, bodies,
functions, and packages from the library, then conﬁgures a cross-section of a design
hierarchy. The simulator and other tools may use this conﬁguration. The simulator

records signal histories and dynamic errors.

An understanding of VHDL semantics and characteristics enables designers to use
VHDL as an economical hardware design testbench. A system can be ﬁrst modeled

behaviorally with a high-level speciﬁcation using appropriate modeling techniques,

verifying the correctness of the design.

decomposed into lower level speciﬁcations, incorporating more implementing tech-
nological constraints. Finally, when the system is modeled in complete structural

descriptions, the precise feasibility and detail of a hardware realization can be as-

 

 

sessed.
Other
Tools
A
VHDL Inter-
Analy- mediate '
zer + form

 

 

 

     

Proﬁler
control

  

 

 

 

Proﬁler

41

Later, the high-level speciﬁcation can be

->'

 

 

 

 

 

Figure 2.13. The flow of design data in VHDL design process.

 

 

Inter-
mediate
form

 
 

 

Simulation
control
and data

 

 

I

 

->

 

Simu-
lator

 

 

 

 

Result

CHAPTER 3

Stochastic Computing in Neural

Networks

 

An approach to performing arithmetic operations using random pulse sequences is
discussed. In this approach, a number is normalized into a fraction from 0 to I. The
fractional number is encoded using a random pulse stream where it is represented by
the probability of a pulse occurrence in each clock period. Algebraic operations are re-
placed by stochastic processes, and computational results expressed as probabilities are
estimated in ﬁnite clock periods. Inaccuracies are inherently associated with stochastic
computing and can be described in terms of mean and variance. In this chapter, the
method for generating random pulse streams is discussed and a new statistical model
for the estimate of probability generated from a random pulse generator is developed.
Stochastic computing techniques, which can be utilized in digital artiﬁcial neural net-

works, are presented.

42

43

3. 1 Introduction

Von Neuman ﬁrst observed that normalized numbers or voltages could be rep-
resented by probabilities and that some properties of the nervous system could be
explained through statistics [80]. He intended to show that simple algebraic opera-
tions such as addition and multiplication could be performed by simple logical gates.
Later, stochastic computing techniques using random pulse streams were proposed in
the 1960’s [81, 82].

In stochastic computation, the operands are normalized and represented by proba-
bilities which are actually encoded in random pulse streams. Probability is estimated
as a relative frequency of ‘1’ pulse occurrences in a ﬁnite but long pulse stream. Since
the probability can not be measured exactly, errors by estimation are introduced in
the form of variance when the stochastic computing techniques are used. At the time
it was originally proposed in the 1960’s, integration technology was not mature and
the hardware cost for arithmetic devices was expensive. A main objective in using
stochastic computing techniques was to implement some algebraic computations by
inexpensive large parallel processors at the cost of speed and accuracy. Since then,
the hardware cost of digital computing elements has continued to drop as VLSI tech-
nology has advanced tremendously. Consequently, the idea of stochastic computing
had been discarded.

However, the idea has been resurgent as an alternative to deterministic computa-
tions in the area of artiﬁcial neural networks since late 1980’s. The main reason is that
stochastic computing using random pulse sequences shares one very important char-
acteristic with ANN dynamics: network performance depends not on the accuracy
of calculations performed in an individual processing element, but on the collective

properties of the network (or system) where each processing element does not nec—

44

essarily perform correct computations. Recently, some neural network architectures
have been proposed based on this idea and applied to some engineering problems such

as associative memory [28] or binary classiﬁcation [32].

3.2 Generating Probability

3.2.1 Pseudo-Random Pulse Sequences

A pseudo-random pulse (or binary) sequence can be generated by a tapped Linear
Feedback Shift Register (LFSR) [67]. Figure 3.1 shows the diagram of an n-bit LFSR.

The feedback function f(iltl, x2, - - - ,xn) is expressed in the form
f(x11329”°ixn)= C1331 $62372 Q " ' $611171:

where each constant c,~ is either 1 or 0, the symbol EB denotes modulo-2 adddition,
and 9:1 and mu indicate the values of the most signiﬁcant and least signiﬁcant bits,
respectively. For a given register length n, the maximal length period of a sequence
is pmaJ: = 2" — 1.

Deﬁne {an} be a PN sequence if and only if it is a binary sequence satisfying a

linear recurrence

a;c = E ciakn (modulo 2)

i=1
and has pm” as a period. There are 2" combinations to select eg’s. Only a limited
number of c,- combinations can form the maximal length PN sequences. In order to
form a maximal length PN sequence, c,- is determined by the primitive polynomial

[67].

45

x1 x2 X3 ' ' ' xn-l xn

 

, MSB - - - LSB

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

(a)
’ MSB LSB
u\
(b)
,. MSB LSB
——<:’
H—
\\\
(C)

Figure 3.1. (a) The block diagram of Linear Feedback Shift Register (LFSR); examples

of LFSRs with a maximal length period where (b) {c1,02, . . ., c7} = { 1000001 } and (c)
[c]. (:2, . . ., c7} = {0101011}.

46

Table 3.1. Number of distinct PN sequences with the maximal length period.

 

InIPmazI anII 11I PmaxIanI

 

 

1 1 1 8 255 16
2 3 1 9 511 48
3 7 2 10 1023 60
4 15 2 11 2047 176
5 31 6 12 4095 144
6 63 6 13 8191 630
7 127 18 14 16383 756

 

 

 

 

 

 

 

 

 

Table 3.1 shows the number of distinct PN sequences, an, with period p = pm”
with respect to the LFSR register length. For a given register length n, a sequence
{an} has the following random properties assuming that each 0 and 1 is replaced by

I and -1, respectively [67]:

1. The number of 1’s is nearly equal to the number of -l’s in a maximal period

pm”. More precisely,

Pma:

[207.131-

71:1

2. Every possible array of n consecutive terms occurs exactly once, except all 0’s.
This indicates that all n-bit integer numbers from 1 to 2" -— 1 are generated

exactly once in pm”.

3. The autocorrelation of an is

 

Pma: I If T = 0
cm = 1 Z) anam =
pm“: 11:1 —1/pmaz If 0 < T < pmax-

These random properties of an are utilized for encoding fractional numbers into cor-

responding random pulse streams.

 

47

3.2.2 Generating Probability

In order to utilize stochastic computing techniques, the values of all operands must
lie between 0 and 1. The fractional numbers represented by probabilities are encoded

in random pulse streams. If the fractional number is stored in an n-bit register, the

 

resolution is 2,34. For example, when n = 8, 0.0 is stored as ‘00000000’, 1/‘255 as
‘00000001’, 2/255 as ‘00000010’, etc. The random pulse stream corresponding to a
fractional number can be generated by comparing the number with a pseudo random
number. The pseudo random numbers can be generated from a PN sequence by tak-
ing all bits of the LFSR in parallel, as indicated in property 2 of the PN sequence.
Fractional numbers from 571:]— to 1 equally spaced by 2,,1—_1 are generated exactly once
in a period 2” — I. The distribution of the pseudo random number is close to an
ideal uniform distribution. Figure 3.2 shows the diagram of a random pulse generator
(RPG) for a fractional number x.

Generating probability a: is deﬁned as the probability of pulse occurrence in the
corresponding random pulse sequence f(n) at each clock. a: is estimated in a sampling
clock period, where the sampling clock period is deﬁned as ﬁnite clock periods taken
for estimation of 3:. Error (or noise) is involved in estimating the generating proba—

bility in ﬁnite clock periods. Thus, estimate 5: can be modeled as an original signal

plus random noise.

x(n)

clk __ I—I Us...

Digital t
LFSR comparator

 

 

 

 

 

 

 

 

 

 

Figure 3.2. A random pulse generator for fractional number x.

48
3.3 Distribution of Estimated Generating Prob-

ability

The generating probability of the random pulse generator has been modeled as a
binomial distribution in the literature [82, 83]. However, the distribution can be more
precisely modeled by regarding the estimate as a hypergeometric random variable.
This new model of the estimate is important, especially for case when a short sampling

clock period is taken for estimation.

3.3.1 Factorial Moment Generating Function

The following terms are deﬁned:
2:: A fractional number or a generating probability.
$0,): The pseudo—random pulse sequence for :r.

N: The period of $(n) such that N = 2" — 1, where n is the order of a maximum

length LFSR.

P3: The sampling clock period for estimation.

X: A capital :1: is a discrete random variable indicating the number of logic level ‘1’

pulses occurring in 33(71), where 0 S X g P, such that X = 0,1,2,. . . ,P,.
E(X): The expected value of X.

Var(X): The variance of X.

It: The estimate of :1: and a random variable such that i: = X / P,

49
The factorial moment generating function of the distribution of a random variable X
is formally deﬁned as follows [84]:

g(t) = E(tX) = / °° e fx(a:)d:1:. (3.1)

Differentiating n(t) k times and substituting 1 for t gives

Me) = iii-EM.-.
dt" —

= E[X(X—1)~-(X—k+1)], (3.2)

where E[X(X — 1) - - - (X — k + 1)] are called the factorial moments.

The variance of X can be computed from the ﬁrst two factorial mements as

Var(X) = E[X(X — 1)] + E(X) — [E(X)]2. (3.3)

3.3.2 Binomial Distribution Model

If the occurrence of successive pulses in a sequence 3(a) is statistically independent,
the sequence is called a Bernoulli sequence. The random variable X of interest is the
number of logic level 1’s occurring in a sampling clock period P,. X is a Bernoulli
variable. Consider X = k indicating k logic level 1’s occur in P, clock periods.

Let the probability of pulse occurrence at each clock period in 3(a) be :1: = p. The

probability of one particular sequence with k logic level 1’s in n clock periods is

p"(1 - 19)“-

50

The number of sequences with k logic level 1’s in 71. clock periods is the same as the

number of ways of taking k objects at a time from n objects. The number is

n!

(2) : k!(n — k)!'

The quantity (2) is called the binomial coefficient.

Thus, the probability function of X can be expressed by

P(X = IC) = (2)1931 -p)""‘. k = 0.1,..-

The density function of X is

n.

, n. (3.4)

fx(=v) = 2(2):)"(1- 19)""‘V<5(=r - k)-

k=0

The distribution of X is binomial.

The factorial moment generating function of a binomial distribution is obtained

using the binomial theorem as follows:

770) = E(tx)

= 2n: t"(’13)p"(1 — P)"""
k=0

= :09th — p)""‘

k=0

= [Pt + (1 —P)l"-

(3.5)

The mean and variance of X can be computed using the ﬁrst two factorial moments,

"‘(1) and 17”(1) as

E(X) = tip and Var(X) = np(1 — p). (3.6)

 

51

Thus, the mean and variance of i: are, respectively

E(iz) = p and Var(ir) = M. (3.7)

n

3.3.3 New Distribution Model

A pseudo-random pulse sequence, 1301), has been modeled in the literature as a
Bernoulli sequence [82, 83]. However, the pulse occurrence in 3(a) is not perfectly
independent because a maximum length LF SR generates fractional numbers between
5},- and I such that each number occurs exactly once in a period. Accordingly, the
pulse occurrence in :1:(,,) has statistical dependency.

If P, = n and x = p = l /N , the probability that k ‘1’ pulses in :1:(,,) occur during
the sampling clock period It is the same as the probability that k black balls are
taken out in n withdrawals from the box containing l black balls and N — l white
balls, one ball being withdrawn at a time without replacement. Thus, the sampling

distribution of X can be more closely modeled by the hypergeometric distribution.

When a: = p = UN and P, _—_ n, the probability function of X is

PM = k, = (up?)

where l is a natural integer, i.e., l E {0,1,2,---,N —1,N}.
Let (k), be the product of r consecutive integers starting with k. Then, the rth

factorial moment is

N
E[(X).] = E(k)rP(X=k)

Ic=r

if“), (1) 2:1)

kzr if)

 

The detailed derivation of equation 3.8 can be found in Appendix A.

The expected value of X is

E(X) = E((X).]|.=1

 

_ ’13
_ N
and for r = 2 in equation 3.8,
ll—lnn—l
E[X(X —1)]= ( N(])V (_ 1) I

From equations 3.9 and 3.10, the variance of X can be computed as

Var(X) = E(X?) — [E(X)]:

= E[X(X —1)]+ E(X ) — [E(X >12

 

 

 

_ l(l—l)n(n—1)+l_n_(_ln)
" N(N—l) N N
__ niN—lN—n
— N N N—l

N—n

Thus, the expected value and variance of :i: are, respectively,

 

E(e) =

”U :[r—a
Zli

(3.8)

(3.9)

(3.10)

(3.11)

(3.12)

53

and

 

Var(si) = $Var(X)
_ 10(1 -P) N --n
— n N _1 (3.13)

 

where 0 S it: 5 1 when l _<_ n S N. As noted from equation 3.13, when N is large
(implying a wide LFSR) the distribution of :3: tends to the binomial distribution. If
P, = N = n, Var(i) = 0 and :1: becomes a constant :13.

This new statistical model is used to perform an analysis on random noise ef-

fects in digital multilayer neural networks (DMNN). The DMNN architecture will be

developed in Chapter 4 and the analysis will be performed in Chapter 5.

3.4 Stochastic Computing in ANNs

Stochastic computing exploits the similarities between probability algebra and
Boolean algebra. Logical operations with simple logical gates over multiple pulse

sequences correspond to pseudo—analog computations.

3.4.1 Basic Stochastic Computations

A random pulse sequence (E(n) is a sequence of pulses whose probability :1: can not
be measured at any one clock period, but it can be approximated by a measurement
of average pulse rate. Any Boolean operation over individual pulses corresponds to
an algebraic operation among variables represented by their respective average pulse
rates [82]. Figure 3.3 shows the duality between logical operations with actual pulse

occurrences and numerical operations with pulse occurrence probabilities.

 

 

“(n)

 

 

V(n)

 

“(ID

 

”(n) _.

 

WIn) = “(n1 AND VIn)
W = u V.

2(n) = “(11) OR V(n)
z = u+v-uv.

It-(n) = NOT um)
§=I-u.

“=05jI—I fl TI Hf] I__II—L

v=0.4[ n r1 I‘l III—‘l

w=02I n r1 [‘1

 

2:0-7I1flljl II I

I
t

Z=0-5[rII—I I—mI—I II

Figure 3.3. Dualtity between Boolean operations and numerical operations, where the samp-
ling clock period = 20 is assumed and the number of ‘1’ pulses generated during the period in
x(,,) is 20 x for x.

If two sequences :1:(,,) and g(n) are statistically independent, the probability of pulse

occurrence in an output sequence z(,,) of an AND gate is

and the probability of pulse occurrence in an output

2 = P(z(,,)

=1)

P($(n) = I /\ y(,,) = I)
P(.’E(n) = 1)P(y(,,) = I)

333!

(3.14)

sequence z(,,) of an OR gate is

 

55

z = P(z(n)=1)
= P(.’E(n) =1 V g(n) =1)
= P($(n) =1)+P(y(n) =1)“ P($(n) =1 A y(n)=1)

= :r+y—:1:y. (3.15)

Instead of being statistically independent, if two sequences are mutually exclu-
sive, implying that no two pulses coincide in two random pulse sequences, P(a:(,,) =
1 A y(n) = 1) = my = 0 in equation 3.15. Thus the logical OR performs a direct
summation.

The NOT gate in Figure 3.3 (c) produces an output pulse whenever no input
pulse occurs. If 1%,) is an input pulse sequence of a NOT gate, the probability of

pulse occurrence in an output sequence z(,,) is

z = P(z(n)=1)
= I - P($(n) = I)

= 1— 1:. (3.16)

A complete set of examples of stochastic computations utilizing the duality be-

tween Boolean operations and algebraic operations can be found in reference [82].

3.4.2 Stochastic Computing in the DMNN

Neural operations in a stochastic neural network of the type considered here are

performed with basic gates using pulse sequences as inputs. Let w;, and v, be the

56

connection weight between neurons i and j and the. neural activation of neuron j,
respectively. If two sequences mm”) and vj(,,) are statistically independent, the prob-

ability of pulse occurrence in an output sequence mm”) of an AND gate is

mu : P(m.-,-(..) =1)
= P(w.~,-(..) = 1 A We) =1)

= wijvj. (317)

Input summation and nonlinear transformation can be performed simultaneously
using logical OR operation. The inputs of an OR gate are product sequences, mm”),
produced from AND gates. Two kinds of synaptic weights 10,1",- and wg- are necessary,
positive (or excitatory) and negative (or inhibitory) for most feedforward neural net-
works. Thus, two separate OR gates per neuron are needed to form excitatory and
inhibitory net inputs. Let net;I be the probability of a pulse occurrence in the output
sequence netﬁn) of an n-input OR gate for an excitatory net input in neuron i and

net: likewise for an inhibitory net input (See Figure 3.4). net?” and net," can be

described by

net?” = P(net$n)=1)

I

= 1— (1 - P(mii(n)=1))(1— P(mii(n) = 1)) ' ' ' (1 — P(m:+n(n) =1»

=1—II(1-m;-§-)

3:1

= 1— ﬁn — 7.0333,) (3.13)

 

57

D— .

Wi1<n> —

m. A
mij(n) ’1‘“) [II] II >
VIIn) '—

 

 

 

 

 

 

(a)
mi1(n) net: E
I s ’ mm “m II II II ,
minal) . I t
(b)
net- + .—
-133, 4:)— 11...]
— ll 111111
| >
(c) ’

Figure 3.4. Stochastic computations in the DMNN (a) synaptic multiplication; (b) logical
OR; (c) neural activation.

and

net,’ 2 1— H(1+ wfjvj). (3.19)

3:1

Two net inputs, formed from dedicated OR gates, AND together to form the activa-

tion function. If two sequences netxn) and netzm are statistically independent, the

probability of a pulse occurrence, 1),, in the activation sequence is

v,- = P(v,-(,,) = l)

= P(net+

i(1i

) = I A net-"(1,, = 0)

= netﬂl — netf)

58

= [1— H(1 -— wit-15)] ﬁ(1 + wfjvj). (3.20)

i=1 3:1

The nonlinear activation function, described in equation 3.20, is continuous and
differentiable, indicating that back-propagation can be used for training [8]. This
form of stochastic computation will be used for developing a generic DMNN archi-

tecture in the next chapter.

3.5 Back-Propagation in the DMNN

The DMNN is a feedforward neural network which can be trained with the back-
propagation algorithm discussed in Section 2.1.3.2. The back-propagation algorithm
performs gradient descent iteratively over a sum-squared error measure. This section
shows how the non-traditional neuron activation function described in the previous
section is incorporated into the back-propagation algorithm.

Deﬁne n; as the number of neurons in the ith layer. The input layer is not
counted as a layer. Accordingly, for a k-layer DMNN, no and n). indicate the number
of elements in an input pattern and the number of output neurons in the output layer,
respectively. The training for the DMNN can be done off-line or on-line using a digital
computer. The choice depends on whether or not the resolution of the DMNN can
represent the changes of synaptic weights during training for a particular application
problem. The resolution of an n-bit DMNN is 271:1. For example, it is approximately
10’3 for an 10-bit DMNN. However, more than 10"5 precision is often required in
most application problems. That is the reason that the DMNN must be trained off-
line in most cases.

Whenever an input pattern is presented to the network, the output pattern of the

59

output layer is compared to the target pattern; the connection weights between layers
are modiﬁed in a backward direction according to the error. Given pattern ,u, the

sum-squared error measure is

= - :20: -v,,,-) (3.21)

2i=1

where t“; is the target output for the ith neuron in the output layer when input
pattern p is presented and v,“- is the ith element of the actual output pattern. The

overall measure of the sum-squared error over p training patterns is

E = 2: E, (3.22)

Thus, the back-propagation rule states that
Aw.,( k): 211,211.,“ (3.23)
:1

where the subscript k denotes the number of iterations and Anus-,- is the change to be
made to the weight from the ith to jth neuron unit following presentation of pattern

,u. The gradient descent rule for positive weights states

as,
aw};

WBE va 8nd,],-
- ﬂavuganetl} 8w}:- . (3’24)

 

411103;“) = ’77

 

Similarly, the weight change for negative weights is given by

77,,BE 8v,“ Brief;-
ﬂlavuganet; 01.0,,-

 

AuwS-(k): (3.25)

 

60

 

 

 

 

Deﬁne
6 _(9E,,
u: — 1
at)”;
+ _ 8E” _ e ‘ 6v,“-
“" — Bnet; — 1.. Bnetti’
8E 61) g
5;; = ——u_ = Cut—p_ - (3.26)
and”; anal,"-
By equations 3.18 to 3.20, we can obtain
81),“-
= 1 — t_.,
Bnetz} ne ‘"
anet+- -
+‘" =(1— net:,)—Z”—f,_-—
610,-,- 1 — tub-v“,
and
3v,"-
—— = —net+-,
Bnet; “‘
Bnet; v -
—T‘ = —(1 — net’,)—M—.
8w,j “ 1 + wfjvu,
In the output layer,
6],; = t“; — Um. (3.27)

In the hidden layers,

__ as.

C - _ _
I“ 8v,“-

 

61

_ 2“ 6E“ Onettk+z_ 8E, and;c
— k anettk 3v,“- anet;k 3v,“-

’6
= £161}, (l—net: k)1—:———

 

 

mic-+1”;

’W]+:[—6;1—net;k)i—Iwi—-]. (3.28)

kiviu'

Then, the changes in positive and negative weights, resulting from the presentation

of training pattern ,u are described respectively by following recursive forms:

 

A.w3§-(k) = 775.11%
= 775:,(1 — net;- l—j-blgv—m (3.29)
and
. .3... =
= —176;,-(1 — new-1:11:35; (3.30)
where 6:,- = (Em-(1 — net;,-) and 6;,- = —e,,,-net,']’,~.

The back-propagation algorithm incorporating the activation function imple-
mented in the DMNN has two forms:

772?: 6+;‘(1 — net+¢)::137 If w,-- = w?-
3...,(1) = I ‘ " 1 ’ ’ (3.31)

—n Zﬂﬂ 6;,(1 — "660% if M5 = 10,—].

Ijv.’

Gradient descent, described above, can be extremely slow for small 7) while it can
oscillate for large n [43]. In order to achieve the most rapid learning, a learning rate 17
which is as large as possible without leading to oscillation must be chosen. One way

to accelerate the learning is to add a momentum term.

P
AngUC) = —77 2 8E“ + aAw;,-(k — I) (3.32)

“=1 5ij

 

62

where a is the momentum parameter such that 0 S a S 1. 0 determines the ef-
fect of past weight changes on the current direction of movement in weight space.
This provides each connection weight 10,-,- with a kind of momentum so that it tends
to change in the direction of the average downhill force instead of oscillating with
high-frequency variations of the error surface in the weight space. In turn, the effec-
tive learning rate can be made larger without divergent oscillations occurring. A C

program implementing the back-propagation in the DMNN is listed in Appendix B.

CHAPTER 4

Pulse-mode Digital Multilayer

Neural Networks

 

In this chapter, digital architectures of basic elements such as synaptic elments
and neuron body elements are developed. Using these basic elements, the modular
architecture for digital feedforward neural networks is developed as a Digital Multi-
layer Neural Network (DMNN). Use of simple logic gates as computing elements and
modular design techniques will lead to the DMNN architecture being relatively com-
pact in size and expandable to any size network. Furthermore, massive parallelism
embedded in stochastic computations using random pulse streams is fully utilized with
this architecture. A generic architecture of a DMNN coprocessor is also presented.
All components in the DMNN and the DMNN coprocessor are modeled and simulated
in VHDL. Use of VHDL as the modeling tool for the DMNN coprocessor is discussed

brieﬂy. Finally, the hardware complexity of the DMNN is estimated.

63

64
4.1 Basic Computing Elements

A random pulse generator, a synaptic element, an input neuron body element,
and a regular neuron body element are developed as basic computing elements in the

DMNN. These basic elements are used to develop a modular network architecture.

4.1.1 Random Pulse Generator

The block diagram of a random pulse generator was presented in Chapter 3. The
random pulse generator (RPG) is comprised of a tapped LFSR and a digital com-
parator. In Figure 4.1(a), the order of a LFSR is 8 and the example feedback function
is f(sc) = 2:2 69 32;, EB :54 ER 328 implemented by XOR logic gates, where the period of
sequence v(,,) is 28 — l = 255. Figure 4.1(b) shows the structure of a random pulse

generator using D ﬂip-ﬂops, XOR logic gates, and a digital comparator.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

x1 XZ - - .. x6 X7 x8
D . D D D D
F» FF FF FF FF FF
elk t t l L

V1 IV

—C f(x)=Jc2691c369Jc,€BJc8 D

 

 

 

    

 

 

 

 

 

(20
vi vl(n)T
[k x vim) ,l] mu Ht ,
C LFSR '—* , ,

Digital

comparator

(b)

Figure 4.1. (a) A maximum length 8-order LFSR where f ( x) = x, e x, a; x, @ x8;
(b) a pseudo-random pulse generator for v,.

65

At every clock period, a logic ‘1’ pulse is generated if v,- 2 3:. Otherwise, a logic ‘0’

pulse is generated.

4.1 .2 Synaptic Element

A large number of synaptic multiplications are required, even for a small size
feedforward neural network. For example, if the network consists of m layers exclud-
ing an input layer, the number of synaptic multiplications required per feedforward

operation is

m
2: n1n1_1
1:]

where n; is the number of neuron elements in the lth layer and no is the number of
input patterns applied to the input layer. Each synaptic multiplication in the DMNN
is performed relatively more slowly than a deterministic calculation done on a digital
computer, but all the multiplications in the network can be performed in parallel.
Let w;,- and v,- be the synaptic weight between neuron elements i and j and the
neural activation in neuron element i, respectively. Figure 4.2 shows the structure
and block diagram of a digital synaptic element (SYN). The VHDL code for a SYN
model is listed in Appendix C. The SYN consists of a random pulse generator (RPG),
a weight register, two AND gates, and two wired-OR lines. Weight m, is represented
as an r-bit fractional number, where the MSB is a sign bit and the rest represent
the magnitude in sign-magnitude format. With ng loaded into a weight register, the
corresponding random pulse stream w,,-(,,) is generated through the RPG. The pulse
stream is transmitted to two AND gates: the upper one for positive weights and the
lower one for negative weights. If the synaptic weight is positive, a resulting product
sequence mil-1'01) is transmitted to an excitatory net-input line. Otherwise, mE'J-(n) is

transmitted to an inhibitory net-input line.

66

net,( ,f net,(n)'

 

 

 

 

sign bit: ‘1’: negative

 

 

 

 

 

 

 

 

 

 

 

 

l

lohd 1:. ‘ register .
select D: A

Clk > on LFSR
vj(n) >

 

 

 

magnitude

 

 

 

 

 

Wij(n)

 

u o

 

 

 

 

 

 

U

(a)

load Clk select

(b)

mij<n)+
mij(n)'

T'"\ "11501)+
J_/ ’
"ti-(Di
T\ J
J_/ ’

Figure 4.2. (a) A synaptic element (SYN); (b) a block diagram of a SYN.

4.1.3 Input Neuron Body Element

An input neuron body element (INB) consists of an n-bit register, a tapped LFSR,

and a digital comparator. Figure 4.3 shows the structure and block diagram of the

INB. The tapped LFSR and the digital comparator forms a random pulse generator

(RPG). The role of the INB is to convert the value of the ith element in an input

pattern, 1),, to a corresponding random pulse sequence v,(,,). No computation occurs

in this element. v,- is loaded into the register at the rising edge of the clock with load

= ‘1’ and select = ‘1’. The select signal corresponds to the word line from an address

decoder. A binary pulse v,(,,) is generated every clock cycle when load 2 ‘0’.

select Vi

 

 

 

 

select v,
load

n-bit register } l
clk ‘ _

..................... Clk I INB

 

Hr

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

‘ LFSR §R
. P
V l
comparator g
_ ., 7., , Vim)
(a) (b)

Figure 4.3 . (a) An input neuron body (INB); (b) a block diagram of INB.

4.1.4 Regular Neuron Body Element

Two net-input pulse streams, transmitted from synaptic elements, are collected
in an up-counter in a regular neuron body element (RNB) through an AND gate to
form a neural activation. Figure 4.4 shows the structure and block diagram of an
RNB. A VHDL model of the RNB is listed in Appendix C. An RNB consists of an
AND gate, an OR gate, an up—counter, 2x1 multiplexers, a buffer, and an RPG. Let
net? and net,- be an excitatory and inhibitory input for neuron i, respectively. In a
DMNN, product sequences 772,-,(,,) from synaptic elements are logically ORed to form

a net-input. netﬁn) is ANDed with (neti—(M) to form a neural activation v,- in neuron

i, where ("Gig—(7.)), is the complement of new"). v,- is estimated as 23,- which is actually
the value of the up-counter after each iteration.
After each iteration, the signal new_iter changes ‘0’ to ‘1’ and then the output

of a counter is transferred to a buffer via a 2x1 multiplexer at the next clock. At

a same time, the up—counter is reset. This output is used to generate a new action

 

68

pulse sequence v,(,,) while the up-counter continues to accumulate incoming pulses. v,
(dotted arrow) is used as an output of a neuron i in the output layer, while vim) (solid
arrow ) is used in the hidden layers. If load is ‘1’, the buffer is reset to an initial input
value at a new clock cycle. Otherwise, a new neuron state is loaded when new_iter

is ‘1’. The buffer is enabled when load or new_iter is ‘1’.

”etim); * ”etim)-

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

clk . U L new_iter
t r
data " ”u" 6 J load
-> . l I <-
Mux (2x1)
select I
-> Buffer
*—
LFSR '
Comparator
Vi(n)l vi F
(a)

data netimf netim)’

l

clk new_iter
select: RNB :: load

vim) i

 

 

 

 

<>+

(b)

Figure 4.4. (a) A regular neuron body element (RNB); (b) a block diagram of RNB.

69

4.2 Modular Architecture

The DMNN can be constructed from four basic modules: input layer module,
synaptic array module, regular neuron body array module, interconnection module.
The input layer module (ILM) is composed of a group of input neuron body elements.
It receives inputs and transforms them into corresponding binary pulse sequences.
Figure 4.5 shows the structure of the ILM. The pulse sequences generated are trans-
mitted to the synaptic elements in the next layer through an interconnection module
(ICM). A synaptic array module (SAM) consists of a group of synaptic elements and
net-input lines. Figure 4.6 shows the structure of the SAM. Synaptic weights 111,-,- are
loaded before network operations start. w,,-(,,)’s from SYNS in the SAM are logically
ANDed with v,(,,) transmitted from the previous layer. All synaptic multiplications
in the same layer are performed simultaneously. A regular neuron body array module
(RNAM) consists of a group of regular neuron bodies. Figure 4.7 shows the structure
of RNAM. Pulses on two net—input sequences transmitted from the SAM are collected
in an up-counter of neuron i through an AND gate to form v,-(,,). All v,(,,)’s from the

RNAM are produced simultaneously.

select data

 

 

ll? l l l

load B i
l - - - l INB —
c l k _> INB [NB _ INB

Vi(n)

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 4.5. An input layer module (ILM).

7O

clk
load select

 

 

 

 

 

 

 

 

 

w” in! in y y y in luv

SYN

 

 

 

 

 

SYN SYN SYN

 

 

 

 

Q
- (9 O

 

 

 

 

SYN SYN SYN YN

 

 

 

 

O O
O O

_(

 

 

 

 

SYN SYN SYN SYN

 

 

 

 

 

 

 

 

 

SYN SYN SYN SYN

 

 

 

 

QC
. QC
QC
QC

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

V W W

”e’i(n)+ "elm-

Figure 4.6. A synaptic array module (SAM).

data "elf "elf
l l l I l l

I l
clock _> _ ;__— __ — —<— new_iter
select —> RNB — RNB —- ' ’ ' RNB —. RNB .—+ [00d

Y Y

Vi

<—

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

¢
4

Pulse streams

Figure 4.7. A regular neuron body array module (RN AM).

71

Since the action pulse sequences from the previous layer are used to form the
action pulse sequences in the neurons of the next layer, there may exist a correlation
between new action pulse sequences in the next layer without the buffering scheme
used in the RNBS. The dedicated pulse generator in each RNB functions as a ﬁlter
eliminating correlation by producing the uncorrelated pulse sequences for the new
neuron activations formed after each iteration. Again, an [CM is a group of connec-
tion lines that transmit action pulse sequences from the previous layer to the SAM
in the next layer.

Using the modules discussed above, any size DMNN with an arbitrary number
of neurons in each layer and an arbitrary number of the layers can be conﬁgured.
Figure 4.8 shows the general architecture of the DMNN. Just like in a pipelined ar-
chitecture, once the layers are full, results from the output layer are available after

each sampling clock period.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Input F i
ILM
layer I I —“'> data
ICM [W .
I I . ‘— clk
Hidden SAM bl
[23‘ net ena e
layer J ' 1 '-
RNAM +— net_load
ICM [ﬁt|
. §<—— select
] l l
Output I SAM L I new_iter
l l
layer RNAM
I F

 

 

 

Figure 4.8. The general architecure of the DMNN.

72

An advantage of this architecture is that there are no unused synaptic elements in the

network and there is no n—bit prescaler in a neuron body. This increases the possible

neuron-density on a chip while retaining the high degree of ﬂexibility and expand-

ability. The DMNN network can be optimized in terms of the minimum number of

neurons in a given number of layers when it is customized for a particular application.

4.3 DMNN Coprocessor

The DMNN can be viewed as a coprocessor as shown in Figure 4.9. The copro-

cessor is composed of a DMNN, a controller, a memory, an iteration counter (IRC),

and a clock generator. The DMNN coprocessor can be attached to a host computer

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

r/w

 

 

clk_enable
CIOCK ‘ Coprocessor
new_iter controller
—> IRC
A A
01k net_load
l net enable
Y Y
data
‘ h
I address >
DMNN Y Y Y
Memory

 

 

 

control

data

address

 

 

 

Figure 4.9. A DMNN coprocessor.

Host

 

 

73

through an standard interface (not shown). The controller consists of a microprocessor
and a control unit which can be microprogrammed or hardwired. The memory and
the DMN N in the coprocessor may have their own address decoders. The network
is trained on the host computer. After training, the network conﬁguration, trained
weights, input patterns, and some control commands are downloaded from the host

memory.

4.4 Behavioral model of a DMNN Coprocessor

The DMNN architecture proposed in the preceding sections is modeled using
VHDL. The design methodology and procedure for developing a DMNN coprocessor
are discussed here. A DMNN coprocessor and a DMNN architecture are used as
example behavioral models using VHDL in this section. Complete VHDL code listings
for all other components required to implement the coprocessor can be found in

Appendix C.

4.4.1 Introduction

Functional behavior and structure of the DMN N can be modeled and simulated
using VHDL. Various advantages are obtained by using VHDL as a modeling tool for

the DMNN architecture. These advantages include:

1. The function and logical organization of the DMNN can be developed and tested

without involving details of the implementing technology.

2. Design modiﬁcation resulting from design errors or changes can be made in an

early stage of design process without additional cost.

74

3. VHDL is ﬂexible, allowing different network conﬁgurations to be modeled with-

out signiﬁcant changes to an original design.

4.4.2 Design Methodology

A top-down design approach is needed to manage the complexity of large systems
like a DMNN. The model of the DMNN coprocessor starts with a high-level spec-
iﬁcation of the network. The high level description is decomposed into lower level
speciﬁcations in a hierarchical fashion. Figure 4.10 shows the design hierarchy of a
DMNN coprocessor. At the highest level, the whole system can be viewed as a co—
processor. This coprocessor is built using a DMNN, a control unit, a clock generator,
an iteration generator, and memory.

Each component (or VHDL design entity), shown in Figure 4.10, is described in
VHDL using either of two styles of descriptions: behavioral or structural. All compo-
nents (dotted boxes or ovals) residing in lower branches of the hierarchy are modeled
using behavioral descriptions. The reason for this is that the structures or gate level
designs of these components are well known and available in most design automation
(DA) libraries. Secondly, the complete structural descriptions for these components
cause the resulting VHDL simulation kernel to run extremely slow and to occupy a
large amount of memory. Each behavioral description can be eventually replaced by
a structural description without changing other entities. All other components are
modeled using structural descriptions.

This hierarchical design scheme combined with the natural modularity of the

DMNN architecture makes design modiﬁcation and simulation much easier.

75

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

DMNN Coprocessm
M DMNN
Input Layer Hidden Layer Output Layer
ILM SAM RNAM SAM RNAM

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

RNBs

 

 

 

 

 

 

 

 

 

     

         

. 1 ' l \i \ r “ "i F“
3..er K2?) 1G.) EEG) (RGG .3...ng as) ten

Figure 4.10. The design hierarchy of a DMNN coprocessor

76

4.4.3 Coprocessor Control in VHDL

In order to model the DMNN coprocessor in VHDL and apply the model to some

example problems, the following procedures have been followed.

Step 1. Back-propagation for the DMNN, described in Chapter 3, trains the net-
work, ﬁnds a network architecture converging below a predeﬁned sum-squared

error, and then writes the ﬁnal synaptic weights and
Step 2. The DMNN coprocessor reads these ﬁles and a ﬁle for test patterns.

Step 3. The coprocessor initializes the LFSRs in the network and loads synaptic
weight registers with the trained weight values before the network operation

starts.

Step 4. The coprocessor initiates the controller, and the controller starts generating

control signals.

Step 5. The network (DMNN) starts classifying input patterns until classiﬁcations
are completed. The output patterns are written into memory each time one

input pattern is classiﬁed.
Step 6. Results are written into a ﬁle.

A part of VHDL code for the DMNN coprocessor is shown as a test bench in F ig-
ure 4.11. The complete code is listed in Appendix C. The network architecture,
problem speciﬁcation, variable deﬁnition, and some subprograms are described in a
package, named dmnn_paclc. Once the run signal changes from ‘0’ to ‘1’, a simulation

kernel autonomously performs Step 2 to Step 6 as above.

 

77

use work.dmnn_pack.all;
entity dmnn_test_bench is
end dmnn_test_bench;

architecture behavior of dmnn_test_bench is
component clock
port(start: in bit;
clk_out: out bit);
end component;

component step_cnt - - - end component;
component dmnn_net - - - end component;
component control_unit - - - end component;
component RAM - - - end component;

type out_bits is array(natural range 1 to out_unit_no) of bit;

signal run, clk_sig, net_1oad: bit:='0';

signal net_ena_sig, clk_ena, new_iter: bit:='0';

signal pattem_sig: neura1_array; signal weight_sig: neural_matrix;
signal state_sig: out_state_array;

signal ram_rw, ram_ena: bit;

signal ram_net_state: pattem_matrix;

for all: clock use entity work.clock(clk_behavior);

for all: step_cnt use entity work.step_cnt(step_cnt_dmnn);
for all: control_unit use entity work.control_unit(behavior);
for all: dmnn_net use entity work.dmnn_net(behavior);

for all: RAM use entity work.RAM(behavior);

begin run <= '1' after 5 ns;
C1k_Blk: clock port map (clk_ena, clk_sig);
Step_B1k: step_cnt port map (clk_sig, new_iter);
Control_B 1k: control_unit port map (run,new_iter,net_load, ram_rw,
clk_ena,net_ena_sig,ram_ena);
dmnn_net_BLK: dmnn_net port map (net_load, clk_sig, net_ena_sig,
new_iter,pattem_sig,weight_sig,state_sig);
RAM_BLK: RAM port map (ram_ena,ram_rw,state_si g,
pattem_sig,weight_sig);
end behavior;

Figure 4.11. VHDL code implementing the DMNN coprocessor.

 

78

4.4.4 DMNN Model in VHDL

The DMNN itself consists of an input layer, a hidden layer(s), and an output
layer. Each layer and all components required in the. layer are generated by compo—
nent instances in structural descriptions. Figure 4.12 shows the part of VHDL code
for the DMNN model. Any size network can be modeled by specifying the network
conﬁguration in the d77mn-pack. The components are connected by wires, modeled
as signals in VHDL. The network supports complete connectivity between neuron
elements, but unnecessary synaptic elements are effectively disconnected by loading

a zero weight for that synapse.

use work.dmnn_pack.all;

entity dmnn_net is port ( net_load, net_clk, net_enable, next_period: in bit;
new_pattern: in neural_array; weight_in: in neural_matn'x;
net_state_out: out out_state_array);

end dmnn_net;

architecture behavior of dmnn_net is

component synapse - - - end component;

component neuron_body - - - end component;

signal clk: bit; signal syn_ran_in: neura1_matrix;

signal body_ran_in, net_output: neural_array; - — -

for all: synapse use entity work.synapse(synap_dmnn);

for all: neuron_body use entity work.neuron_body(nbody_dmnn);

begin Main_blk: block(net_enable=’1‘) begin
clk<= guarded net_enable and net_clk;
ex_wired_or<=guarded Wired_Or(ex_or);
in__wired_or<=guarded Wired_Or(in__or);
end block;
lnput_Layer: - - -
Hidden_Layer: - - -
Output__Layer: - - -
end behavior;

Figure 4.12. VHDL code implementing the DMNN.

 

79

4.5 Hardware Complexity

To access the hardware complexity of the DMNN, deﬁne Ag and A00 as the chip
areas required by an n-input NAND gate and by component X, respectively. For a
reasonable number of inputs, say n S 10, it is assumed that an inverter is viewed
as a special case of an n-input NAND and an n-input AND is modeled as an n-
input NAND followed by an inverter. If all components and modules in the DMN N
are designed only using N AND gates and necessary connections, a measure of the

hardware complexity of a DMNN with m layers is given by

A(DMNN) = AUNB) ' no + Auma) Z n; + A(SYN) Z 71; 'ni-l (4-1)

t=1 1:]

where no and n,- are the number of elements in an input pattern applied to the net-
work and the number of neurons in the ith layer, respectively.

Assume that a D or J-K ﬂip flop (FF) is implemented by 5 NAND gates
(Aurp) = 5A,,), a 2x1 multiplexer (MUX) by 4 NAND gates (A(MUX) = 4A,,), and
a two-input exclusive OR (XOR) gate by 4 NAND gates (Agog) = 4A,). The gate
level schematics of essential basic digital components, such as an n-bit register, an
n—bit comparator, and an n-bit up-counter, can be found in Appendix D. Table 4.1
presents the relative chip area required by each of these components based on the
unit NAND gate.

In Table 4.1, it is assumed that 3 XOR gates are needed to implement a feedback
function of a LFSR. This leads to the assessment of hardware complexity for the IN B,
RNB, and RNB modules. Table 4.2 shows the chip area required by these DMNN
elements.

Using equation 4.1 and Table 4.2, the chip area required by any size DMN N can

be estimated. Table 4.3 presents the chip area of two example networks with respect

80

to the register length n and the unit N AN D gate area Ag. A network conﬁguration
in this table is represented by no x n1 x .. ., where no and n,- denote the number of

neurons in the input layer and the number of neurons in the ith layer, respectively.

Table 4.1. Chip area required by basic digital components.

 

 

 

Component Chip area
n-bit register (RGT) nA(Fp) + 2nAg = 7nAg
n-order LFSR. nA(pp) + 3A(XOR) = (5n + 12)Ag

n 2x1 MUXs (MUXs) nA(MUx) = 4nAg
n-bit comparator (CMP) 6nAg + 2Ag = (6n + 2)Ag
n-bit up-counter (CNT) nA(pp) + 2(n f 1)Ag = (7n — 2)Ag

 

 

 

 

 

Table 4.2. Chip area required by DMNN elements.

 

 

 

DMN N element Chip area
INB Amer) + A(LFSR) + A(CMP) = (1872 + 14)Ag
RNB A(INB) + A(CNT) + A(MUX..) + 5A9 = (2972 + 17)Ag
SYN AUNB) + 5149 = (1877. +19)Ag

 

 

 

 

 

Table 4.3. Chip area required by two example networks.

 

 

 

 

Register Network conﬁguration

length (n) 36 x 9 x 10 36 x 30 x 10
8 77,901 Ag 226,788 Ag
9 86,554 Ag 253,436 Ag
10 95,203 Ag 280,084 Ag

 

 

 

 

 

 

CHAPTER 5

Analysis of the DMNN

 

The statistical model of the estimated probability generated from the random pulse
generator was developed in Chapter 3. Based on this model, statistical models of
synaptic multiplication and signal integration are developed. Then the models are ex-
tended to hidden layers and the output layer. An analysis which predicts the error
bounds on the results of the computations occurring in the DMNN is also presented.

A critical comparison to deterministically computed results can then be made.

5. 1 Statistical Models

For analysis, it is assumed that multiple pulse sequences are statistically uncor-
related and the effect of quantization is negligible when the register length is greater

than 7 [67, 78].

81

82

5.1.1 Synaptic Multiplication

An algebraic expression for the output of an AND gate in a synaptic element is
mij = w;,-v,-. Since the estimations w},- and 23, of w;,- and Uj can be represented by the
original signal plus random noise (error) in the ﬁrst hidden layer, the estimation mg,-

of m,,- can be described by

"iii : (wij + Ant-MW '1' A18)

= ngvj + ngAo‘j + vjAwﬁ,‘ + Aw1jAU3' (5.1)

where A5: is random noise with zero mean and variance Var(i). The expected value

Of Tl’lgj lS

E(ml'j) = me" = wa'vj (5-2)

and the variance of rrigj is

Var(rri,,-) = 10,2. Var(tij) + v]? - Var(uiij) (5.3)

where the second order term is omitted from equation 5.1. In reality, the second order
term is not zero because two pulse sequences have a very small, but still negligible,

cross-correlation [67].

5.1.2 Two-input Logical OR

Before developing the statistic model of an n-input logical OR, the model for
a 2-input OR is developed. Let mil 2 mam”) = max[m.-1,m,-2] and let nk be the

number of neurons in the kth hidden layer. The statistic model for net,- is complex

OO
00

because the output estimation results from the nonlinear transformation of nib-’5.
It can be approximated, however, using the hypergeometric distribution. The key
reason for using this model is that the deterministic nature can be observed in the
output sequence, as shown in dotted lines in Figure 5.1.
The pulses of 7n,1(,,) always appear on the output sequence net,(,,). W'hen 771,-] =
1,1/N and mo 2 liz/N,
NET,- =li1 +1V.

where W is a random variable indicating the number of pulse occurrences in mm”)
during N — [,1 clock cycles and NET,- (with capital letters) represents the number of

pulses in net,(,,) observed during the sampling period.

mi1(n)
mi2(n) .

neti = mu + mi2 - mu mi2

H H n H H

 

 

 

 

 

H n n n
rm n n H .

Figure 5.1. 2-input logical OR where the dotted lines illustrate the deterministic nature
of the output sequence netim).

84

The probability function of W is

P(W= k) = W

The expected value of net,- is

_ N
E(net,) = %+%Zk-P(W= k)
k=0
In (N — 151)].‘2
N + N2
= mil + mi2 " mama
2

= 1— H(1 - mg). (5.4)

 

 

 

j=1
The variance of net,- is
Var(net,) = Nl—zvaﬁW)
_ NJ-V-z’li'17m2(1 _ mi2)Nl,: 1 (5.5)

Figure 5.2 shows the standard deviation, amen) = W, obtained from
equation 5.5 (a,d) and those obtained by simulation (b,c,e,f). In this ﬁgure, N = 127
for (a) to (c) and N = 255 for (d) to (f). The same feedback functions with different
LFSR initial values for mu and mi; are used in (b) and (e) while different polynomial
functions are used in (c) and (f). 2:1 63 232 69 2:3 EB :34 is used as a feedback function for
LFSRs for mu and 777.52 in (b) and (e). (131 G) 2:2 EB x3 GB 14 and $2 ED x3 63 .15., GB 235 are

used for mu and mg; in (c) and (f), respectively.

8:3

 

 

      
  
    
   
         

, , M
, o @321!!! .
r . "o. w" ‘~ as
///Il/I’I"’I/:'I'":"9‘:‘%’V\G“ of “S‘s
Ill/lgtvdtltirtta‘ti‘i‘ﬁtttt
0%1‘0““\\\\\ t

(”fill/Ill,” t \\\\\

 

lII ' .3 “tin

Ill/7’7 7 1111 iii

III'
‘ \

Ill l\\\\

   

ll

Figure 5.2. one“. obtained from equation 5.5 when (a) N = 127; d) N 255, and

( :
one}, obtained from actual simulations when (b,c) N = 127; (e,f) N = 255.

86

--- 0.023168

I
I
'0‘ \

 

 

111111

 

 

Figure 5.2. Continued.

87

l'" 0.013943

 

 

 

 

Figure 5.2. Continued.

88

5.2 Effects of Random Noise in Hidden Layers

5.2.1 First Hidden Layer

In the following, assume P, = n = N, as is the case in the DMNN. This as- '
sumption eliminates any error associated with the conversion of a synaptic weight w,
and a neuron activation v,- into the average pulse rates of the corresponding pulse
sequences w;,-(,,) and v,-(,,). Since Var(tij) = Var(uiij) = 0 from equation 3.13 when
n = N, then VClT'(7Tl:{J') == 0 in equation 5.3. This deterministic nature of the synap-
tic multiplications in the ﬁrst hidden layer contributes to the high accuracy of the
stochastic computing technique.

Two assumptions are made for an n-input OR. The validity of these assumptions

has been justiﬁed by experimental results. Deﬁne Nah" as the effective period of an

LFSR in the kth hidden layer. Noff(= N) is the effective period of an LFSR in the

6

input layer.

Assumption 1: An increase in the number of OR inputs in the ﬁrst hidden layer
has the same effect as increasing the period N of the LFSR to Nelff with the
sampling period, P, = N, unchanged, and holding the ratio lgj/N such that
me = l.,-/N = (michlffl/Nelff'

Assumption 2: As n increases, the deterministic ratio in net,- decreases such that
m,(ma,,) = mu == max[m,-1, mig, . . . ,m,,,] = ﬂO—ZL; for a ﬁxed net,, where 0 < ,6 S

1.

The ratio, Nclff/N, and ﬂ depend on the application problem, the network architec-
ture, and the input patterns. It is impossible to ﬁnd N61” and E in closed forms.
Based on our experimental results for binary classiﬁcation, however, N61,, can be ap-

proximated as N + a1(no — 2)N for no _>_ 2 where 01 is an incremental parameter for

the ﬁrst hidden layer and no is the number of inputs in the input layer. 01 varies

89

between 0.03 and 0.04. In the following, 6 = 1 is assumed.

Based on these assumptions, equations 5.4 and 5.5 for the 2-input OR can be gen-
eralized for an no—input OR in the ﬁrst hidden layer. To do so, separate the no mgj’s
into two quantities: mil = mam”) = 11"} and m;, = 1-ng,(1—m.—,) = mngclff/Nelff
Now E(nét;) and Var(net,) can be obtained in a straightforward manner. N ET,- is

the summation of a constant l.-1 and a random variable W,
N ET.- = 1.1 + W.

The probability function of W for an n-input OR operation is

(ﬁliera )(Nla "mierir)

 

P(W = k) = N’f'h‘k
(Ne-ill“)
Thus, the expected value of net,- is
E( at) “1+1ﬁvjk P(W k)
ne 5 = — — ' =
N N k=0
_ In (N - li1)lir
— N + —N2—’
no
= 1— (I — m;1)H(1— mgj)
i=2
= 1— H(I - m,,). (5.6)
i=1
The variance of net,- is
(N—ln) Neln"(N—ln)

Var(netg) =

 

90

If (net.- — m;1)/(1 — mg) and net,/, /no are substituted for m,,. and mil, respectively,

 

 

 

. 1 — —l— N1 — N :-
Var(net,-) = i gnetgﬂ — netg) C”, + ‘1
N1 — 71,—: Ne}, — 1
1 1
= I{i}\{{b net,(1 — netg), (5'7)

where K; = (1 — VlT—ol/(l — 3%) and K51 = (N61,, — N + l;1)/(Nclff — ).

5.2.2 Kth Hidden Layer

The procedure developed in the previous section for the ﬁrst hidden layer can be
expanded to generalized stochastic models for a synaptic multiplication and a logical
OR in the kth hidden layer. In the kth hidden layer, E(TTiij) and VGT(TT£{J') can be
described by equations 5.2 and 5.3. Var(Trlij) is non-zero in the kth hidden layer
when k 2 2 because the ﬁrst term in the right side of equation 5.3 is non-zero due
to the random noise introduced by the lower layer. One more assumption is made to
take this effect into account, where the input layer and the output layer are regarded

as the 0th hidden layer and the last hidden layer, respectively.

Assumption 3: The random noise introduced from the lower layer ((k-1)th hidden
layer) causes al. to increase such that a], Z ak_1, while assumptions 1 and 2 are

still held. For our analysis, on. = ak-1 + 0.005 has been used.

Thus, the effective period of an LFSR in the kth hidden layer is

= Nf,-,‘[1 + dun.-. — 2)]

N H[l + a,(n,-_1 — 2)]. (5.8)

i=1

91

As a result, the generalized forms of equations 5.6 and 5.7 for the kth hidden layer

 

are
. nk—l
E(net,) = I - 11(1— mgj) (5.9)
i=1
and
k k
Var(net,) = KR?" net,(1—net,-), (5.10)

 

where K5: (1— wail/(1" —’;—°‘/_fl_:) and K: = (fof — N + 1,1)“an — 1).

If statistically independent Bernoulli sequences are assumed, K 5K: = 1 and the
distribution of net,- is binomial. In equation 5.10, Var(net,) is bounded to that of
the binomial distribution as the ng’s, for i < k, become very large. Figure 5.3(a)
shows Kfo with respect to Ti]; and lc when N = 255, no = 36, and net,- = 0.55. As
seen in Figure 5.3(a), as n1 and n2 become large the distribution tends towards the
binomial. Figure 5.3(b) shows the standard deviation of nét, with respect to net,- for
various network conﬁgurations. The standard deviation of the binomial distribution
is shown for comparison and the standard deviation of net,- is skewed to the right

because 1(1fo increases as net,- increases in equation 5.10.

92

 

 

 

 

 

 

 

 

 

l
0.9»
0.8 ~
0.7 ~
.2
.D
.3: x:k=2andn1=n
a: 0.6~ ‘ o:k=3,nl=20,andn2=n ~
+:k=3,n1=50,andn2=n
*:k=3,n1 = 100, and [12:11
0.5, -:k=3.n1=200,and n2=n .
0.4? o -
X
02 1 1 1 1 1 1 1
2 10 30 50 70 90 200 400 600 n 800 1000
(a)
0.035 r I I I I l 1 I
0.03 v x x -
. i i + + i : .
X + +
0.025 - x + i -
+ . . . ° 0 X
x . o +
+ . o ' . 1
f 0.02- i . l -
b: . . ° 3
0.015 ~ ' ' ﬂ
‘ *:k=2.no=36.andn1=10 .
+1k=2,no=36,andnl =50
0-0'” . x:k=2,no=36,andn1=100 ‘
-: binomial distribution
0.005 - -
O 1 1 1 1 1 1 1 1 1
0 0.1 02 0‘1 04 0.5 06 07 08 09 l

net,

(1))

Figure 5.3. (a) K 5K ,1“ with various network conﬁgurations when net,- = 0.55, no = 36,
and lc > 1; (b) standard deviation of nét, with respect to net,.

93

5.2.3 Neural Activation

Finally, the statistical model of estimation 13,- of a neuron activation in any layer

can be expressed as

23, = (net;F + Am?) (1 — net,— + Ami-ti")

= netf(1 — net?) + nethm-g + (1 — netf)AM‘t,

t.

where the second-order term is omitted as before. The expected value and variance

of 23,- in the kth hidden layer are
E(zig) = netf(1— netf) (5.11)
and
VGT(’(3,’) = netszar(net,-') + (l — netf)2Var(ndt§"). (5.12)

As n), and k become very large in equation 5.10, the distributions of ndtf and (1
- net: ) are both approximately binomial. Furthermore, if P, = N is very large, the
distributions are close to Gaussian. Since the random error of 13,- is approximated as
the linear transformation of the random errors of net? and net," in equation 5.12, the
distribution of 13, is also close to Gaussian. Figure 5.4(a,b) shows the actual distri-
bution of a neuron in the output layer as obtained by simulation with v, = 0.54 (a)
or 0.45 (b), k = 2,no = 25,n1 = 5, and n; = 5. As Figure 5.4 indicates, the output

distribution is observed to be close to Gaussian while the variance is relatively small.

94

 

 

 

 

 

 

 

 

 

800 -
*: binomial distribution
700 _ + +: distribution in the output layer
, + 0: distribution in the hidden layer ‘
9
600 ~ ‘ + -
>~. 5m F o o 0* “
s . °
‘5 o .‘u' i
3' 400 » o . ’f" A
d: o" .
' ‘1’
300 " t ’0 ‘
6 ob
9 + ¢ 0
2m "’ . 9 +00 ‘
100 " 3002 Q .. ..
.. * +0 0
32v“ 4”
- we. . add...
0.25 0.3 0.35 0.4 0.45 0 5 0.55 v- 0 6 0.65
l
(a)
800 - 2 -
’ *: binomial distribution
700 _ . +: distribution in the output layer _
i 0: distribution in the hidden layer
0
600 '- ’ 4» '1
o 000
500 — °° ° ~
>5
8
g .0....' 6
g. m " :0 . ..
IL. + 0°
0 0°
3m _ .a ’. d
.o *;
2m " . +0. ‘
. O + .
0.00;. 0 e
100 " :00 9 ,0 o. d
3.02;: +9'9 .0.
0 .- - ’ . 1 Mar“ - 1
0.35 0.4 0.45 0 5 0.55 0.6 0 65 v 0 7 0.75
I
(b)

Figure 5.4. The distribution of 23,- in the hidden layer (0) and the output layer (+) for

8825 tests compared to a binomial distribution when v, = (a) 0.45 or (b) 0.54, k = 2,

no 2 25, n] = 5, and no = 5.

95

5.3 Network Performance Model

To effectively use this statistical model for analysis, architectural information on
the number of layers and the number of neurons in each layer, as well as represen-
tational information on the values of the input and target patterns are needed. In
addition, some assumptions on the values of net? and net,’ must be made. Those
values depend on input and target patterns as well as the application problem. In
addition to obtaining the variances of estimated outputs in the output layer using
equations 5.9 to 5.10, the model will enable the differences between the results ob-
tained from the DMNN and the results obtained from deterministic calculations to
be presented.

Binary classiﬁcation problems are one of the possible target applications of the
DMNN and are considered here for analysis. The number of the output neurons cor-
responds to the number of classiﬁcations. 0.45 and 0.55 are used to represent ‘on’ and
‘off’ of a neuron in the output layer, respectively. The network is trained in such a
way that only one output neuron is ‘on’ for a given input pattern. After training, the
architectural and representational information is ﬁxed. Two network conﬁgurations
are considered: a two-layer feedforward network with k = 2,no = 25,n1 = 5, and
n2 = 5 and a three-layer feedforward network with It = 3,no = 25,n1 = 10,n2 = 5,
and n3 = 5. Tables 5.1 and 5.2 show the standard deviations of the estimated output
values in the given network conﬁguration with respect to various possible combina-
tions of net?" and net,- leading to the target values (0.45 and 0.55); 01 = 0.035 and
ﬂ = 1 are assumed.

Taking the distributions of 13,-(‘on’) and 13,-(‘off ’) to be Gaussian, the network cor-
rectly classiﬁes the input pattern as the classiﬁcation i if 13; > 0.5 and 13, S 0.5 for
all j 75 i. Consider ﬁrst the case of a network with two output neurons and deﬁne

the random variables V0,, 2 ti,(‘on’) and Voff = 13,-(‘off ’). The density functions of Von

96

and V0“ are
1

fvon(v0n) = -———C_(v°n—ﬁm)2/2a©on
#27?me

and
1 — —’ 2 202
e (”on ”all)/ v0,

2
27rovou

fVo/j(v0ff)= I,

where 60,, and 6,,” are the mean values of V0,, and V,” respectively.

Thus the probability of the event {Von S 0.5} is

P(V,., g 0.5) = va(0.5)

0.5
= 0 fv...(€)d€
and the probability of the event {Von > 0.5} is

P(v.,,,go.5) = 1—FV,,,(0.5)

= 1— [imam

where Fvo..(von) is the distribution function of Van. va(vo,,) can be described in
terms of the standard Gaussian distribution function F (u) by making the variable

change
(van _ Fan)
O’V .

071

u:

Consequently, Fvon(vo,,) = F(m) and Fv0”(v,ff) = PUMP—17°11).

0V0" 0V0]!

Thus, for a given input pattern, the probability that an input pattern will be correctly

classiﬁed, Pm, in the network with two output neurons is

Pm, = P(V,,. > 0.5 /\ V0,, 3 0.5)
= P(V.,.. > 0.5)P(Vou .<_ 0.5)

= (1 — va(0.5))Fv,,,(0-5)

97

= (1_ F(0.5— 0.55))F(0.5— 0.45),

Oven 0V0]!

where V0,, and V,” are assumed to be statistically uncorrelated.

Therefore, in the general case where the network has n), output neurons

“I:

Pea,- = P(V0n > 0.5) HP(Voff S 0.5)
i?“

P(V.,.. > 0.5)P(V.,,, g 0.5)“-1

: (I — Fvon(0.5))FVoH(0.5)“-1

= (1 _ F(M))p(w)nrl, (5.13)
0V0" 0V,"

where the 2333 are assumed to be statistically uncorrelated.

5.4 Simulations

Performance results for the VHDL models developed in Chapter 4 are presented
here to demonstrate the validity of the statistical analysis.

Two basic experiments have been conducted. The ﬁrst involves classifying 5 digits
from 0 to 4 while the second involves classifying 10 digits from 0 to 9. The networks
are trained with two sets of training data. The ﬁrst training set consists of ideal
digits (1 pattern per digit) and the second consists of the ﬁrst data set plus 10%
noisy patterns (3 patterns per digit). P% noisy patterns were created by randomly
inverting P% of the pixels in the ideal digits. For experiment 1, each pattern consists
of 6x4 pixels and for experiment 2 each pattern has 7x5 pixels.

Table 5.3 shows Poo, for three different network conﬁgurations obtained from equa-
tion 5.13. In this table, the ﬁrst column indicates the order of the LFSR, the second
column shows Pm directly obtained from equation 5.13 while the third column shows

correct classiﬁcation rates obtained from VHDL simulation. The top row in the table

98

shows the network conﬁguration in form of no x n1 x no where no, n1, and n2 indicate
the number of neurons in the input layer, the ﬁrst hidden layer, and the output layer,
respectively. Figures in parenthesis in the third column indicate the correct classiﬁca-
tion rates when the classiﬁcation is determined by the neuron with maximum value.
Only two-layer networks (one hidden layer and one output layer) are considered here.
Each ﬁgure in the third column is based on extensive tests (> 20,000 runs). Twenty
sets of weights are used and more than 1000 different sets of initial LF SR values per
set of weights are used. As shown in Table 5.3, the DMNN character recognizers pro-
duce the correct classiﬁcation rates close to PCO,’s in the given network conﬁgurations.
Differences are mainly caused by the fact that the exact values of parameters a), and
6 can not be found although they are bounded for binary classiﬁcation as described in
assumptions 1 and 2. Thus, these values must be selected as arbitrary points within
the boundary to calculate fof. To obtain Pea, in the table, al = 0035,01; = 0.04,
and ﬂ = 1 have been used. Thus, the statistical analysis is shown to produce valid

results for the DMNN character recognizers.

99

Table 5.1. Standard deviations of 23,-(‘on’) and 13,-(‘off’) when no = 25, n1 2 5, n2 = 5,

and (a) N =127; (b) N = 255; (c) N = 511.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

E7175): 0.55 J] E(tig) = 0.45
net? [I 0.6 0.7 0.8 0.9 1.0 net: 0.6 0.7 0.8 0.9 1.0
net,’ H 0.083 0.214 0.313 0.389 0.45 net,- 0.25 0.357 0.438 0.5 0.55
03,-, [I 0.026 0.025 0.025 0.025 0.026 09-, 0.024 0.024 0.025 0.026 0.027
dog/0% H 0.587 0.563 0.561 0.572 0.592 dog/03 0.543 0.543 0.558 0.581 0.601
(a)

( E(ﬁg) :: 0.55 “ E(tlg) = 0.45
net? H 0.6 0.7 0.8 0.9 1.0 net; 0.6 0.7 0.8 0.9 1.0
net,’ H 0.083 0.214 0.313 0.389 0.45 net,’ 0.25 0.357 0.438 0.5 0.55
do... H 0.018 0.018 0.018 0.018 0.018 do“. 0.017 0.017 0.018 0.018 0.019

Uo-i/O'B ll 0.587 0.563 0.560 0.572 0.592 dog/0’3 0.543 0.543 0.557 0.581 0.609
(b)

[ E7175): 0.55 H E(ﬁg) = 0.45 J
net?” H 0.6 0.7 0.8 0.9 1.0 net: H 0.6 0.7 0.8 0.9 1.0
net,‘ I] 0.083 0.214 0.313 0.389 0.45 net,‘ H 0.25 0.357 0.438 0.5 0.55
do, H 0.013 0.012 0.012 0.013 0.013 do". H 0.012 0.012 0.012 0.013 0.013

dog/03 H 0.586 0.562 0.560 0.572 0.592 dog/0'3 H 0.542 0.542 0.557 0.581 0.609
(C)

* 03 indicates the standard deviation of 13,- when Bernoulli sequences are used.

 

 

 

1

00

Table 5.2. Standard deviations of 73,-(‘on’) and ti;(‘off’) when no = 25, n1 = 10, n2 = 5,

n3 = 5, and (a) N =127; (b) N = 255; (c) N = 511.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

[ E07,) = 0.55 1] E(6,) = 0.45 1
net: |] 0.6 0.7 0.8 0.9 1.0 net: 0.6 0.7 0.8 0.9 1.0
net,- [I 0.083 0.214 0.313 0.389 0.45 net,- 025 0.357 0.438 0.5 0.55
a... ”0.029 0.028 0.028 0.029 0.030 6.... 0.027 0.027 0.028 0.029 0.031
(ya/cg H 0.668 0.640 0.639 0.655 0.681 mm; 0.621 0.619 0.635 0.661 0.694

(a)
676..) = 0.55 [1 E(6,) = 0.45
net:r u 0.6 0.7 0.8 0.9 1.0 net: 0.6 0.7 0.8 0.9 1.0
net,- "0.083 0.214 0.313 0.389 0.45 net,- 0.25 0.357 0.438 0.5 0.55
03;, I] 0.020 0.020 0.020 0.020 0.021 a... 0.019 0.019 0.020 0.021 0.022
avg/03 [I 0.667 0.640 0.638 0.654 0.681 Jo‘s/0'3 0.621 0.619 0.634 0.661 0.694
(b)
E(v‘.) = 0.55 [1 13(5) = 0.45
net;F I] 0.6 0.7 0.8 0.9 1.0 net;P 0.6 0.7 0.8 0.9 1.0
net,- [I 0.083 0.214 0.313 0.389 0.45 net,- 0.25 0.357 0.438 0.5 0.55
09-, H 0.015 0.014 0.014 0.014 0.015 a... 0.014 0.014 0.014 0.014 0.015
a..-,/a3 H 0.667 0.640 0.638 0.654 0.680 aux/03 0.620 0.618 0.634 0.660 0.693
(C)

* 03 indicates the standard deviation of 13,- when Bernoulli sequences are used.

 

 

 

101

Table 5.3. Pea, obtained from equation 5.13 and correct classiﬁcation rates from

VHDL simulations.

 

25 x 5 x 5 for 5—digit classiﬁcation

 

Pea, obtained
from equation 5.13

correct classiﬁcation rate
obtained from VHDL simulations

 

8—bit
9-bit

 

 

 

86.55% ~ 90.91%
98.59% ~ 99.18%

 

92.5%(97.9%)
99.3%(99.65%)

 

 

36 x 9 x 10 for 10-digit classiﬁcation

 

Pea, obtained
from equation 5.13

correct classiﬁcation rate
obtained from VHDL simulations

 

 

 

 

 

8-bit 86.64% ~ 92.94% 86.73%(97.8%)
9-bit 97.93% ~ 99.74% 92.6%(99.94%)
10-bit 100% 98.7%(99.06%)

 

 

36 x 30 x 10 for 10-digit classiﬁcation

 

Pea, obtained
from equation 5.13

correct classiﬁcation rate
obtained from VHDL simulations

 

 

 

 

 

8-bit 74.25% ~ 83.7% 80.27%(92.4%)
9511 96.39% ~ 98.64% 91.2%(99.1%)
10-bit 99.91% ~ 100% 99.1%(99.6%)

 

 

 

 

CHAPTER 6

DMNN Application: Pattern

Classiﬁcation

 

A DMNN architecture can be viewed as a coprocessor with the addition of a control
unit and memory components. A DMNN coprocessor for classifying binary patterns is
modeled and simulated in VHDL. A general design procedure for constructing DMNN
binary classiﬁers is discussed. XOR and encoding problems are applied as testbench
problems. Then numeric character classiﬁcation problems are applied. Convergence
properties in DMNNs during training are also discussed. Classiﬁcation performance
for these networks is evaluated in comparison to that obtained from deterministic

DMNN simulations and ordinary back-propagation networks.

6. 1 Introduction

The purpose of pattern classiﬁcation is to perform a mapping from observation
space to interpretation space by extracting features from observed data and classify-

ing the collected features into a certain category [85]. An important aspect of pattern

102

103

classiﬁcation is the determination of the discriminant (or operator) that produces an
estimate of the class membership of an input pattern. Some of the early work in
pattern classiﬁcation was inspired by the idea that a network of processing elements
arranged in a manner similar to a biological neural network might be able to learn
the requisite operators in an autonomous manner.

The Perceptron was the ﬁrst signiﬁcant development of such in the early 1960s,
suggesting a general approach to the automatic learning discriminants for pattern
classiﬁcation [42]. However, at that time, even though the power of the generalized
Perceptron devices were well understood, no practical method for training layered
Perceptron devices was known. Furthermore, the analysis of single-layer Perceptron
devices by Minsky and Papert in 1969 revealed the limitation of the Perceptron [2].
That is, a single-layer Perceptron can not classify linearly unseparable patterns. Since
then, no signiﬁcant further progress in Perceptron-related research had been achieved
until the early 19803. i

The early disappointments with the Perceptron approach caused most traditional
pattern recognition researchers to concentrate on geometric or structural approaches.
In these approaches, pattern recognition systems made use of the results of statistical
communication and estimation theory or mathematical linguistics.

In 1986, Rumelhart, et al. introduced a new algorithm, called the generalized
delta rule (or back-propagation), for training multilayer Perceptrons. The means of
autonomous mapping from observation space to interpretation space is now possible.
However, learning using the ordinary back-propagation algorithm is impractically
slow. Many versions of the ordinary back-propagation have been developed to accel-
erate learning [43].

One current research emphasis is on implementing pattern classiﬁcation systems
using the neural network approach on advanced computer architectures or in dedi-

cated VLSI hardware. In this chapter, binary classiﬁcation problems are applied to

 

 

104

the DMN N to test its applicability to pattern classiﬁcation using the VHDL model

of the DMNN described in previous chapters.

6.2 Methodology

A generic DMN N architecture has been modeled as a coprocessor in VHDL. Binary
classiﬁcation problems are applied to the DMNN coprocessor as testbench problems.
In order to construct the network for solving a particular classiﬁcation problem, sam-
pling data (or patterns) must be selected carefully. The sampling data are often
divided into training data and testing data. 'The DMNN must be trained off-line on
a host computer. Once training is completed, test data are applied to the DMNN
coprocessor whose network architecture is set up as a result of training. Based on the
test results, the performance of a DMNN pattern classiﬁer is evaluated.

A back-propagation algorithm has been programmed in C. Each pattern is of
an array of binary values. The number of neurons in the input layer of a DMNN
equals the number of elements in the pattern vector plus one. The additional input
is required to provide the synaptic elements located in upper layers with a constant
threshold input (1.0). The number of neurons in the output layer is the same as the
number of categories into which the sampling patterns are to be classiﬁed. Only one
output neuron is turned on when an input pattern is applied to the network. For
example, if an input pattern belongs to category # 1, only the ﬁrst output neuron is
turned on and all the other neurons are turned off. The values representing ‘on’ and
‘off’ of output neurons are determined during the training session. These values can

be any intermediate values between 0.0 and 1.0.

105

6.2.1 Training and Classiﬁcation

A DMN N must be trained with training and target patterns before it is used for
classiﬁcation. During training, network conﬁguration, synaptic weights, and values
representing the ‘on’ and ‘off’ thresholds of output neurons are determined. This
information is needed to construct the DMNN for testing or classifying patterns.

Training requires the following steps:

Step 1. Start with an initial network conﬁguration.

 

Step 2. Initialize synaptic weights to random values generated between 0.0 and 1.0.
Step 3. Apply training patterns and corresponding target patterns one by one.

Step 4. Once all the training patterns are applied, perform one pass of feedforward
operations and calculate a sum-squared error of the network. Then propagate

the error in a backward direction and modify the synaptic weights.

Step 5. Repeat step 4 iteratively until the network converges in such a way that the
sum-squared error reaches a predeﬁned error-threshold. If the network oscillates
or is stuck at a high local minimum in error surface, increase the number of
neurons in the hidden layers (and the number of hidden layers if necessary)‘

and repeat steps 2 to 5.

 

Step 6. Once the network reaches the predeﬁned error-threshold, check to see if all
the synaptic weights are valid, i.e., —1.0 S Wij S 1.0 for all i and j. If one

or more synaptic weights are not valid, increase the number of neurons in the

 

‘See reference [43] for detailed procedures suggested by some researchers.

106

hidden layers (and the number of hidden layers if necessary) and repeat steps

2 to 6.
Step 7. Save the ﬁnal synaptic weights and the network conﬁguration.

Our experimental experience suggests two layers (one hidden layer and one output
layer) are enough for binary classiﬁcation using the DMNN architecture. It has also
been observed, through extensive training with an arbitrary size DMNN, that there is
a relationship between the existence of solutions, equivalently low sum-squared error
states, and the ratio of an input range and an output range. 0.1 was found to work
well as the ratio as this always produced valid trained weights with the given input
pattern representations. 0.0 and 1.0 have been used to represent ‘0’ and ‘1’ binary
values respectively. 0.45 and 0.55 have been used as target values corresponding to
‘on’ and ‘off’ thresholds at the output neurons.

Once the training has been completed, the network conﬁguration and synaptic
weights obtained from the training are loaded into the DMN N coprocessor. Then
test patterns are applied to the network to see how well the network classiﬁes them.

Deﬁne the minimal network conﬁguration for solving a particular problem as the
network that converges to an error state equal to or below the predeﬁned error thresh-
old with a mimum number of neurons in the hidden layers and a minimum number of
hidden layers. A network conﬁguration can be represented by no x n1 x - - -, where no
and n1 denote the number of elements in an input pattern and the number of neurons
in the ﬁrst hidden layer, respectively.

In following two sections, some benchmark problems and character classiﬁcation
problems are applied and their performance is evaluated; only minimal network con-

ﬁgurations for these problems are illustrated.

107

6.3 Benchmark Problems

Two benchmark problems are applied to the DMN N coprocessor to test its ability
classifying linearly unseparable patterns (XOR problem) and to test whether or not
the DMNN can work as a data compressor (Encoding problem). In these testbench

problems, the training data and the testing data are the same.

6.3.1 DMNN XOR Problem Solver

The XOR problem is a typical example of 4 input patterns that can not be cor—
rectly classiﬁed by a single-layer Perceptron. Thus, it is often used to test whether or
not an artiﬁcial neural network can classify linearly unseparable data. Table 6.1 shows
the representation of the XOR problem in a DMNN. Figure 6.1 shows an example
of the two-layer DMN N for solving the XOR problem, where the values of synaptic
weights are set at one before quantization is applied.

Table 6.2 shows the actual outputs of the network illustrated in Figure 6.1 with
respect to the register length n. In Table 6.2, each ﬁgure is rounded to 3 decimal
places. As shown in Table 6.2, when n > 7, the DMNN can classify the 4 input

patterns correctly. However, when n S 6, the network misclassiﬁes one of them be-

Table 6.1. Representation of the XOR problem in a DMNN.

 

 

11 12 O

0.0 0.0 0.45
0.0 1.0 0.55
1.0 0.0 0.55
1.0 1.0 0.45

 

 

 

 

Output layer

Hidden layer

Input layer

Figure 6.1. An example DMNN for solving the XOR problem.

 

 

 

 

108

 

 

w50 = 0.443
w53 = 0.539
W54 = -0.684
W40 = -10‘4
W41 = 0.344
W42 = 0.348
W30 = 0.023
W31 = 0.915
W32 = 0.924

Table 6.2. Actual outputs of an n-bit two-layer DMNN for solving XOR problem.

 

 

 

 

 

 

 

 

 

 

 

 

I] 12 77210 72:9 7128 7227 n=6 n=5
0.0 0.0 0.451 0.456 0.461 0.468 0.469 0.500
0.0 1.0 0.549 0.556 0.563 0.563 0.688 0.563
1.0 0.0 0.549 0.556 0.570 0.578 0.719 0.563
1.0 1.0 0.457 0.449 0.445 0.484 0.625 0.438

 

 

 

109

cause errors on synaptic weights caused by quantization become too large. The min-

imal network conﬁguration for solving the XOR problem is 3 x 2 x 1.

6.3.2 DMNN Encoder

An encoding problem is used here to test whether or not the DMN N has the abil-
ity to compress data. An 8-to-3 encoding problem is considered here as an example.
Table 6.3 shows the representation of the problem in the DMNN.

Figure 6.2 shows an example of the two-layer DMNN for solving the 8-to—3 encod-
ing problem. The minimal network conﬁguration is 9 x 3 x 3. The DMNN successfully
encodes 8-bit data to 3-bit data when n 2 8, verifying that the DMNN is able to
compress data.

Data for network conﬁgurations, training and test patterns, and initial LF SR val-

ues for solving the XOR and the 8-to—3 encoding problems are listed in Appendix D.

Table 6.3. Representation of the 8-to—3 encoding problem in a DMNN.

 

11 I; 1;, I4 15 16 I7 18 01 02 03

 

1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.45 0.45 0.45
0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.55 0.45 0.45
0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.45 0.55 0.45
0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.55 0.55 0.45
0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.45 0.45 0.55
0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.55 0.45 0.55
0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.45 0.55 0.55
0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.55 0.55 0.55

 

 

 

 

 

 

W9’1 = - 0.0026
W9’2 = - 0.0018
W93 = 0.3277
169,4 = 0.3240
W95 = -0.0006
W9’6 = -0.0011
w9,7 = 0.2861
W93 = 0.2822
Wg’o = 0.0001

 

W10’1 = -0025
W102 = 0.3106
W103 = -0.0016
W10’4 = 0.3107
W10’5 = -0.0003
Wm, = 0.3032
W10’7 = -0.0007
“2,0,3 = 0.3022
W100 = 0.

110

W11’1 = -0.0442
Wm = -0.0201
W11’3 = 0.0149
”411,4 = -0.0001
W115 = 0.2971
W11,6 = 0.2867
W11,7 = 0.3223
W113 = 0.3129
W1”) = 0.0563

 

8."

W129 = -0.0413
W12’10 = 0.0207
W12,“ = 0.6153
“1120 = 0.4310

W13’9 = 0.5892
W13’10 = 0.0101
W13,11 = 0.0811

W13,0 = 0-

W14,9 = 0.0001
1914,10 = 0.5928
W14,“ = 0.0164
W14,0 = 0.4481

Figure 6.2 An example DMNN for solving the 8-to-3 encoding problem.

111

6 -4 DMNN Character Recognizer

Two experiments have been conducted involving the classiﬁcation of numeric char»

acters. The ﬁrst experiment involves classifying 5 digits from O to 4 while the second

experiment involves classifying 10 digits from 0 to 9.

6 -4.1 Data Set

For each experiment, a network is trained with two sets of training data. The ﬁrst

set consists of ideal digits (1 pattern per digit) and the second consists of the ﬁrst

 

data set plus 10% noisy patterns (3 patterns per digit). The test data set is composed
Of the ﬁrst data set, the second data set, and 20 % noisy patterns (3 patterns per
digit). P% noisy patterns are created by randomly inverting P% of the pixels in the
ideal digits. For experiment 1, each pattern consists of 6x4 pixels and for experiment
2 each pattern has 7x5 pixels. Figure 6.3 and 6.4 show the pixel images of a typical
data set used in experiments 1 and 2, respectively. The average Hamming distances

between ideal digits are 10.2 and 12.1 for experiment 1 and 2, respectively.

6.4.2 Experimental Results and Network Performance

Experimental results are compared to results from the program ‘annclass’ sim-
ulating ordinary multilayer neural networks built on the Rochester Connectionist
Simulator. A continuous sigmoid function is used as a neuron transfer function in

the comparison simulation. The purpose of experiment 1 is to ﬁnd the relationship

between network convergence and network conﬁguration.

 

 

mum
ﬂWEﬂM

 

112

 

 

 

 

 

 

 

 

 

 

 

 

 

 

(a) (b) (C)
digit “0” “1” “2n “3” “4”
“O” O 13 9 10 13
“1” 13 O 6 11 8
“2” 9 6 0 9 12

13
“3” 10 11 9 O 11
“4” 8 12 11 0
(d)

Figure 6.3. For S-digit classiﬁcation in experiment 1: (a) ideal digits; (b) typical 10% noisy
digits; (c) typical 20 % noisy digits; (d) Hamming distance between ideal digits.

 

 

113

 

 

 

 

 

 

 

 

llll
lFlT

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

H
mm.

(a) (b) (C) (a) (b) (C)

 

cﬁgh “O” “l” “2” “3" “4” “5” “6” “7" “8” “9”
“0” 0 17 11 10 19 11 9 9 5 11
“1” 17 0 10 15 20 14 16 18 16 20
“2” 11 10 0 9 20 12 12 14 8 l4
“3” 10 15 9 0 17 7 9 ll 7 9
“4” 19 20 20 17 0 14 16 18 18 14
“5” ll 14 12 7 14 0 4 12 8

 

 

 

 

 

 

 

 

 

“6” 9 16 12 9 16 4 0 12 8
“7” 9 18 14 ll 18 12 12 0 12
“8” 5 16 8 7 18 8 8 12 0 10

 

 

 

 

 

 

 

 

 

 

 

 

 

“9” 11 20 14 9 14 8 8 4 10 0

 

(d)

Figure 6.4. For lO-digit classiﬁcation in experiment 2: (a) ideal digits; (b) typical 10% noisy
digits; (c) typical 20 % noisy digits; (d) Hamming distance between ideal digits.

114

Table 6.4. Average number of iterations required for training in experiment 1.

 

 

 

 

 

DMNN ‘annclass’
Trained with a = 0.0 Trained with a = 0.5 Trained with a = 0.5
N1 N2 N0. of iterations N1 N; No. of iterations N1 N2 N0. of iterations
*4 0 2086 *4 0 1738 *5 0 13311
10 0 420 10 O 293 10 0 17212
30 0 332 30 0 182 15 0 11905
50 0 274 50 0 135 25 0 14108
6 4 12263 6 4 7482 8 4 79320
8 6 10446 8 6 4959 8 8 35480
10 8 7427 10 8 2972 10 8 44520
12 10 6135 12 10 1820 12 10 55025

 

 

 

 

 

 

 

 

 

 

 

 

Table 6.4 shows the average number of iterations required to converge with a given
network conﬁguration for the DMNN and ‘annclass’ when they are trained with the 5
ideal digits. Each ﬁgure in the table is the average value based on 10 different initial
sets of weights. For the DMNN, 1; (learning rate) = 0.2, a (momentum parameter)
= 0.5, and Erntd (error threshold) = 10‘4 are used, while 17 = 1.0, a = 0.5, and
Err-td = 10"2 are used for ‘annclass’. The use of smaller values of 17 and Err-td for
training the DMNN is affected by the constraints in its operation range. In Table
6.4, * indicates a minimal network conﬁguration, and N1 and N2 are the numbers of
neurons in the ﬁrst and second hidden layers, respectively.

As shown in Table 6.4, the number of iterations increases in general as the number
of hidden layers increases or the number of neurons in the hidden layer decreases in
DMNNs, but there is no clear relationship between the network conﬁguration and the

number of iterations in ‘annclass’. It is noted that the DMNN network can be train-

115

Table 6.5. Performance of the DMNN 5-digit recognizer.

 

Trained with the ideal digits

 

 

Resubstitution Misclassiﬁed error Misclassiﬁed error
error rate rate on 10% noisy data rate on 20% noisy data
8-bit DMNN 7.5% 32.2% 55.6%
9-bit DMNN 0.7% 11.7% 30%
C simulation 0% 8.5% 43%

 

 

 

 

 

 

Trained with the ideal digits plus 10% noisy data

 

 

Resubstitution Misclassiﬁed error
error rate rate on 20% noisy data
8-bit DMNN 5% 38.7%
9-bit DMNN 0% 26.7%
C simulation 0% 35.3%

 

 

 

 

 

 

ed with a signiﬁcantly reduced number of iterations compared to ‘annclass’, which
requires approximately 14000 (average) iterations for two-layer networks and 55000
for three-layer networks. The fast convergence obtained in the DMN N results from
the fact that the values of output neurons can take on intermediate values between 0
and 1 as targets.

Table 6.5 shows the performance results of the DMN N 5—digit recognizer with 8
and 9 bit register lengths. The results of a deterministic DMNN simulation (in C)
directly calculating the network operations are also given for comparison. The clas-
siﬁcation of test patterns is determined by the MSBs of the magnitude parts of the
numbers in the output neurons. If the output of a neuron in the output layer is greater

than or equal to 0.5, the neuron is considered to be ‘on’. If no neuron or more than

 

116

1 neuron are turned ‘on’, the test is considered a misclassiﬁcation. Typical synaptic
weights and network conﬁguration for 5-digit classiﬁcation are listed in Appendix D.

Each ﬁgure in Table 6.5 is based on 20 tests. For each test, new synaptic weights
are loaded. Resubstitution error is obtained by testing the DMNN with the training
data set. As shown in this table, the DMN N character recognizer with a 9-bit register
length produces results whose performance level is close to that of the determinis-
tic simulation on resubstitution data. It is noted that the performance of the 9-bit
DMNN is better than that of the deterministic simulation on 20% noisy data. The
reason is perhaps that the random noise may help enhance the network performance.
It is also shown that the DMNN classiﬁes patterns better when the training data
contain noisy data. Meanwhile, when the register length is less than 8 bits, the mis—
classiﬁcation error rate is more than 50%, even on resubstitution tests.

For experiment 2, the 5-digit classiﬁcation problem is extended to a 10-digit clas-
siﬁcation problem and the pixel images of each pattern are increased to 7X5. The
network performance is again compared with that of the deterministic DMNN sim-
ulation and ‘annclass’ in terms of a correct classiﬁcation rate. Table 6.6 shows the
performance results of the DMNN 10-digit recognizer when the register length is 8,
9, or 10. For training, 17 = 0.05, a = 0.0, and Err_td = 0.005 are used. Only two-
layer networks (one hidden layer and one output layer) are considered here. For the
DMNN, the minimal network conﬁgurations are 36 x 9 x 10 and 36 X 30 x 10 when
it is trained with the ﬁrst set of training data and the second set of training data,
respectively. The number of neurons in the hidden layer for ‘annclass’ ranges from
5 to 20 for both. The ﬁrst three rows show the results obtained from the DMNN
coprocessor, followed by the results of the deterministic simulation and data from
‘annclass’. An example of synaptic weights and network conﬁguration for 10-digit

classiﬁcation is listed in Appendix D.

 

117

Table 6.6. Performance of the DMNN 10-digit recognizer.

 

Trained with the ideal digits on the 36x9x10 DMN N

 

 

 

 

 

 

 

 

 

 

Resubstitution Misclassiﬁed error Misclassified error
error rate rate on 10% noisy data rate on 20% noisy data
8-bit DMNN 2.2% 16.2% 43.3%
9-bit DMNN 0.06% 9.7% 36.6%
10-bit DMNN 0.94% 7.8% 37.1%
C simulation 0.0% 6.7% 35.0%
annclass 0.0% 15.8% 30.8%
Trained with the ideal digits plus 10% noisy data on the 36x30x10 DMNN
Resubstitution Misclassiﬁed error
error rate rate on 20% noisy data
8-bit DMNN 7.6% 25.4%
9-bit DMNN 0.9% 20.3%
10-bit DMNN 0.4% 15.1%
C simulation 0.0% 13.5%
annclass 0.0% 16.7%

 

 

 

 

 

 

Each entry in Table 6.6 is based on extensive tests (> 20,000 runs). Twenty sets of
weights are used and more than 1000 different sets of initial LFSR values per set of
weights are used. Correct classiﬁcation is determined by the output neuron with a
maximum value and the test is considered a misclassiﬁcation when an incorrect neuron
has a maximum value. As shown in Table 6.6, the DMNN character recognizer with
a 9-bit register length produces results whose performance level is close to that of
the deterministic simulation and competitive to that of the ‘annclass’. The correct

classiﬁcation rates obtained from the DMNN are close to those for the deterministic

118

simulation, which indicates the quantization does not have signiﬁcant effect on the
network performance when the register length is greater than 8. Better performance
from the DMNN than average values shown in Table 6.6 can be obtained by choosing

a set of initial values of LFSR’S carefully for a given set of weights.

6.5 Summary

Digital multilyer neural networks for binary classiﬁcation problems have been
modeled and simulated in VHDL. Any size feedforward neural network can be con-
structed by simply interconnecting the desired number of predeﬁned modules. All
operations in the same layer are performed in parallel and all operations between the
layers are performed in a pipelined manner. Thus, the DMNN architecture utilizes
full parallelism embedded in ANN computations. Its processing speed depends only
on the clock frequency and the register length, not on the network size. When an

n-bit register length and W MHz clock-cycle are used for the DMN N character rec-

leO8
2"-1

 

ognizer, the processing speed is about patterns per second.

The DMNN architecture is also extremely compact. The chip areas required by
10-digit DMNNs are less than 1.0 x 105/19 and 3.0 x 105/lg for 36 x 9 x 10 and 36 x
30 x 10 network conﬁgurations, respectively, as was determined in section 4.5. The
classiﬁcation performance of two-layer DMNNs for 5-digit and 10-digit classiﬁcations

is found to be competitive to that of deterministic DMNN simulations and ordinary

back-propagation networks when the register length is greater than 7.

 

 

CHAPTER 7

Conclusion

 

A new architecture for a VLSI-based digital multilayer neural network has been
developed in this dissertation. Statistical models for stochastic computations utilized
in this architecture have been developed. Network analysis has been performed based
on the statistical models. VHDL models of DMNN pattern classiﬁers indicate that the
classiﬁcation performance of this architecture is competitive to that of deterministic
simulations or ordinary back-propagation networks while retaining the desirable prop-
erties of high speed and high density on a chip. This chapter begins with a summary
of the research work, followed by an identiﬁcation of the major contributions of this

work. A discussion offuture research issues concludes the chapter.

7. 1 Summary

The development of fast, space-efﬁcient, and programmable architectures for
dedicated VLSI ANNs which are applicable to real engineering problems is a current
challenge in ANN research. Analog VLSI ANNs have the potential for high speed

and high density on a chip. However, the unavailability of reliable permanent analog

119

120

storage devices makes it difﬁcult to build programmable analog ANNs and design
parameter variations, such as noise and temperature, diminish the accuracy in analog
computations. Furthermore, high parasitic capacitances on external I/O pins and
difﬁculties in designing large analog systems make it difﬁcult to build large size or
multi-chip ANNs. A conventional digital approach using ALU units or special hard-
ware as computing elements leads to increased hardware requirements and decreased
parallelism. A promising solution to these problems has been demonstrated in this
research work.

A pulse-mode digital multilayer neural network (DMNN) architecture, which can
be implemented using simple logic gates and simplistic digital components, has been
developed. The DMNN performs pseudo-analog computations using stochastic com-
puting techniques and random pulse streams. The modular DMNN architecture is
programmable and compact in size. Any size feedforward neural network can be built
by interconnecting the desired number of modules.

In the DMNN, all operations in the same layer are performed in parallel and
all operations between the layers are performed in pipeline. Thus full parallelism is
utilized. Processing speed depends only on the clock frequency and register length,
not on the network size. When the network is used for pattern classiﬁcation, about
200,000 patterns per second can be classiﬁed with a 50 MHz clock assuming an 8-bit
register length.

Statistical models of operations occurring in the DMNN have also been developed.
The variance of the random noise (or error) resulting from the estimation of the gener-
ating probabilities has been shown to be bounded to the binomial distribution which
can be obtained when Bernoulli sequences are assumed. Network analysis for binary
classiﬁcation has been performed. Experimental results show that the successful re—
covery rate of target patterns for training data closely matches what was anticipated

by the analytical results. It is also shown that the low variances in estimated compu-

121

tational results in the DMNN lead to the high success rate in the recovery of target
patterns.

The DMNN has been applied successfully to binary classiﬁcation problems such as
XOR, encoding, and character classiﬁcation. Two-layer DMNNs for these problems
have been simulated in VHDL. Their performance has been found to be competitive to
that obtained from deterministic DMNN simulations and ordinary back-propagation
networks when the register length is greater than 7.

The hardware requirement for implementing DMNNS is signiﬁcantly reduced com-
pared to other digital approaches. For example, the 8, 9, and 10-bit DMNNs, whose
network conﬁguration is 36 x 30 x10, can be built using about 2.27 x 105, 2.53 x 105,
and 2.80 X 105 NAND gates, respectively. This indicates that a 10-bit register length

DMNN for 10-digit recognition can be built on a single chip.

7 .2 Contributions

The major contributions of this research are:

1. A VLSI-based pulse-mode digital multilayer neural network (DMNN) architec-
ture has been developed. Stochastic computing techniques and use of simple
logic gates as computing elements lead to a space-efﬁcient network architec-
ture. The DMNN is programmable so that many network conﬁgurations can
be formed and its modular architecture makes the networks expandable to any

size.

2. Statistical models for estimated probabilities resulting from network operations
have been developed using the statistical dependence of pulse occurrence in a

pseudo-random pulse stream generated from a tapped LFSR and the determin-

122

istic nature observed in gate operations. The random noise effects on network

operations have been quantiﬁed.

3. VHDL has been used as a modeling tool for a DMNN coprocessor. The 11i-
erarchical design capability supported by VHDL, combined with the natural

modularity of the DMNN architecture makes design analysis simple.

4. Character classiﬁcation problems have been applied to this model and it has
been shown that the classiﬁcation performance of DMNN 5-digit and lO-digit
classiﬁers is competitive to that obtained from deterministic DMNN simulations
and ordinary back-propagation networks while retaining the desired high speed

and high hardware density.

7 .3 Future Research

The DMNN architecture has been developed in a mixed mode using gate-level
and register-transfer level designs, but it can be fully described at the gate-level. To
implement the DMNN architecture as a dedicated VLSI ANN, two approaches are
possible: custom-design and semicustom-design. The DMNN with a minimal network
conﬁguration, dedicated to a particular application problem, can be built on a single
chip using a custom-design methodology. But, a semicustom-design method is more
appropriate when modularity or expandability is a more important factor than size.
Any size network can be built using the deﬁned modules. These modules can be
implemented using gate arrays and/or standard cells.

There is no reason that feedback networks, such as the Hopﬁeld network or the
Boltzmann machine, can not be constructed using the modules developed in this

work. However, in order for the modules to be used for feedback networks, statistical

123

models for analizing the random noise effects caused by stochastic computations on
network dynamics must be developed.

Finally, DMNN binary classiﬁers described in this thesis may be further used for
solving more complex pattern classiﬁcation problems. In an n-bit DMNN, 2" — 1
levels are available to represent the values of elements in input and output patterns.
Thus, each multi-level input pattern can contain more information or can represent a
more complex pattern. Furthermore, multi—level output neurons can make it possible
to represent multiple categories in an encoded form using fewer output neurons. A
simple example of such was illustrated in the 8-to-3 encoder in Chapter 6. However,
in order to make use of this advantage, one condition must be satisﬁed. That is,
networks with given input and output data representations must converge during
training to an equilibrium point in the error surface which is equal to or below a

desired error threshold.

 

 

APPENDICES

APPENDIX A

Derivation of the rth Factorial

Moment of a Hypergeometric

Random Variable X

The probability function for a hypergeometric random variable X is

PM, 2 1..) = (vs/3731!)

where l is a natural number (i.e., l 6 0,1,2,..., N — 1, N).
Let (k), be the product of r consecutive integers starting with k such that (k), =

k(k — l) - - - (k — r + 1). Then, the rth factorial moment is

N
E[(X),] = :(k)TP(X=l)

k=r
: inn—(0)531). (A.l)

By using the identity

 

125
we can obtain

N N

Sauna-.5) = 3042:9635)

k=r kzr
N l (N )(1 l
: (llr 2;)(j—T)((7.__r3_—j 4 l
J:

N (l—r)((N—r)-—(l—r))

 

= (0.63:); ’ (£117
: (”Tail—:1.)
n T N

Thus, substituting equation A.2 for equation A.1 gives

 

 

APPENDIX B

Program for DMN N Back-Propagation
Training

 

 

a ﬁtmenaaoc:
r 0:250: 0:. 8:. t

A
”8 088
829888 is ...&Vm.m§vw
Ar...“ ”o:l:080:v 6.1.98
Ti 6:..808Aamx 6888
x :80:
”930... 3.30 :08: 8.88 .888
a 382i... .asieiwmww
A 85880888 is .8153:
8 Ti no:l:8=0:v x188
Ar; ”8:808me 5888
n :80:
”990... 3.80 :08: 8.80%.888
a Edits} $818888?
”8:830 was .858
”1.....88 8......“
W 1:28: He 68% Ramses: 80388
”AMOS HOHﬁ «FH05 :VEQW
x..uu0o§:0_om 880 m8 88:8..Wt:ta
”0880 .. 8. .808
ﬁlamugﬁw 88:_...Wb:tq
” 80 3.88 . 880
x .uumww 88:8..1888
8.133 1.. 88.8 @058
. 08.8
a 82:88.58 yes ...ev_u§e
8 AT; 6:188:05. ”ono:8
A++_ 6:1:88m:¥ 6888
x 880:
”980... 3.88 :08: 8.88 .888
C 382nu8... sanitamamwwwﬁ
8% :o v 8:: 8 0:: 80888 8 8:08:80 0 #:8088888
:3 o as: I 005:

 

”00.51608 $081800— :80
8036888203888 8 a

”23.8 880
no :0 88
8906308808

P. ESMOE 552 *\

”mama... .58
"Ho: 8080:
.o:l:o80:§:m_03I30:.Ho:l:o80:=o:l:§3&_au:8 888
30:88:05 838080381808 3:88:05 3:18083mx0d0: 858
”88883880880.3:88:0578:8080 3088 888
”3:88:05
HEI:080:§:303.78:88:05385080380888 :88
608 .o:|%_:.€:l:0805HSI:o805:8J0:.onwacl:ou 88

r. 8888.0 058? 8838 t

sebum g I 0:803
o— :0 8 8: 0:809:
o 0.581% 0:803

8 5.8-2 2:08
mm 808218 0:808»

N 8J0?— 08.802"

cm 8:580: 0:808“
9. 8150qu 0:803
vm 00888 0:803.
88:83 0855*
88020qu 8805*
A8289 08.68%

r.
80888850 M550: 08 0080. 8 88:0: 80:3

088.803 8088 538 E088 8 8 800180: 22.29
”0:: 888800
.8880 8803 :8 0200 8 80: 8.8080 .m 83:02

380: Sham—:8 .0.ow 802 :5 wﬁﬁab 858: .0 8500—
580888 .083 05 003886 .2229 @080: .88on 8F .1

 

 

126

 

 

 

AAEE0::_::::I80:
55883:.9:::._A..o-.s-as-A.:Essa-es
moduASEeﬁsé .
A A.1. A o:.-:o:=0:v.A ”on. :8
no:u::::_A::H:0:U:0:
5:85:70 :0: E0:
: A++: A o:l:o:=0:V: 58:8
AABAG—8:08:88":580888880:
.I. Ao:l:o:=0:v :A I : :o
A J A++x: o:I::0:::0mWI:A: AWEAwB:
AER-8.805 Ao:l::0::a&::l:0:l80: :08:
A3:88:05Ao:l::0::a&x0l:0:l80:.Ao:-.:o::05::0::0:8|80:_:8=
”8.5:. .: :8

081803838
A. 22:: 0:: 5 8080:: 8.68.8: -:

”A882“:
A 82-...8-55838828
.:-.A-o:|:o::0:.:...:m.§ "A3: A a: 580899.888
A-. A : .I-An:0.Aa—I:=ouo:o
”Ag-.6080: 3.9.: u :38: E088 .88.:
:A++:Ao:l::0:::v:n new-Mm
HA..Illl ma—H—WQWH it! : be.
5:: 3.38:8:
. .3582:
82:80:03: “as: ...::v 8:
A++.A no: :880:v. ”A188
A++: 6:88:05: 5888
. a s x was...
A > :0... £3: : 38:0. .:8:
A 382uuAA..:.. .>0s++.:=0:o:u::::A:8_0

\. 0.5.50: 0:: :8: L

A A
AA:-A-o:l:o::0::::8::0:o:

 

:

 

is: 558.52: "Ts-Ase 8820:...ome8nu
"Ede: 85053880:
.:-.:.8-=e8:.:..:m.s "was 8.: $8889.88:
A--. A : "A. .:0.A8I::ono:o
"A: + : ...e..: u .088 508: .88:
”A: + : .8: u 5:88 508: 88::
: Enos-508 Sues:
”803-82:
8.: u 88> :82: 5:3 98:03 :o 5:828:88:
”80368.6:
.8: u 82.; :82: 5:3 38:03 8 8:52.88:

A
A x5:§@§..:ns ...-:::::::::
2+8?» 8:988:95 8%:8:
8.7832888; AQREEE :03“:
: A++. no: :080:V.A ”on. :8
: A.1: Ao:l:o::0:v: ”on-5:8
”enema-8:08:
. A A: 580:
Abba-Imam? 05::08: :.:«0..v:::8:
2882193.. .25: 8: :58 :03..:5:o:n~:::::_
5---- 80:03 8:: ..... 1:8:
AA0A0A0...::: A-.-- 88:80:: .8 598:: 3:0,:- ----- ..W:::::A:
A: u wag-:8 AooooomnII0_0.Ao8:

A nEAaaﬁsauaguszzﬁsa
A++.A 6:I:8:0:v_. ”ouo:8
A A.1: 6:88:05: Aou:v:o
”A0_o:o...A-----§ A: 8:22 .08:
”A2881 ..... B: : 88:8... 8.8::
I: ”AB-:wee 23.3:
::+0:0.A0u0_0 A0
”SA:Ae:w:03-38u53.8803
MIA Ao:l:o80:v.A Ac". :8
: .1: 6:188:05: no": :8
”0:91:88 08:
A 8635880:
: Wag-83223
5:03:95
Aer-0:98

 

 

127

 

 

 

 

A AI: : Eggs-M: v1: anus:
.:o «A A +AAu:A:A
coho-.838: + :oboIE=m:chBo:n:o5IE=m: mIBQ:
Ao.~\:o::oIE=m:A:w "8:818:88:
I I A
”E:o::o...5:o::o+:o::o 83:8: u:o::o 83:8:
A5:o::o+:o::o-.B: "BEA-8: @on
”5:88-88:13: "Stung: Ao.ovS:o::oA.A:A
”Erma;- ssmu 35:? «:8 2:
”AAA . :w? are: 2: I
8:5» + 2:. 5:883 32: "MA: 82863 32:
”Egg-288:. mamas-388:. mag-3::

- A
28:: 5.35:? «:8
aim-:8

- 38:: A

I ASA—axe :08... I
ASE? :8 - o.::-Am85w§uaauwa 3:8

IAEA. :w? «:08... I
A:AA5:2:5:o:-AAAA55 :28. Eco-souaaaa «:2:

A
”AA :A: 858:
EAAAAw:::A¥+o.:A*AAAAm:L3-:::%u.EA=::w3-§%
A: u: U. a. a. o.o-v 3:23:83:
A Ti. 6:: 8:55: ”ouxA :8
0on

A
”8:35-3:68
AB: :32: - o8 Aeouoﬁongauwausg
:2: cm...- sg-TAaEq-ga- -
AA. A55 :25: LEAVE-3385:? sAa:
” A AA: 858:...
A::A_Aa=::oa-:.:A*A:mAMAA:La-ssuuaaswa-sg
A: "A :A a. a e: um 3:58:35:
A A++:A no: 858v 538:8
A 6.: "3: Aamrwawséz
“A 0880
A 83550-5553:
”o.:uA:AE::w3-§%

A

A

A A++n 2:88:05: EHAA:8
A5A585:o:-A_AEw&:m:uAAA:oE
A .38: 858$.
A A185 «3:388:85:
6.3858858:
£93388:
uodnAAASE
A++A 6:88:05: AonAA:8
A A1: 5:188:88: "5:8
5.8":05I88: mIBoa
nod-hounds: :A+:::ol:o:Au:8I:oA:A
A nod-A93? 3:2: .129? .22: 2o
”odusaaa as: ”559: EL. 33:83 38
A A++A AoalcoSochncno:8
A A++: ”oclaoSoavgnAAS:
”owﬁnoIommIgo: 49:88:55AoaucoSoEEBISAonlvo :88
AodﬂoEIEaEmIEO . 88355388505 383428 :88
:oboIE=::A:m.:o::oIE=m: mlko:.:o::olmna.38850588: :88
”338505388505 walazou :88
”38850533850: :wBISAoA: :88
55.8.58: :5
68:04:: :5 8:8: A

AEoIanoExoB
A... Z 235 2A: a: 83808 aoumwamoaioam ...\

A

A "A:sszA-IEHEQWA32:02 A
”858:5 :8 5:85:70 :2:
8:482:85 eBuAEAéSSB:

A.1-A: 8:: 8:35: ”ended:
"AA:A:_=A-:o=-ee-o.:A-EExnw-scéoﬁaeaﬁamao:
”SEAS::a-ao::.:uEA:§ :2: so:
”525 :8 so:-o.:uEA:A5-§-§A::

"ASEuxchAIEo: ...

5:58:38. AA...AAAEAEIscuesuAAAEASIER-EM“ o
A

 

 

 

128

 

 

 

 

535m A» Gmmmobxoﬁoﬁu FEB
"A + Eu SanEo 52
£32 + 9mm m8: :5: n im: 0on
608 EA u :8: snag 55:
Auxucdqulﬁu E can; A
Anaemia—AA EA
ABauaAvoﬁia :A:
I A» 588
. AAAVQE 925A _. Amos to Anne u A
on; QZ<M+xux @0865 Enx
A» :3: R 2: A

028W; :3:
r. couabcow .382. 8853— t

A A
A A
AAAAAAAAsﬁsaédnAAA. "A333. 3.25
n SAWS: an A»; 86
”028m gmuaasﬁsa AAuu . £838:
A A++_. ”onlcosgvm Aoumwé
wmAAo
AAAAAAAAEwAua..A.A.;Am§ "A? AA§§=V€E
no. A. as: A03 as
5.3 32%; Ar"?
A A++_.Ao=1=o§o:v.~ ”cum A8
A 2 - 81:83: nu A __ Ham—IEVAA:
A A++A ”cannon—EVA ”ensue
AQUA—manna? soc easy.
”BEA
osmium?

A... 3&6? 325 SA 52228» 885m .1
A SEIEAAESIBB. "Ebola . . a:
A A We» L. evwaamw

”Coronasevmds : £3» "85 mm
"CouolmnaﬁAAm ﬂotatBﬁ..&Abcc&

 

5cbul3qhﬁ ﬂotatgypbca

n AAA H A8 3m. A» Agnew:
”Auwac =8 CoboJSukuobo EamuchBAo: a:

AAAAAQwansAoPu ESE: «:8 2o

”E 2&3 3% 29.. u
«Es—«$2: $333 Bonus: 8:303 Bo:
”AAASAwBISAoEAAAA. snonauaoauAAAA. £30313»:
”ﬁes 5.135%; £8 A
2.58

233353 ”ﬁes A
2:53:38 ...5Ea££§§mﬁa53.2%

A
”AA AA 3228
EAAAsAAons+odLAAAwAmausscnwAAAA—Awausac
AA "A U. a a Qouv 852%; a
A A++x ”o: =8:on "ouuAA 8
86

A
”E 2335.8 2:53:33...
AAA:A=A=Au§-o.v*AssufsouAAAAQwausAoc

A
”AA :2 5:38
AAAAAAsaona-o.c*AAAAWAA: BISAoouAAAS “was?
AA "A A a a o.ounA 3:. 23an
A Ti. 6c =88:va :5qu
:3 a?» 852 53:
”A 88
A 2:580:85 @235
5.335%; «:8
A Ti 6: :83: VA ACHAASA
ASQSHobonEBS

AA:EsgwsfsAauwausaissﬁusss A

0on
”53.3569
S35:33...EAAAAwBJﬁESBEuA€28
9.3 3 362088:
A Ti. :99: S932 vx Aiuxre

A

 

 

129

APPENDIX C

VHDL Code and Corresponding Schematics

 

 

 

 

 

AoAAAa>IAAAAuAAa>IAAAA
Ewen
ABAAAAIAAAAAAQAA ”333.3 05:2?
”AAAAAAAAAAA A3312: 05:3;
3 oAAAAAIAEAAoAA 588 283:: nonﬁ>IAAAAAoAAAAAIoAIAAAA 83:3

65
AE>IAAAA EBB
aooA AAAAo
AAA 28
AAA-AA:N+A§..A:AuHA§..A:A :2: A.A.uAAA::A:AAAA
:8A A 2:38 AA - 33.5 :A A 5:
:Awun
AonAASAAAaAA AE>IAAAA 033:?
3 A28? EBB AQAAAAAIAAASQAAAAAQAAAAA§JSAA 5:83

65
38:38 EBB
38:82 32 use
AAAAIAAooA Ex: 95
AAA 95
A.A.uAAAAA8uBAo :2: A.A.nAoAAAA:AA:AAAA
82 8:50:12 8 A AAA .A A8
AAAAIaooA
98A 5:5ch 9 A AAA A A8
”AsolaooA
Swen
AAoA8>IAAAAIASAAo= ”50:85 05:93
:A A8
-8>IAAAAIA:SoAA EBB AxAbaAAAIAAAAIAAAAAAoAAAAAAAAAAAAAOIBAAB 53:8

6::
3:363:03 Eae
AqooAuco
”@25qu :5 802:3qu 5: AAAAJAQAAAABQAAAAAAJABAA:8A
:8. A 2:38 AA - aAguzA :A A .8
Emma
”bug: AAAAAJAQBAABA 2%.53
wA AB 532 8391352: ”25le .oAAAAAAAIwEAAAAAAIyAmnAuA :oAAoAAa

 

mA Moaalacnﬁ :63 $863
3894886 95

AoSnIREQ: EBB 5053815333848: 53:3

:8: 838 AoAAAAAIAAAAAAo:AoAAAAAIEAASWoAIuSAA .8525

6391330: 832 283:: AoAAAa>IAAAAAoAASIoAIAAAA A5583

”Egan: 588 AoAAAAAIASAAoAAAAAAAAAAAAAgloAAAAA A8383
AAvoo>IAAAAIAAAAAAoAA

8 x5: IAI 5095:: I AA :0
”AAA. 838 AoANAAIAangAA. ”by 532:? EAAAA nAWAomnAgA :oﬂuoﬁ

”35:35:

Ac AEoAAaAAIZ 2 A owed: AAAABAAAAAAAAE: mA anﬁlabAAaq 25
”Qualﬁao:

Ao £85812 9 A owﬁa anguish: 35581350: RS
Signage: Ac

AoAAIAAAAAAIAao 8 A 35: AAAAAAAAAAAAAAAAHA: 3 3583550 ab
”23435:

Ao AcoBoAAIZ 9 A owns: ﬁnancing AAA @5433: 2A.:
AAoA8>IAAAAIAAAAA§A Ac

955812 9 A owﬁa gaging mA anﬁlAAaIRSoAA 2A.:

AAA :38 :oAAAoAAuszAOHUgIPAm AAA A28>IAAAAI=2=2A 252::

”AA 9:38 sApnzAmBomA/ntm AA mining: 335::
AmuAASBAAAA AcclAAAAAAIAso A5338

AAVuAAAAABAE AoAAIAAAAAAIAAoAAoE AAAAAAmAAoo

€~ua€3~= AoAAIAAAAAAIE A5328

AmmuAAAAABAE AEBAAAAAI Z 6388

AmuAAﬁAAAaAA ”onaAlz A5388

AvMuAAaAAAAmAA AAAOAAAQAAIZ A5338

AmuAAAAAAAAAAAA AﬁAAAIZ A5388

AomuAAgAAAwAAAAvotoAAJAovo A5338

mA :AomAAISAAAAAA owaxoaq

$8.3: 455 3.0

AA:AA:A52 222: SA 350 AB; 5

 

 

130

 

 

388x88 :8
55 8: A8188.830:.:oI:AAo.EI8:H:8Au8:
:A: 9 8818285888
283888 88888
388888 :8
AAAAaualasmlAAAo A8 8:108:18:
888488: :A 83:83
£88428: 8 8:88:38:
8: :A 8:88.8885:818:861888218588
888:: 88888
88888 :8
”AAA: 8: ”88:30:
:8 8 38:83:08
80:88 88888
88888 :8
53 8o ”Bola:
us :A 8938:
:88 88888

8 8:888:188: A: 8858: :88an

:88888l88: :8
:A 8:888:88: 3:8
23.888838? 8:

888.8: 222: 3.:
668188: :8

8::
8238»: :88:
AAA :8

A.A.HAAAAEAIQSAA 85 8A HA GAAAAIZ... *O.N\O.AA\A:>IA8:A.AA

A82 :8
:88
no uHAAAA§u2AAAA 8:
”88882528.: - :>|:o:u8>u.:8
I A.A.uHAAAAguoS:
:2: AA: "A 8.95 2:585:83 :8:

 

89 A 9:38 AA - 35.5 :A A :9
A: :8
AAAA.A-A...A§nA:m:uHA§M8:
A.A.uHA9A: 28319::
:2: A883 8::
A8.— EHAAA; A8:
:88
”8:88: 859:: 28.83
AoduAAAwu: AA:> A8: 2883
:A 8:488: :88: A8:AA:8I:AA8.AAAI8IA8: 88:8
8::
AA:>IA8.A :88:
GA :8
AA:.A-A*:>uA§nA:>uA§ :2: A.A .uAaA:uzA9.A:u:A:A

AAA :8
Ao.Au”A:>..A§ :2: 2.882383:
Aa8A :8
A: :8
A A+ AI:8nA AJ8
AAAA - 8:182:65.A+A§IA§uAA§nm2
:2: A.A .uAAAoi: :AA:
89 A 9:38 AA - 93.2: :A A2:
:88

88:8: “Ala: 28:?
AodnAAao: A948: 28.8?
:A A8: :88: 6:31:88:88:1:AAA8WBIQSAA 88:8

8::
28:2»: :88:
A82 :8
A: :8
6 "883108 8:
AAAAAAINA - A:>IA:AuAA:bIA:A
A.A.uHAAAA? :9»:
:2: AA "A 27:18:: 8:
89 A 9:38 AA - 8:17: :A A :9

 

 

131

 

 

::0:00>::B:380:
8:88:35 8:88:88 8:08:80 3:w3
”888380: 898:8: 8:88:80: 3:3:
8338:380: ”8:88:83 3:9:
”:3 H .0 383
m :
”888380: ::0 ”::&:0::0::A:::w: 800 :5

”:5 ::0 8:08:898
:0SDWM:80: 8 ”80:88:88
:0 @4380: 8 ”0:33:88
”:8 8 80:80::
50:80: 88:80: .::::_::0:::_ .:::8::0::x0v:8:
888080: 808.800
3:80:88 :8
”:3 ::0 ”::0::0:::: .::0::0:::A0
”:3 8 8:08:88:
”0:3:380: 8 ”8:88:83 83038»:
”:5 8 ”80:8: 82:83:85:
0:98»: 808888

:A :0::::8: .8 8:38: 08:00:28:

”0:::8:0:.:3::::::83A:83.::: 0::
”0.88:8: 0::

30::88: :8
:A>:8:0:::::::0 ::0 :::0:0::::::0:
8338:380: 8 ”8:830?
$83:380: 8 “80:38:30:
”:5 :: 60:89:80: 6335::0: .:=0::0: .::0_::0:
8o:
:: :0::::8: >85

”ﬁg—08:88:28; 0::
38.502 NZZEG mad

”888:0: :5
uA:3:803:58w3:::0:::::30: .w3:0::::::0:
.m3:3::88.w3::8:83b 9:8 :8: 32m ”xqmli
"33:03::58:3:30:30:
.w3:80:3::30:.w3::8:30: .w3:::0l:0:
.w3::=0 83:82:85 :38 :8: :0::::: 03:30:88—

 

”R3:::0:88.w3::5::0:
.w3:::0::=0.m3:3::88 .w3:::0A::0:
8388:3888: :38 :8: 88:88:00 35:88:00
”83:89.8: 83:8: :38 :8: 80:53 85:88

"3:38 8:83.on 8:. 8: :8: ”8:8
3: m 5:3 .A. uv :8

”AA-::_::z:A:::::3:::8:uvAas::38
”m:-::EHZWMvww3H03::H:0:HVM:WEw=H::0
. 783 Z m 3: 03:: :0:uv m 8w: ::0
AA95:86:_::::s:::o:uv€:::=::8
AA83:283823383:8:58
:38

823882883 3:8 8: 25: "A: 8:
5032085888303 >85 0:: :0::::: ”A3 8:
60380888888038? 8:5 0:: 88:88:00 :3 8:
uA8:I::0IQ0::V::0:8::£83 5:8 0:: 80:83 :3 8:
”C0A>::0:::=0§00A0.:_83 >88 0:: 80:0 :3 8:

8:38:88: :0:S:::0::88 3:3:
”:5 83:35:88 383
:5 83:53:88 3:8:
”88:08::80 83:03:88: 3:3:
”5:38:380: ”8:880:88: 383
”88:380: 8388:3830: 38 3
roman ”8:68:30: 383
51:: 83 :5:on 3:8:
”bung: 83:80::0: 3:3:
51:: 8363030: 383
6.5:: ”:88 8::
”bugs ”:8 3:3:
”35:80 ”833:0 3:3:
”:5 .8 A0:::A:::::0 0: A 088 38:35:83 :: ::E:::0 0%:
80:0:800 :5
5:88:88: ::0 80:88:03
”88:380: ::0 ”8:38:30:
i::3:03:::::0 8 ”8:033
”:8 8 ”38.033583580:
E 808888

 

 

132

 

 

80580198$1 59:0 888%
Any—81859188053885.1586 8 ~ _ +8 5511:: 5 .a .8 1
”>135? mmm<2>m O
888% A81855152815551535.6888:a
8 A_+o=1~5=1=8¢5+81~5=15v 5 x 88 883515850

#8315qu 888% can
u s ole. §m1 o. 151
28;. hozwmrﬁvocﬁawbmpimﬂﬁﬁﬁwo.pmwr
.88185315.351353185885 tom .6315. 581
. .8581 B1 1 :o 81 A 8 1&8580 mom :85:
”M o5817i~mw8mthmwoaoawvwgcmﬁu 15$
.Aao581zv851550358.3818858. on 0188.8 1
”8 8.8 5885:. 2
8858188281: 228% 8»
n _ 815. _ 51x0
. 658 Ampwwqoxmvmmwbwaﬁgm
.85 515 53.50.38 85 55. 88 05.8.3
80580185m1=8ci
088% 8155.15 8 5 5 :8 S<M~21mmm<2>m1m
888% 8:18:15an
$5515 9 2:818:15 E .38 8.531585
8831555 888% 80

n2839818.8:58:88 518188
.8888138688818856.88188
.85185315.8818531585 805 331858
”EQEoBQnomIEQE

” 5 5 .815. 5 c .815. 5 535155. 5 35158188
2 X v .axmwﬁwﬁwanB.Axwoﬁaoﬂocvqrw. 8mm 088.3

80580 maimdaﬁ
888% 8155.15 8 5 5 5 88 8831582

6885 88

”885588

”5: 2: 88 58»

”goo—cco

”moo. 80
na282ml:8812159285515.15”
5855815 5882582
”AS 51551235688688

 

noo— 5058 Z8 _ 5 _. 88
58 55.812 8 2+8 55.15% 5 _ .8
H o8 85
”908 88
5 oAuse:85.131250885183.:
5 88
”$885515
1 050
58855 5 £582.88
Amu— .51561Bo56a865x8
caucus .
moo 585812 8 _ 5 88
o8 8155815 8 m 5 m 88
5 oo— 85
”8551:8888? 5515.188
x _ :8b15 . 886.58
x .5 :89 8588638
58 805812 8 5 5 _ 88
”A..:V .oaﬁm 3”: "u 3 ”A....v .CZHEW 3”: "H MA
Ewen

Snag: 885138 82553 ”585185 ”55158855,
£8318.“ ”55158853 5.58.1358 ”559315828?
$88 .858 6815 8253, £23.88 ”3 .3 82553
rungs—188133.. 5 5 Cam—H.638 H515832 85
{351581553 5 5 Cam—H888 ”5158.512 85

1 $88 LE
.8 88:8 2 8 _ %=8 35555.35 5 558. 85 85

8%85 .8 883812 8 _ %=8 888K81a 5 85185 8.3
8085 ”073155

1 1 1 5183 80
”Co 580 853 303%uv8 85B 5
501on51853 Buapwuv5108531xo

3:0 88 88 8880188 885% "V50
1 51mm:
3.3580 888.082 53 582

1 =wa
.. 55.81885

.68 5558.553 .580 8: .88. 8058 5a 88
”28581585588in83 base 08 088.? 5a 88

”588: 521858 8015 .813 Rama

 

 

133

 

 

 

 

00000:: ”—05:00Imza

”00000:: 0:0
" 20:0
”anagram
”m: cm :05. .. "3:800:96:
:05 00.304" 54:80.
3 + :0 : "0:06.:
”050005688 05.10 =0 as 0:3
:30:
using: 0:060: 050.5; I
00000:: ”—00:00 80:0:
”00000:: 0:0
”.o.uv:00§ml~:8
”m: :0 a?" uv 0 goal 0” uv Sal 0:
ow t. .o. n 0. 0: .cmm: 0:03: has
”.ruvnsngEE ”.o.HV>CI:§ ”.o.uv:0=BmI0=0
I I a: a :8 :03
”0580,8030: 30: 8: 0:0 .ruEQﬁ: 30: 3:: :03
”.o.nv:00_BmIES

.2 ON 5% .O. .H."VUNC— HQ: ..H."mwu0mm.&w H. W?

”.ruv:0=BmI:a: ”.rquIES ”.ruvasrszE:
”m: n :8 :03
53:
«0000:: 30:80:88
”00000:: 0:0
5 0:0
”03::50:
:05 85.00.3586 00: 0:0 .rnwaIQSmV:
Ew0:

A:_I:o0vmm000a ”_o::00I:0:0:E:0H
£005 0:0
E0330I0=0 Busswuvglxﬁ
50050100: Baaswnvglv:
50:308.“: :02: :wnv:0I:§
Ew0:
pruoséqueﬁoz ”30:86.

a: on :33 :_I:o0 nv E0450
Ew0:

 

a

”bug: ”$04600 .:_I:00 =0:me

61:: 50:33.6
.:0:I30: .E0:0:I30: £83633: .:0=30I00: 3:30

0: gal—05:00 08 2:90:00 05000203
Hagan—0006:0056 0303:5000 00:

32:40:80 ::0
33 0:0 NEVIS“: .:0I00: .:0I0:0 .EIES .000—I00:
as s ”2.38 0.3560
3.8:
n: “8:405:00 bus
”ﬁg—0093.50.50? 00:

«ED 35:00 v4.0

”0:30:09 0:0

”2:o::0:IZvS&:0J0: A:o::0:Izv::00:mJ:0
A:o::0:IZv:mI:SI€o: A:8:0:IZVE0§:IB0:
60:21 0x0:.0=0.:02I00:.A:o::0:lzv:0I:0:_3I:_
A:o::0:IZv:0I:0:3Ime :2: to:
30:30:00: ”80820 anomlgonmdﬁh
I “$505032 n:o::0:IZv:oI:_ I
.2050: 72:0 : 2r: 0 9050: 2:505 0:0
.A:050:IZX:0::0:IZV:MI:8I:~$.A:o::0:IZv
A:o:=0:IZv:mIEw_03. 8.000305 92: 0:
00:0 0 ”05620680 0403:0029
”BQHISBRV 880:0» 0:0

M8:JE:I:0£0 -o:I=::I:70c0:0I0§m 00:

. 5500:0130 muﬁlgwlzcab QVE0000QI30:

60:0: 38:50:02 00:.9v:0I:0:3I:_

AﬁSIBhBIﬁEwE to: Awwméewa: I
H 0 0 0 s s
”AA:0::0:IZVHQV5._:_ com 0 9 O
A:o::0:IZx0c:0Ix0 .A:o::0:IZVEm0bmJ=o
.A:o::0:IZX0c:_I:SI:~$.A:0::0:IZVQV:_IEw_0B

50.80305 :08 to: 080:? HomqaimIEEBEIO
§<Immm<~u> o 880:0» 0:0

I I cw I
n A. 0::: :13 x :0 x0 . 8005 So x E :0: Saw
.AmvwwaIEmBBAxv—qcas 00500:. to: 0m 05?

 

 

134

 

 

 

: :+:::0I::0::0:nu :::0I::0::0:
:0:: nomadv:
”0580050593: :0: 0:0 .:.H030:0I::0: :0:: =03
”Au-=V .OZHEW 30: "u M‘H
Ew00
”xE0::I:00I_00: “:00I0:0:m 030:?
Hm:2:.00::0: ”MI: 050.5?
u..w0:0:ml::0I::w:0Im.. :00 0: E3005: ”Sol”: 0:::
”ou::0::::0: ”6:03:30: .:::0I::0::0: 030.0?
00000:: ”EID080E

60000:: 0:0
”0:05:50:
we 00: :o: :03

”:00: 0:0
u28830001.:00:3I8I00FX085053003
ASSEE0BIE £638,005:
ANI— .N::I&0:=000:.0_:x0:
000: :0::0:I2 0: : E .: :0:
900— GOEOGIZ OH A—ITOGIuEn—Iﬁmv :_ — no.“

”:00: 0:0

”:00: 0:0
u2883003-::0::0IO:I:8:nv§050E200;
0: 0:0
5.0 "Macsmwsaa: 020

5.?682 :25:

=050u::::

:00 :0::0:I2 0: : :: :0:

00: 0:I:_::I:_ 0: .m E :8
n 00— 0:0
”:00: 0:0
uA.2c:5:09:00?0I0:I_00:nvc.x5::0::I::0::00
”28850000: 00008.85:
5: 05:08:22,005:
:00m:0::0:I2 0: : E .n :8
00: E0000I2 0: : E _ :8
5...: 028,5 30: u“ S 5...: 022.5 30: n: 3
:_w00
”x:::0EI_0:00:I_00: ”£9033: 050:?

 

3:80:38: “50:00:: 030:?
I I I $250000: ”NI: .3 080:?
9.00003 :0: :00 0.. a: 0: Edie ”0:: I: 0::
9.60000000000: a: 0 50.0006: MEI: 0::
00000:: 00003085
8:00
”0:80.:4050: ”E08430? was
”0.5058000: :::0::I0:0:0 ..:»:0:::II::0::00 00:30
. 0:0 :00: :0
9:000:12 0: : 0w:0: 3:205:05 m: xE0EI::0I:00: 09A:
:00: :0
8:I::::I::0 0: : 0:00: 0:305:05 0105300400: 0:»:
”>0::0I_00: :0
£80052 0: : 0w:0: 3:805:05 m: xE0EI_0::0:I:00: 00>:
. 30.5000:
:0 3:000:72 0: : 0w:0: 3:305:05 m: xE0EIH0: 0:»:
:00: :0 2:050:I2 0: : 0w:0: 0:805:05 01050400: 0::»:

0: 2:3: :0 55000: 0:300:20:0

”0:0:::::0:.0:00:0:w 50380.00 00:

”00:80:: 00:

"2:5: 0:0

”0:8084050: :00 “:00I::w:0
$054050: :00 8:000:96:
30::0I0:0:0I::0 E H:_I0:0:m
”:5 :_ ”BI: .0_00:0I::0::::00
0: 25: 00:0
”=0.::00:I::::0.0_:03 0m:

090:5: méd
30:20:00 0:0
”00000:: 0:0

a: 0m :0::0 . ..:.MV::I:00
:0:: 2:000 I2n::0I::&::
”: + ::0I:::nu::0l::m
”05088000930: :0: 0:0 .:.HE0::0:I30: :0:: :03
:_w00
Noun—0:20: ”::0I::: 030E?

 

 

135

 

 

”:0:—00000600003 0:0
325.: 0:3: 5:32 5.0
:0::00I000»: 000
”80080 0:0

”:0 v .080 :00000I0: "V:00I:00I::
”:0 .0 :0::0 :00000I00 "V:00I:0:Ix0
0052:0300? :5

. 0000:: :000M :0 :00I0:00 "0000003:
.33 2 M00 :0 .03 80.000 I
0:000: 08:00 000 :00 0:00 "000000 00

. I . I 502:
:3 000000 0: :00000 00 0300?

8000:0008 :00I0:00.A33IZV:00I::w:03:::W00:0
5:00:30? ”0000:000 0:000»m

33030»: .:=0I0»: .000:I0»:: 000: 000 0.0.0 ”OB—I0};

5:00:00 .E00I00: 0033303: 000: 000 0:20 “:IAEU
500000:

.0._0I0»:.000_I0»:.0:I00:I0»:: 000: ::00 07B “:IDZM

0&0:

"AgwIEzUWEzUJSB »:0:0 0:0 020 0:0 :8
x: :02: 020.003 005 0:: 020 :0 :0:
0506 000050.02, 0:8 0:: ":00 00 :o:
03 ”:00 :000 00$:
”0:340:00: ”:00I::w:03 .E00I00: 00%:
000000080 000
53 :00 :00I0Eo
”0:340:00: 0: ”N::I0:00 .:0:I0:00:::00
020 :00000080
000000050 000
”9:340:00: :00 ”:00Im0:
”:3 0: u=0l0w: .000:Iw::
:0:»0I_0:000 0: ”0:000:00
020 :00000800
000000000 000
”0:340:00: 0: ”:00I:w: .::I:w:
:3 0: u=0I:w: 00033000

 

POM :00000800

:: 0:000»: :0 000:0I000»: 000000000:

”8000»: 000

so: :00 ”:00I:00I:: :00I:00Ix0

03 0: ”8005100000

”0:340:00: 0: ”0_I:0:I:»: .::w:03I0»:
03 0: 0:00»: 00003:»:qu :: 0:000»: »:::00
”—0.000000800003 0:0

2:00:05 000003 04.0

083000 000
:::000:0 000
u: 000 u: 000
"enum:00I0:0::00
00:0 A00000IZHN:00I0:0::00:::
”000: 000
”000: 000
n33:00::I:0w:03ux2:080:30?
000: 00:000I2 0: : 01:8
nA:X~:00I::0::00:0:00:I0:0::00nvA:VE0::00IB00
000: 00:000I2 0: : 0: : :8
: :+~:00I0:0::00uu 6000:0000
00:: ».:.u3I::::
0:: 000
u:: 000
”on” ::00I0:0::00
”000: 0:0
”000: 0:0
0:1: .5000000903005:
”0.080305: .0 0:03.85:
H0casualagvﬁﬁeIBBAA. 008.20:
000: 00I::00I:00 0: : 0: m :0:
000: E0000IZ 0: : 0: : :0:
00:: A50000IZH::00I0:0::00V::
”:0 m :0: :0:?
”000— 000
x:::_I0:0::uvcx::00I0:0000::000:I0:0::
000: 00I::00I:00 0: : 0: : :0:

 

 

136

 

 

”:55 H03 0:0
"0000000 05
I I E 0:0
a: S 005 5 3w: uv :5 3w: :0
“@53—

355300 s: 05 ._. n 0830. 05 ._. u :0 500
35535000:
I :30:
a ,5: 0o 55 ,5: 055:

F9. 0:0
300.3435: 3:0 3:0I3w:
”03013005: E ”:quw:
33 5 35:30:
35 :_ ”000— 503000

3 ROM 3:5
33300955353 00:

.3300: 0.3.0

”5:64—00:73 05

”0553550 uv S9:0I:0S0:
38003530900
.E::I:2.0§0I3:05v 00:: :00 $40 :ImZU
5:05:00

0:048:6818958535 9:: :8 02: ”0.02:
5:555:00
3:04.030 3:045: .9039: 00:: to: 3.20 ”MP—.2300
30550550 .mﬁISoJSE
0:18: 52.5030 93. :8 5: 35050
AwaJnonE:
.000—400030553500 .0555: :0:. :00 052 3Ix32
300005 05
a: v 55 354380 00 0003800 uv 0:0—I505: w
E 0
0543030 .000—#:0805005 “0030935:

I I I I 300090 05
a: v 55 3:05 30: :_ 30: 05 35:3 30: x0 uv ma 0

 

:50:
0:0:_I00:I:_ 3:0:_I30:I5v0000000 35:30::w
:. 0:
”3:890 028530503 33:0 00: 920 3:0 :00
5 526025.33 00:0 a: 02: 5 5:
”MEEcIHZUWPZUgSB 30:0 00: 3.20 33 :00
u 28m 5: 5:383 008 a: .8: ”a 80
x5 5:353:15? 33:0 00: X32 ”:0 :00
33 ”mad: .002I5tan 00:90
”033485: ”0553550 30:90
”003485: “9535408 3:953:50 .5:I:8 30:90
35:00:80 05
55 .00 3004380
”033485: :_ “N:_I9:0 3543:5300:
RED 35:00:80
35:09:00 05
x5555: 3:0 3:0IwE
3E :— 05IwE .000—5::
”0:34.005: :3 ”03:03:00
02% 35:09:00
35:00:80 05
”A030IEH5: 3:0 3:0I3:0
33 :3 3000: .0=0I3:0 .EI053000
3.20 35:09:00
35:09:00 05
30304005: 50 3:0I0w:
”003485: :_ ”:39
05 5 0:30: 52.0050:
3.0m 35:09:00
35:09:00 05
530485: :5 3:0I§E
3E :_ 30503::
”035500: :_ ”mslxa: 3:385:00
I K92 35:09:00
,3 ~60: :85: 00 :95 0002 0:50:55

£000I:050: 05
3003485: 50 3:900:85:
3B :5 ”503530900
. w 55: :3 3:005:95 .0555:
35 E n 3543030 .0=0I 000 35:30:83 3:0:_I30:Ix0v300
3 3:05:85: 335

 

 

137

 

833..ch 3&5 2.5

”5:6.qu one
8885 c5
5 c5
a: 9 bad AbucsoolqsoSnISJE nv 251:8
x+§§8ua=uu§§8d= 55
crux-.20 Ea Spawn—".160 8: c5 Buy—3:38 has

.8 2 bad C258 @383 Bowmwmwwlﬁw 55
brag-3.8 n5 Essay—01538 23 a .uiolﬁov .:
Ewen

noun—833 nausea-d: 29%.?
961283085 ”amoooalqalao
-. Ewen
mm hzu no EEG p.20 830328“

FZU can
”Ania-.350: So use-.30
35 E 388 $6180 6353-59
a 920 has
gigoaJEEQﬁoB 8:

.3550 2.5

”dlwozm 98
”888a can
a: 2 Sam Stands: NV :5qu
E 95 5 new
”ma-vasoﬁu”: - asnzvbtsdé
”quoo_qoo_cco
x_+§t3d§ué§3d§
82 a - £56 9 _ 5 _ .8 ”Waco.
nAoaalwﬁaotsndgvaalmmmq "$315388 83
”EIwEnuo-asnlmg 55 .r u vac—£8 :
5:. Enaisde 8: Ba ._. u xs-wev:

I Ewan
”L _ ~ ~ 550.. "”83 :35: ”25le 2.85:8

 

 

as ”ma-gang 2&5,
”QSDIESB mugging 033:;
golwﬁvmmoooa
Ewen
ﬁ 072 he QIwOZM ”.389an

..::§8.. 9585585 .........
....... mm nogauaoo 08:3 62% N 25 a mm 6 M52): 5....-

Elmo—IE one
5885 95
”ma 2 Sta Stan-.95 uv :8le
E 98 E 98
”muuxanooeué - azuzvbtsdé

”munch: moo— c5
5ésmsdﬁnébusdé
- a8. - £36 2 _ 5 d an ”N452
x25 magmas :85 mm": uuwu “.8583 as
”Elwﬁunuotanqu 55 .r u 32le b
55 Aussie-ME 8: no“ .r n £339 m
Swen
r. 59: _ 5o: "Hogan—«=5: 690le 63.88
as ”mu-x0282 03%?
"BB-.350: ”Staging 053:;
golwﬁvmmoooa
Ewen

mm 02M no :IwOZM 2:88:63
Len: ~ #8.. AHZ: u :xx: H WMV uuuuuuuu
nnnnnnn mm Goaﬂswﬁs DwOﬁB OZ“ ﬁt 3%“ 5 mm Hulwozm IIIIIII

672 c5
”Egan—«So: :5 ”=5le

33 E u=olmE .32le
632438: 5 £le Hon

2 02m 35
”ﬁguaqlqaeqﬁoa 8:

hOuﬂhUn—QU hQQE—uz EO—ugwm GJMV

 

 

138

 

 

”8322);? E0
”8008: 0:0
5 0:0
”0: 790590.020 :8 x?» 00—0
a: r. $053.88 .8 :2,
”.953on a: r. “Bonﬁgoos 5: :«3
”L .nvSo 0:0 :05 Ari-:33:
500:
$008:
500:
m_ 0.020 .8 5320:le 050228:
. £020 ::0
"GE :5 ”Belg—0
:5 5 ”:0: 0:8
mm 0.020 bus
”ﬁgualgauacoa 0m:

8.2250 0.86 3.3

”5.535466 ::0
”8008: ::0
E 20
Hone; :05 ﬁEéBqu-Enaov:
m: ::0
a: On :0:: .o. .0: 2 :0:: .r uv Q0HmI30:
5.: EEéBIZVISuae:
”30:0":5
”03803—0Im0a 8: :5 Eng—0&0; 5:: :03
5mg
”CHESS: 3:0 055:?
00008:
Sag

ﬂ E0457. 0: 5890:0453 83.08203
2:04:00: ::0
55 So ”Q0343:
35 :_ u=0d0§tom
ﬂ “5&0:- 52:0

 

saxoaqugiuoa 0m:
08550 55.33— 24.0

”ZZQJSE 20
$0008: 0:0

5 ::0

a: v :0:“: «Elna: uv SoJBE 0.20

a: v :0:: 333:. nv SonB: :05 .r u ~00Ix2: a

£03
A_00Ix:E.N:_Ix:E. 333500008: ”mgalxsa
:33

m_ E .8 ZZQJBE 0:500:28:
I I USE 9:0
A093 330: :5 as: 03::

33 E ”_00I§E
”0:34:50: 5 NSI§EJ£I§E§OQ
0: X32 55:0
”ﬁg—89:50:03 0m:

00:23:: «Ed

”:SEVIAEU “:0
”0008:: 0:0
I E ::0
a: o— :0:: .r nv So 9:0 030
”m: 2 :80: .o. uv Solqao
:05 8 u NE :0 as v Bob
x3d§ﬁ>u§: u” a: ”cs-aesaﬁog u” 2:
500:
”—05:09?“ 45 053:?
ANEIan .~:_I:80vmw0ooa ”0:09:00
I :30:
x E20 00 :55 E20 0:300:28:
£20 0:0
”GE :5 uaolaao
”0:34:50: 5 ”NEIan JEIQEovtom
am $20 35:0
”:0:—00% 58:18? 00:

 

 

139

140

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

C.2 Figures
On 011-1 02 O1
FF FF FF FF
A D A D . . . A lab—T A D
. clear
3 ' clk
load
In In-l 12 I1
Figure C.1. An n-bit register with parallel load.

 

141

an-l
bn-l

    

A = (an-1 an-Z - - - a1 30)
B =(bu-~1bn-2 ' - - b1 b0)

bn-3

 

(A2 B)

Figure C.2. Ann-bit magnitude comparator.

142

Increment

 

clear

clk

 

Figure C.3. An n-bit up-counter.

APPENDIX D

Input Data for DMNN Binary Classifiers

 

 

 

q q q
o o O o
QCQQQQQ. m N N Q o q
ph—iv—u-H-‘F‘F‘F‘ v-I 1‘ M o d C
OOOQQQQQ 3 g N Q C O
. . I o N . o
OCOOOOQ bl. , m C o c
oCQQQQQQ o 9 d m N q
ddQQQOoo OOOQQQQQN ~ m Q : g _
oOQQQQQQ ddo’oooOO" g g; 8 co .. q
~~~~ dQQQOOOC ccoOQQQQ c w - Q Q o
""Soocc oocqqqqq ddo’o'QOOQ. «3 q C o c: q
ogcoccc dddocooc oOOQQQo-C- C O E S O
O ooo OOOOQQQQ dddooOOQh a w N q
oooc . . - ' 93c; can 9 .— Ox
cow—a—U-‘O OOOOO ccqqqq . . b O V) O
go¢_~—o QQQQQQQQ ddcoooOO a d d 6 q
OCQ—u—n—to ooooooog quQQQQQdeho-glx N .n O
.—u—1—IOOOO OQQQQQQ 0 OCOOOOOO 0" ' meg N V) O,
__ocoo dOOQQQQ” GOQQQQQQ ” ”“3 3 o
'— v—IOOOC OOOQQQQO. ddcoooccgg. . O. . .n q
”2.00:: dddocc"° COOQQQQQ o~° C? d d o
"ﬂﬁoooc OOQQQQQQ dddoOOQQQQQQdQc o q q
~~_oooo ddOOOﬁOO ooOQQQQQOQoQ' d d O Q
Sim—coca OQQQQQQQ dddOOOO—‘gq <3.ch a q o,
ddO~QQ°° OOQQQQQQdOdOdOd d o o
OOQQQQQQ ddooo~OQwQ'QoQo c Q Q
OOQQQQQQ -= ddOO~O°C9~Q8Q89o o c, O.
dgooOOGQ EPoQOQQQQQSO; QC“ 6 o O
uooqqqqqq ‘3’ do‘dv—oocodqdqqqc o o, O.
EédoOOOOQ C,OCCQQQQQ'CO V05 dao o
g '5 dd—‘OQQoogmnggc omqgQ
“3 a‘ooqqqqqq 5:1 NgNo'"C5 O°°©
3 g d—‘QQoogg :3 8 .:\O$O .Q3.Q
W CCQQQ..'-d-o-'°doooo
ﬂ ZJdeooooqu'Q'coo
I
I; I
“Dug :8
gﬁagag
.Uo
ggmmga m"8
o.” 39% . :7; m
:awOHQb N 3 OOOOCOO
«WEB-ED >.. C '9' OOOOCOO
an m:._.*“~ -O
u .333'“0m—n C 'o QOOOOOO
8 000 0 33H qqo- cQOOOOC
5
do 3535 QQN coooOOQ
OU§>%H0 GOOCV'O OQOOOOO
g 5 ”g :55 o'cSC-o-‘QF': cocooc"
00 DO pu—u-H" “"F‘ ' '61 O‘O '— CDC
0 t:"" >45; CO CO Coat-am E CO C
u-oozmo ooOOOO QQ- - mo ﬂooocc
:0:: at: ooo°~° Oco o'o'wco z ~oooo<=°
§Xé385= oooogg 85:0 ooN~QQ g _ooooooo
.HQ . o~~~ "W o
svmwéxg c__oo qqqq °"~§°o n. a
:Sgioa T oooo QQq-oo w o
CWHUEOB '8 dv-‘Oﬁ .. .
88350§= E “ QQQQ 3 a
awacagm 2 a) oo~~ ﬁ, = E
«I =3 ‘90) H a. .a .. '8 W
592§8m ° 5 3 8
0501,3538 n‘: o E M an:
out "' U
30 $25“ 3 é 5
3512 > K 155 .
.20 . u-sw‘d) o O at 3‘ a g
5688°§5 * E a i a z
gﬁﬁgsﬂ : g H m a .
§£°5°5 : . ' '
”"3oﬂw

 

M3

 

 

Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q
COCv-‘OOv-lo—ﬂccv-Iv-iv-Iv—v-ﬂv-tv-‘v-lc-tv-l
Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q
v-t—‘r—v-‘v-‘v-‘v—QQQQQQ Q—‘QQ—tQQ

Q
Qo-QQ QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
!-: v-:v—: v—n—u—Iv—u—u—u—u—u—uvuOv—IOFuOv—uo—tcv—tcv—tcv—to—nv—v—Omcv—u—u—tO—no
QOQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
o' dd~c>~ooc—no—o~o~c~o~o~o~o~o~o~o~o~ooo~o~
QQQQ QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
QCQQ QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ~Qv-‘QQQv-t
QQQQ QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
v-JQ v-‘Qv—QF‘Q—‘QQQQQQQQQQQQQQQQQQQQQQQQQQQv-‘QQ
QQQQ QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
~Q v-‘Qv—Q—‘Qw—QQQQQQQQQQQQQQQv-Qv-tQQQv-Q—‘QQQv-‘Q—t
QQQQ QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
QOQQ QQQQQQQQQQHQ—th—Qv-tQv—Qv-‘Qv-‘QQQQQQQQQQQQQQ
QQQQ QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
QOQQ QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
QQQQ QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
AQQQ “Q'—QQQ—‘Q—tQQQ—‘QQQQQ—‘QQQQQQQQQQQQQQQQQQ
QQQQ QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
AQ—‘Q v-‘Qv—Qv-‘Qv-‘Q—‘QQQQQQQQQQQQQQQv-‘Qv-‘Qv—Q—‘Q—‘Q—‘Q—t
QQQQ QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
QQQQ QQQQQQv—QQQv-‘Qv-‘Qv-‘Q—tQv-‘Qv-‘Q—‘QQQQQQQv-‘QQQQQ—t
QQQQ QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
QQQQ QQQQv-‘QQv-‘QQQv-‘Qv-‘Qv-H-t—tQQ—H-‘ﬁ—‘Qv-‘Q—tQ—tQv-‘Q—tQv-Q
QQQQ QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
v—:v-uv—:r-: v—u—u—nv—av—tv—u—th—u—nH—uOv—u—u—u—cHQu—nv—v—u-u—u—u—u—u—u—u—u—n-nv—u—u—tOv—n
USQQQQ QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
EQAQA Qv-‘Qv—ﬁ—iQ—‘Qv—Q—‘Qv-‘Qv-‘QQQv-‘QQv-‘QQv-‘Qv-‘Qv-‘Qv-‘Qv-‘Q—tQ
gQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
w—tc—o v—aOv—nov—tOu—nOv—n—tu—n—u—tv—Ao—u—n—u—t—tOv—u—u—tv—u—u—u—u—u—u—u—u—u—u—u—no
QQQ QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
v—:v—;v—:Q v—u—u—t'—t—no—n—n—u—uv—no—to—u—u—tOv—Ov—n—v—IC—tcv—tooc—to—cOQO—t
[—-QQQQ QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
.QQQQ ~QQQQ—*—*QQQQv-th—Qv-‘Q—‘v-H-‘Qv-‘Q—tQQQQQv-‘QQQQ—tv-‘Q

 

 

6%

IS

QQQQQQQQQQQQQQQQQQQQQQQQ—‘ﬁ—H—v-‘v-‘v-‘v-‘v-H-t
QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
QQQQQQQQQQQQQQQQQQQQQQQQQQQQ—‘v-‘F'v-‘v-‘Q
QQQQQQQQQQQQQQQQQQQQQQQQQQQQ—t—‘F‘F‘v—‘Q
QQQQQQQQQQQQQQQQQQQQQQQQQQQQF‘F‘F‘F‘F‘Q

O

the test pat-
git, three 1

 

D.3 5~digit Classiﬁcation Problem

tterns, and 20 % no
one idea] di

problems,

gits, 10 % noisy pa

t classiﬁcation
patterns

i
In test

t

and three 20 % noisy patterns are listed in a row.

it

For S-di git and lO-dig
nonsy pa

terns consist of ideal di
patterns. For each digi
ems,

QQQQQQQQQQQQQQQQQQQQQQQQQQQQ—‘v-H-H-H-‘Q
QQQQQQQQQQQQQQQQQQQQQQQ—‘F‘F‘F‘HQQQQQQ
QQQQQQQQQQQQQQQQQQQQQQv-‘Qv-H-‘v-‘HQQQQQQ
QQQQQQQQQQQQQQQQQQQQQv-‘QQv-H-‘v-‘v-‘QQQQQQ
QQQQQQQQQQQQQQQQQQQQ—tQQQv-‘v-H-H-‘QQQQQQ
QQQQQQQQQQQQQQQQQQQ—‘QQQQ—‘v-‘F‘v-tQQQQQQ
QQQQQQQQQQQQQQQQQQ—‘QQQQQF‘F‘HHQQQQQQ
QQQQQQQQQQQQQQQQQHQQOQQQv-‘v-‘ﬁ—‘QQQQQQ
QQQQQQQOQQQQQQQQF‘QQQQQQQF‘F‘V—‘HQQOQQQ
QQQQQQQQQQQQQQQF‘QQQQQQQQF-H-H-‘v-ﬂQQQQQQ
QQQQQQQQQQQQQQF‘QQQQQQQQQF‘F‘F‘F‘QQQQQQ
QQQOQQQQQQQQQF‘QQOQQQQQQQF‘F‘F‘F‘QQQQQQ
GOOQQQQQOQQQQv—QQQQQQQQQQQv-H-H-H—QQQQQQ
uQQQQQQQQQQQ—‘QQQQQQQQQQQQv-‘v-‘F'v-tQQQQQQ
QQQQQQQQQQQHQQQQQQQQQQQQQF‘F‘F‘F‘QQQQQQ
QQQQQQQQQF‘QQQQQQQQQQQQQQ—‘v-t—‘F‘QQQQQQ
~QQQQQQQQ—‘QQQQQQQQQQQQQQQ—H-‘v-H-‘QQQQQQ
QQQQQQQ—‘QQQQQQQQQQQQQQQQ—‘F‘Fﬂ-‘QQQQQQ
8QQQQQQ—‘QQQQQQQQQQQQQQQQQ—‘F‘F‘F‘QQQQQQ
gQQQQQv-‘QQQQQQQQQQQQQQQQQQv-‘v—F‘"QQQQQQ
SQOQQF‘QQOOQQQQQQQQQQQQQQQF‘F‘F‘F‘QQOOQQ
§QQQ~QQQOQQQQQQQQQQQQQQQQ—H-‘v-‘F‘QQOQQQ
QQF‘QQQOOOOQOOOQQQQQOQQQQ—‘F‘F‘F‘QQOQQQ
ZQF‘COCOOOQQOQOQQQQQQQQQQOF‘F‘F‘F‘QQQQOO
IFlOOOOCOOOOOOOOQOOCOOOOOOF‘F‘F‘F‘COCOCO

 

144

 

 

Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q
QQQQQQQQQQQ—‘QQQQQQ.
Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q o
QQQQQQQQQQﬁQQQQQQQQ‘h"

ooooocoooocoooooooooooocoocooooocooooggao

ooooocoooooocoooooo~oooooocccoooooooog~~c
ocooooooooocooooooooooocooooooooooooc ddgq
ocoooooooocoooooowooooooooooooooooooo awe
ooooooooooooooocoococcoocoooooooooooo; mq
cooooooooocoooo~ooooooooooooocooooooo mad
oooooooooooooooooooooocooooooooooocooo ._q
ocococcoooooc~ooooooooocoooccocooooocccoo
ocooocoooooooooooooooooooooocoooooooowﬁmo
ococooooooo~oooooooooooooooooooooooooono
ooooooococooooocoooooocooooocoooooooog dgo
oooocoooo~ooooocooooooooooooooocooooo d
ocooooocooooooooooooocoooocoooooocoocouOo
ooocooo~ooocooocooooooooooooooooooooo hhd
oooooooooooooooooooooooooooocoocoooco ch
ooooo~ooooooooooooooooooooooooooooooo NNgd
oooooooocooooooooooooooooooooooooooooo

ooo~oooooocooccocooooocooooooooooooooo .d
ooocooooooooocooooooooooooooooooooooouc
o—ococoooocooocoooooooooocooooooooooo gmmg
oooooooocoooooooooooooooooooocooooooomo 08
ddddddooooooooooooocoooooocoocooocoo—Q .dQ

oooooooocococcoocoooooooooooccooocoooQ'QQ
oooooooooooooooooooocooooooooooooowcoVNQ”
ooooooooocoooooooccoooooooooooooooooomgs3

02

8

99

QQQQQQQQQQQQQQQQQ QQQQQQQQQQQQQQQﬁQQQQm
QQQQQQQQQQQQQQQQQ QQQQQQQQQQQQQQQQQQQQOOOC
QQQQQQQQQQQQQQQQQ QQQQQQQQQQQQQF‘QQQQQmemc
QQQQQQQQQQQQQQQQQ QQQQQQQQQQQQQQQQQQQQNQOO
QQQQQQQQQQQQQQQQQ QQQQQQQQQQQF‘QQQQQQQQ QQQ
QQQQQQQQQQQQQQQQQ QQQQQQQQQQQQQQQQQQQQ QQF‘

QQQQQQQQQQQQQQQQQ QQQQQQQQQF‘QQQQQQQQQQQQQQQ

 

.2

6%
86

 

 

Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q Q
QQQQQQQQQQQQQQHQQQQQQ
Q Q Q Q Q Q Q Q Q . . Q Q Q 3Q Q Q Q Q Q
QQQQQQQQQQQQQQ'ﬂQQQ Q

QQQQQQQQQQQQQQQQQQQQQQQQQQQQQ aQQQQQQQQQQQ
v—u—av—u—nv—u—u—dOv—dov—nv—dv—nOv—IOv—IQv—u—H—Ov—aQ—d—u—acv— ROOQOOOOOCOO
QQQQQQQQQQQQQQQQQQQQQQQQQQQQQ oQQQQQQQQQQQ
Q—‘Q—‘Q—‘Qv-‘Qv-tQQQ—'QF‘Q—‘Q—‘Q—‘Q—‘Q—‘Qv—Q EQQQQQQQQQQQ
ocooooooooocoooocoooooooooooo qqqqqqqqqqq
Qv-th—Q—th-‘QQQv—Qv—QQQQQQQQQQQQQQQ - QQQQQQQQQQQ
coooooooooooooooooooooooooooo qqqqqqqqqqq
QQQQQQQQQQQQQQQQQv-‘QQQQQv-‘QQQQQ QQQQQQQQQQQ
QQQQQQQQQQQQQQQQQQQQQQQQQQQQQ :QQQQQQQQQQQ
Qv-‘Qv-‘Q—th-‘Q—tQ—‘Qv-‘Qv-‘Qv-‘Q—‘Q—‘Qv-‘Q—‘QF‘Q EQQQQQQQQQQQ
QQQQQQQQQQQQQQQQQQQQQQQQQQQQQ bQQQQQQQQQQQ
deQv-‘Qv-‘Qv-‘Qv-tQv—Qv—Qv-‘Qv-‘Qv-‘Qv-‘Qv-th—QF‘Q .Z’QQQQQQQQQQQ
ooooooooooooooooooooococooooc qqqqqqqqqqq
Qv-4Qv-th-4Qv-‘QQQ—‘Qv-‘Qv-‘Qv-th-‘Qv-‘Q—‘Qv-‘Qv-‘Q QQQQQQQQQQQ
QQQQQQQQQQQQQQQQQQQQQQQQQQQQQ QQQQQQQQQQQ
QQQQQQQQQQQQQQQ—‘Qv-‘QQQv-‘QQQQQ—‘Q QQQQQQQQQQQQ
QQQQQQQQQQQQQQQQQQQQQQQQQQQQQ oQQQQQQQQQQQ
QMQQQ—‘Qv—tQQQv-‘Qv-‘QQQQQQQQQQQQQQQ aQQQQQQQQQQQ
QQQQQQQQQQQQQQQQQQQQQQQQQQQQQ QQQQQQQQQQQQ
QQQQQv—QQQ—tQv-‘QQQHQﬁQ—‘Qv-‘Q—th-‘Q—‘Q oQQQQQQQQQQQ
QQQQQQQQQQQQQQQQQQQQQQQQQQQQQ :QQQQQQQQQQQ
mooooooooooo~ooooc~ocooo~oo~o 3ddoddddddd~
oooooooooooooccoooooooooooooo ﬁqqqqqqqqqqq
v-‘Ov—tv-‘v—QOv—‘Ov—‘Ov—COv-IOv—‘v-ﬂv-‘v-dv—iF-dv—lo—I‘v—tv—v—IOOQ—I wQQQQQQQQ—‘QQ
QQQQQQQQQQQQQQQQQQQQQQQQQQQQQ ’BQQQQQQQQQQQ
F‘Qv-‘Qv-‘QF'Qv-‘Qv-‘Q—‘Q—‘QQQQQQQ—H-‘QQ—‘Q—t BQQQQQQ—‘QQQQ
QQQQQQQQQQQQQQQQQQQQQQQQQQQQQ QQQQQQQQQQQQ
Qv-‘v-‘v-‘v-‘v-‘QQv-iv-tv-H-‘QQQHQQQv-‘Q—tQQQQQv-‘Q uQQQQéQQQQQQ
QQQQQQQQQQQQQQQQQQQQQQQQQQQQQ QQQQQQQQQQ
o~~~~~~~~~~~~~~ooocooooooooc~ gpd~dddddddd
QQQQQQQQQQQQQQQQQQQQQQQQQQQQQ mQQQQQQQQQQQ
v—u—tO—uo—no—nOc—nQQOoo—u—u—u—u—u—u—u—n—u—dv—u—u—u—I OAQQQQQQQQQQ

 

 

M5

 

000
0000
0000

00

Q Q
QQ
QQ

000

Q
o
q

0000
0000
0000
0000
0000

00
L000

0000
0000
00L0
0000
00

0000

o
d d d d d d m
oooocooooooooooooooooocooooooocoooococooom
oooooocooooooooooooocovocoooooooooooooooh
oooooooooooooooooooooooooooooooooooooooooo
dddoddddddoooooocoocvocooooooooooooooooo

oocoooooooooooooooooooooooooooooooccooooca
oocoooooocoooocooooAoooocoooococooooooooog
oooooocooooocoooooooooocooooooooooocoooooh
ooooocoooooooooovooooocooooooooooooooooo

cooooooocooooocooooooocoocooooooocoooooqq

ocooooooooocoooAocoooooooooooooooooooooooQ
QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQb
ooooocoooocooAooooooooococoocooocoooooooo0°
coooocooocoooooocccoooocoooooooooooocoooo

ooooocooocvocooooooocooooooooocooooooooo

ooocooooocooooooooooooooocoooocooooooooooa
ccooococo—ocooooooooocooooooooocooooooooo“
ocooocooooooooooooocooooocoocoocooooooocoo
oooooocAooooooooooocoooocooooooooooooooooo
cooooooocoooooooooooooooooooooococooooooow
cooooAccoocoooooooooooocooooooooooooooooo

oooooooooooooooooooooooococooooocooooooooo
cooAooooooooooooooooooooooooocooooooccooA

cooooooococooooooooooooocccoooooocooooooo

voooooooooooooooocooooooooooooooooocvoo

ooooooooooooooooocooooocoocooocoooooooooow
ooooooooooooocoocooooooooooooooooocvoocog
ooococoooooocoocooccoooooooocooooooocoooco
ddddddddddodooooooooooooooooooooovoooooo

ooooooooocoooooooooooooooooooooqqoqooooqqQ
oooooocooooooooooooocoooooocooooAoooooooo0°
ooooooocoooococooocoooooooocooocooooooocoo
ooooooooooocoooooooooocoooocovoooooocooco

 

 

Q Q QQ QQ QQ Q Q Q Q Q

. o0 CC oo co oo o c o c o o
'5 #3 .mm QQ Qd QQ QQ QQ Q " Q Q Q Q
30% :ﬁh 5m cq oo oo oo oo o c o o o c
oAmQ ﬁgeq SgaQ QQ Q5 ¢_ Qm Qw QQ 5 QQQQQQQ
Qng Sch gqﬁo ea c 5g 92 CN GA 8 ooooooo
5Q5Q dOoQ QQQQ QN Q Qm Q8 Q QQ o QQQQQQQ
o__o végo QowQ ea c oQ o. cﬁ co 2 ocooooc
ngQ wwa ORQQ Qd Q. Q? Q? Q? QQ QQQQQQQ
gage gMEO 288° o. co Qw Qw o: co 3 ooooooo

v-1 0
QdQQ Q35Q addQ Q2 Q: Q3 Q8 QM QQ QQQQQQQ
o Co oc o d .o o cw c Q“ oo oo 3 ooooooc
onQ mmgQ QQQ Q“ Q“ Q Q” Q9 QQ Q QQQQQQQ
$550 ggao ”530 O: o; 09 o? o? so a ddddddd
cng MNBQ SAQ Q Q Q: Qm Qo QQ b QQQQQQQ
ﬁdﬁQ ﬁﬁdQ dﬁQ C ox co 0% 03 co 3 ooooooo
Q'QQ QQ'Q d'QQ Q” Q Q2 QM QA QQ z QQQQQQQ
oo 0 (’ﬁ

gggo aggo owQQ cat,2 o. c? o. cQ co ooooocc
NNwQ chQ gon Qo Qo Q Q? Q0 QQ QQQQQQQ
Nome o co ﬁe 9 GM co o~ :35 Oh co ddddddd
qqqo QQNO oﬁao om Ob o5 on o0 oo Q qooqooq
gde Sdi dSQd o(\ 53 68 5g dm dd g ddodddd
mmhm mm_A w~N¢ om qA qd qq qd QC 5 qoocho
msgg gmwg ﬁaﬂ" oo oo o. as o. oo o ddddddd
Somg 5§8g 35mg QQ QQ QQ QQ QQ QQ g QQQQQQQ
dqu dddd 653; ooNoomoo cowooooo ddddddd
“mmh 0 Oh hNM quqqmoc QQQQQFQQ 8 qoqqqoo
gmom 33:0 ﬁgba ddgdd 55355355255 & ddddddd
mQVM Shmh wmhw oqmqq QQMQQWQQBQQ 5gqoooqqq
@883 ,8Q8 gqgg ooooooooooooooooo BaOOOOOO~
3555 g5?5 5955 QQQQQQQQQQQQQQQQQ QQQQQQQQQ
amﬁathNVFMBWw occoooooooooooooo =>ddddAdd
wwmAmcmm>momovmqqqqqqqqqqqqqqqqo ﬁﬁoqqooqq
Bzwggsggggnggw oocoooooooooooood gaddAdodd
dQQQQdQQQQQQQ.QQQQQQQQQQQQQQQQQQ mQQQQQQQQ
.oooo.ococcoooocoococoocooooocoo Aooooco

 

M6

 

 

 

ideal

11111111111111]
11111111111111
11111111111111
11111111111111
11111111111111
11111111111111

111111111111111

111111111111111

111111111111111

000000000000000

000000000000000

000000000000000

000000000000000

000000000000000

000000000000000

000000000000000

000000000000000

000000000000000

000000000000000

1
1
1
1
1

v—u—aq—tv—u—nq—nv—u—n—u—uv—tv—u—tq—u—u—u—u—tO—nO—no—dOv—Ov—Ov—tov—IO—no—no—n
v-‘Q'-*Q—‘Q—‘Q—‘Q—‘Q—ﬁQv-‘Qv-iQQQQQQQQQQQQQQQQQQQQQ
v—Qv-‘QHQv-‘Qv-‘Qv-‘Qv—Qv-‘QF'QQQQQQQQQQQQQQQQQQQQQ
v-‘Qv-‘Qv-‘Qv-‘Qv-Qv-‘Q—*Qv-‘Qv-‘QQQQQQQQQQQQQQQQQQQQQ
v-‘Qv-‘Q—‘Qv-‘Qv-‘Qv-‘Qv—Q—‘Q—‘QQQQQQQQQQQQQQQQQQQQQ
v-‘Qv—QHQ—‘QF-‘Qv-‘QHQ'—QHQQQQQQQQQQQQQQQQQQQQQ
~Q~Q~Q~Q~Q~Q~QMQv—QQQQQQQQQQQQQQQQQQQQQ
v-‘Qv—Qv-‘Q—‘Qv—Q—‘Q—ﬁQv—Qv‘QQQQQQQQQQQQQQQQQQQQQ
v-‘Qv-‘Qv-‘OF‘Ov-‘Qv-‘QﬁQHQHQQOQQQQQQQQQQQQQQQOOO
r-‘Qv-‘Qv-tQr-‘Qv-‘Q—tQ—tQ—tQF‘QQQQQQQQQQQQQQQQQQQQQ
v-‘Qv-‘Ov—Qv-‘Qv—Q—‘O—tO'—Q—‘QQQQQQQQQQQQQQQQQQQQQ
HQv-‘Qv-‘Ov-‘Qv-‘Ov-‘Ov-‘Qv—Qv-‘QQF‘Q—‘Q—‘Q—‘Q—‘Q—‘QHQ—‘Q—‘Q—t
v-‘Qv-‘Q—‘Qr-‘Q—‘Q—‘Q—tQv-‘Qv-‘QQv‘Q~Q~Q~Q~Q~Q~Q~Q~Q~
~Q~Q~Q~Q~Q~Q~Qv-‘Qv-‘QQv—Qv-‘Qv-‘Qv-‘Qv-‘Qv-‘Qv-‘Qv-‘Qv-‘Q—t
v-‘Q—tQ'—QF-‘Qv-‘Qv-‘Qv—Qv-‘Qv-‘QQv-‘Qv-‘Qv-‘QF‘Qv-‘Qv-‘Qv-‘Qv-‘Qv-‘Qﬁ
HOv—‘OHOHOF‘Ov—Cv—OHOv-‘Ocﬁ-ﬂOC-COU-‘Ov-‘Ov-‘OV-IOv—iOQ-QOHOF‘
Ov-‘Ov-‘Q~O~Q~Q~Qv—Q—‘QQv—Qv-‘Qv-‘Qv-‘Qv-‘Q—‘Qv-‘Q—‘Qv-‘Q—t
OO—Io—uo—uO—nO—no—uo—aO—aO—IOO—nOv—Io—nOv—o—nc—nO—no—to—nO—t
6~Q~QQQ~Q~Q~Q~Q~Q~QQ~Q~Q~Q~Q~Q~Q~Q~Q~Q~
v-‘Ov—Ov-‘Ov—Qv—OHOv—Qv—Q—‘Qov—Q—‘Q—‘Q—‘Q—‘Q—‘Q—‘Qv—‘Q—‘Qﬁ

 

t
1

twork conﬁguration when the DMN N is trained wi

W$

mum
-Ne

 

 

QQ Q QQ Q QQ Q
QQ Cd QQ Qo QQ Q
QQ Qo QQ Cd QQ Q
I O I O o.

I

q QQ Cd QQ Cc QQ

Q Qm QN Q QQ Qh QQ
Q QN Ow on Co Qm QQ
Q Q0 Q8 Q5 QQ Qm QQ
Q Q. Qo QN Q3 QQ QQ
Q o. Q5 QQ Qv Q QQ

oOv-o' dtx Dd db“, dd

V5

5 Q% Qv Q3 Qg Q QQ

Q Q_ Qg Om QQ Q QQ
QW QN Q? QQ Q5 QQ
QQ Cd QQ Qﬁ O' QQ
Q" Qo QN Q3 Qa QQ
Q~ Q3 Qh Q Ow QQ
Q2 Q Q5 Q§ QN QQ
Cd Q, Qﬁ Q_ Q? QQ
Q' QQ Q? QQ Q QQ
Qoo Qv Q Do 6% QQ
Q§ Qw Q Q Q QQ
Q Q8 Q2 Q Q QQ
_ v
Qd Q- Qt Q. Q- QQ
Q. Q? QQ QQ Q? QQ
QQ QQ QQ QQ QQ QQ
QQ oQQoQQhQQNQQaQQ

QQ QQVQQ—‘QQQQQNQQ
QQ ~QQ8QQOQQ°° QQMQQ
QQVQQMQQWQQ—‘QQOOQQ
QQQQQQQQQQQQQQQQQ
QQQQQQQQQQQQQQQQQ
QQQQQQQQQQQQQQQQQ
QQQQQQQQQQQQQQQQQ
QQQQQQQQQQQQQQQQQ
QQQQQQQQQQQQQQQQQ
QQQQQQQQQQQQQQQQQQ

00009

0.01547 -0.00002 0.00535 0.03131 000003 000023 000352 -
0.02188 0.03823 0.37132 0.09455 -0.00001 0.09812 -0.00001 -
0.02849 002788 002831 003360 032018 0. 00089 002716 -
0.00001 001593 009840 0.0 0.0 0.0 0.0 0.

Network conﬁguration and syaptic weights for the input layer
are omitted here. Only those for the hidden layer and output layers

0.06780 -0.01893 -0.10872 0.19052 -0.18966 -0.l6158 -0.14315
0.05473 -0.00001 -0.26931 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

0.00221 -0.04125 -0.12754 -0.00001 0.18903 0.00661 0.14060 -

0.00758 -0.00002 0.18032 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 -

0.00002

0.26599 002935 -0.00001 0.00868 0.13180 -020208 027351 -
0 05087 -0 01591 0 14653 0.20231 0 00004 0 09641 -0.04758
-0.07860 0.16221 0.01949 0.09311 -0.03029 -0.00422 0.13984 -
008432 -003299 -0.10703 0.15937 -0.02753 0.42001 0.03077 -
0.02721 -0.02125 -0.01576 0.07280 0.29793 0.09075 0.00194

0.18747 0.00001 0.26337 0.0 0.0 00.00 0.0 00 0.0 0.0 0.0 -

0.00001

0.06153 0.31482 0.06430 0.02372 0.04495 0.05690 -0.02502

D.4 10-digit Classiﬁcation Problem

 

 

M7

 

0000000000000000000000000000000000

QQQ
QQQ.
QQQ
QQQ

QQQ
v—QQ
QQQ

QQQ
QQQ
QQQ
Qv-‘Q
QQQ
Qv-4Q
QQQ

QQQ
QQQ.
QQQ

QQQ
—‘QQ
QQQ
Qv-‘Q
QQQ
v-‘v-Io
QQQ
F‘V-‘O
QQQ
QQQ
QQQ
Qv-‘Q
QQQ

QQQ
QQQ
QQQ
F‘QQ
QQQ

QQQ
QQQ
QQQ
Q—‘Q
QQQ
QQQ
QQQ
v—u—no
QQQ
Qv—Q
QQQ
QQQ
QQQ
QQQ
QQQ
QQQ
QQQ
v—QQ
QQQ
QQQ
QQQ

 

 

 

v—‘QQ
QQQ
”QQQ
QQQ
F‘QQ
§QQQ
“vac—IO
QQ

EFL—3O

QQQ
QQQ
QQQ
v—QQ
QQQ
v—nv—to
QQQ
Qv-‘Q
QQQ
QQQ
QQQ
Qv-‘Q
QQQ
v-‘QQ
QQQ
v-‘QQ
QQQ
QQQ
QQQ
Qv—Q
QQQ
QQQ
QQQ
v-‘QQ
QQQ
QQQ
QQQ
v—:v-IO
QQQ
v—n—tO
QQQ

F-‘F-‘o

QQQ
QQQ
QQQ
QQQ
QQQ
v-iv—to
QQQ
v—nw—tc
QQQ
Q—‘Q
QQQ
QQQ
QQQ
Qv-‘Q
QQQ
v—u-nO
QQQ
v-‘QQ
QQQ
QQQ
QQQ
v—u—ao
QQQ
Qv-‘Q
QQQ
QQQ
QQQ
QQQ
QQQ
F‘QQ
QQQ
v—Iv—uo
QQQ
v-‘QQ

QQQ
QQQ
QQQ
QQQ
QQQ
v-lv-Io
QQQ
v-‘v—dc
QQQ
QHQ
QQQ
QQQ
QQQ
Qv-‘Q
QQQ
v-‘QQ
QQQ
QQQ
QQQ
QQQ
QQQ
Q—‘Q
QQQ
Qv-‘Q
QQQ
v-tQQ
QQQ
QQQ
QQQ
v-‘QQ
QQQ
v-t—Io
QQQ

v-lv—IC

QQQ
QQQ
v—u—tc
QQQ
v—u—nc
QQQ

v-n-no

QQQ
QQQ
QQQ
QQQ
Qv-tQ
QQQ
v—u—tc
QQQ
v-IQQ
QQQ
Qv-ﬂQ
QQQ
v-4QQ
QQQ
Q-QQ
QQQ
QQQ
QQQ
Q—‘Q
QQQ
“QQ
QQQ
F‘U-‘C
QQQ
v-JQQ

Q Q QQQ
"QC MQQ
QdQ QQQ
QOQ Q~Q
QQQ QQQ
~OQ Q~Q
QdQ QQQ
~CQ Q~Q
QQQ QQQ
QQQ Q~Q
QdQ QQQ
QOQ ~~Q
QQQ QQQ
QCQ QQQ
QdQ QQQ
~OQ QQQ
QQQ QQQ
QQQ QHQ
QdQ QQQ
Q Q QQQ
QQQ QQQ
QQQ ~QQ
QQQ QQQ
~vQ QQQ
QQQ QQQ
v-‘QQ v-‘QQ
QQQ QQQ
QQQ QQQ
QQQ QQQ
~QQ QQQ
QQQ QQQ
s—u—ao v—uoo
QQQQQQQ
v-u—lo—lv—Aoo

QQQ
HQQ
QQQ
QQQ
QQQ
Qv-‘Q
QQQ
Qv-nQ
QQQ
Qv—Q
QQQ
v—u—ac
QQQ
QQQ
QQQ
QQQ
QQQ
Qv—Q
QQQ
QQQ
QQQ
v—QQ
QQQ
—‘QQ
QQQ
v-‘QQ
QQQ
Qv-iQ'
QQQ
QQQ
QQQ
v-3QQ
QQQ
QQQ

E—QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ QQQQ QQ
IQQQF‘QQQF‘OOQF‘F‘OQF‘QQQF‘QQQFN—‘QQOOQOF‘ QQQ; Q—J

 

M8

 

 

 

Q QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
Q v-‘QQ v-‘QQ QQQ v-‘QQ v-‘QQ '-‘QQ v-‘QQ v-‘QQ ~QQ “QQ
Q QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
Q v-‘QQ QQQ '-'QQ v-‘QQ v-‘QQ F‘QQ v-‘QQ v-‘QQ v-'QQ F‘QQ
Q QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
Q v—u—no QQQ v—u—no v—u—no v-Iv—IO QQQ. v-u-nc v—u—nc QQQ v-‘v-io
Q QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
Q QQQ QQQ QQQ QQQ Q—‘Q Q—‘Q Qv—Q Qv-‘Q Q—'Q Qv-‘Q
Q QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
Q v-‘QQ v-‘QQ v-‘QQ ~QQ Q—ﬁQ Qv-‘Q Qv-‘Q Qv-‘Q Qv-‘Q Qv-‘Q
Q QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
Q QQQ QQQ QQQ QQQ QF-‘Q Qv-'Q QQQ Qv-‘Q QQQ QQQ
Q QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
Q QQQ QQQ QQQ QQQ Qv-‘Q Qv-*Q Qv-‘Q Q—‘Q Qv-‘Q Qv-‘Q
Q QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
Q Qv-‘Q F‘v-‘Q ﬁ—‘IQ Qv-‘Q F'QQ —'QQ v-H-‘Q QQQ v-‘QQ v-H-‘Q
Q QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
Q QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
Q QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
Q v-‘QQ v-‘v-‘Q ~v-‘Q v-‘v—Q QQQ QQQ QQQ v-‘QQ QQQ QQQ
Q QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
Q QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ v-‘QQ
Q QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
Q QQQ QQQ v—QQ “QQ Qv-‘Q Qv-‘Q Q—‘Q QQQ Qv-‘Q v-‘v-‘Q
Q QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
Q Qv-‘Q v-‘v-‘Q v-‘v—Q Qv—Q v-‘QQ v-‘QQ v—QQ ﬁQQ v-‘QQ QQQ
Q QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
Q QQQ QQQ QQQ QQQ v-‘QQ QQQ QQQ v-‘QQ Qv-‘Q v-‘QQ
Q QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
Q Q—JQ v-v—Q v-QQ Q—‘Q v-‘QQ QQQ F'QQ v—QQ Qv-‘Q v-‘QQ
Q QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
Q QQQ Q—'Q QQQ Qv-‘Q v-‘QQ v-‘QQ v-‘v-‘Q F‘QQ v-‘v-‘Q v-H-‘Q
Q QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
Q Qv—Q v-‘v-‘Q v-‘QQ QQQ v-‘QQ v-<QQ v-‘QQ v-‘QQ v-‘QQ v—QQ

 

 

QQQ
v-‘QQ
QQQ
F‘QQ
QQQ
QQQ
QQQ
v—lv-lo
QQQ
v—QQ
QQQ
QQQ

Q~Q
QQQ
Qv-‘Q
QQQ
Qv—Q
QQQ
QQQ
QQQ
HQQ
QQQ
QQQ
QQQ
QQQ
QQQ
Qv-‘Q
QQQ
QQQ
QQQ
QQQ
QQQ
v—QQ
QQQ
v-‘QQ
QQQ

Pip-cc

Q~Q
QQQ
QQQ
QQQ
Q—‘Q
QQQ
QQQ
QQQ
v—QQ
QQQ
QQQ
QQQ
QQQ
QQQ
Qv—Q
QQQ
v—QQ
QQQ
QQQ
QQQ
v—QQ
QQQ
v—QQ
QQQ

v—n—:o

QHQ
QQQ
Qv-‘Q
QQQ
QQQ
QQQ
QQQ
QQQ
v-‘QQ
QQQ
QQQ
QQQ
QQQ
QQQ
v—n—to
QQQ
QQQ
QQQ
Q—zQ
QQQ
v-IQQ
QQQ
v—u—ac
QQQ

v—u—to

QQQ
QQQ
QQQ
QQQ
Qv—Q
QQQ
QQQ
QQQ
QQQ
QQQ
v-‘QQ
QQQ
v-‘QQ
QQQ
Q—‘Q

QQQ
Qv—Q
QQQ
v-3QQ
QQQ
QQQ
QQQ
v-JQQ
QQQ
v—QQ
QQQ
v-‘QQ

QQQ
QQQ
Q—‘Q
QQQ
v-‘v-to
QQQ
v—u—no
QQQ

v—Iv-ao

QQQ
QQQ
QQQ
v—QQ
QQQ
Q—'Q
QQQ
v-u—no
QQQ
Qv-‘Q
QQQ
v-:QQ
QQQ
v—u—tO
QQQ
QQQ
QQQ
v-‘QQ
QQQ
QQQ
QQQ
QQQ
QQQ
Q—‘Q
QQQ
QQQ
QQQ
Q—‘Q
QQQ
—tQQ
QQQ
v—QQ
QQQ
Qv-'Q

QQQ
~QQ
QQQ
v-iQQ
QQQ
v-u—nc
QQQ
QQQ
QQQ
F‘QQ
QQQ
QQQ
QQQ
QQQ
QQQ
v—u—ao
QQQ
QQQ
QQQ
~QQ
QQQ
QQQ
QQQ
QQQ
QQQ
v-IF:O
QQQ
QQQ
QQQ
QQQ
QQQ
QQQ
QQQ
Qv-3Q

~QQ
QQQ
QQQ
QQQ
QQQ
QQQ
v-u—ao
QQQ
QQQ
QQQ
v-‘v-lo
QQQ
QQQ
QQQ
QQQ
QQQ
q—u—cc
QQQ
QQQ
QQQ
QQQ
QQQ
QQQ
QQQ

v—u—ac

QQQQQQQQQQQQQQQQ QQQQQQQQQQQQQQQQQQQQQQQQQQ

o—‘d—nm—nd—‘v—‘md—ié—Joq— v-‘OOv—u-u—Qv—tm—to—u—IOO—I—v—no—n—u-tOv—tOv—a

 

M9

 

 

QQ
QQ
QQ
HQ
QQ
QQ
QQ
QQ
QQ
QQ
QQ
QQ
QQ
HQ
QQ
QQ
QQ
QQ
QQ
QQ
QQ
QQ
QQ
HQ
QQ
QQ
QQ
QQ
QQ
QQ
QQ
QQ
QQ
HQ

QQQ
QQQ
QQQ
QHQ
QQQ
QQQ
QQQ
HQQ
QQQ
QQQ
QQQ
QQQ
QQQ
Q—IQ
QQQ
HQQ
QQQ
HQQ
QQQ
QQQ
QQQ
HQQ
QQQ
QHQ
QQQ
QQQ
QQQ
HQQ
QQQ
HQQ
QQQ
v-3QQ
QQQ

v—u—Io

QQQ
QQQ
QQQ

~—u—Ic
QQQ
QQQ
QQQ
HQQ
QQQ
QQQ
QQQ
QQQ
QQQ
H—‘o
QQQ
QQQ
QQQ
v—Iv—IO
QQQ
QQQ
QQQ
HQQ
QQQ
v-u—:o
QQQ
HQQ
QQQ
HQQ
QQQ
HQQ
QQQ
HQQ
QQQ

v-‘v-‘O

QQQ
HQQ
QQQ

0 O 0

q—u—nc
QQQ
HQQ
QQQ
FJQQ
QQQ
QQQ
QQQ
QQQ
QQQ
v-iv—io
QQQ
HQQ
QQQ
HQQ
QQQ
QQQ
QQQ
QHQ
QQQ
v—u—to
QQQ
QQQ
QQQ
HQQ
QQQ
HQQ
QQQ
HQQ
QQQ

v—n—o

QQQ
QQQ
QQQ
QHQ
QQQ
QQQ
QQQ
HQQ
QQQ
QQQ
QQQ
HQQ
QQQ
QHQ
QQQ
HQQ
QQQ
HQQ
QQQ
QQQ
QQQ
v-‘QQ
QQQ
QHQ
QQQ
QQQ
QQQ
v—v—Io
QQQ
q—u—dc
QQQ
HQQ
QQQ

v-Iv-to

QQQ
HQQ
QQQ
HQQ
QQQ
QHQ
QQQ
v—n—no
QQQ
QHQ
QQQ
QQQ
QQQ
QHQ
QQQ
HQQ
QQQ
HQQ
QQQ
QQQ
QQQ
Q—‘Q
QQQ
QHQ
QQQ
HQQ
QQQ
QQQ
QQQ
HQQ
QQQ
v—lv-do
QQQ
HQQ

QQQ
HQQ
QQQ
v-3QQ
QQQ
HQQ
QQQ
v—u—ao
QQQ
QHQ
QQQ
QQQ
QQQ
QHQ
QQQ
HQQ
QQQ
HQQ
QQQ
QQQ
QQQ
QHQ
QQQ
QHQ
QQQ
HQQ
QQQ
QQQ
QQQ
HQQ
QQQ
v—u—nc
QQQ
HQQ

QQQ
v-iQQ
QQQ
HQQ
QQQ
QQQ
QQQ
v—u—no
QQQ
QHQ
QQQ
QHQ
QQQ
QHQ
QQQ
HQQ
QQQ
HQQ
QQQ
QQQ
QQQ
QHQ
QQQ
QHQ
QQQ
HQQ
QQQ
QQQ
QQQ
q—n—:o
QQQ
HQQ
QQQ
HQQ

QQQ
HQQ
QQQ
HQQ
QQQ
QHQ
QQQ
v—u—to
QQQ
QHQ
QQQ
QQQ
QQQ
QHQ
QQQ
HQQ
QQQ
HQQ
QQQ
QQQ
QQQ
v-:v-;o
QQQ
QHQ
QQQ
HQQ
QQQ
QQQ
QQQ
v—u—O
QQQ
QHQ
QQQ
QQQ

QQQ
HQQ
QQQ
HQQ
QQQ
HQQ
QQQ
r-u—ac
QQQ
QHQ
QQQ
QQQ
QQQ
QHQ
QQQ
HQQ
QQQ
HQQ
QQQ
QHQ
QQQ
m—‘O
QQQ
QHQ
QQQ
—:QQ
QQQ
QHQ
QQQ
HQQ
QQQ
HQQ
QQQ
HQQ

QQQ
HQQ
QQQ
v—uv—tc
QQQ
QQQ
QQQ
v-lv-lo
QQQ
v—u—uo
QQQ
QHQ
QQQ
QHQ
QQQ
v-iv-lo
QQQ
HQQ
QQQ
QQQ
QQQ
w-u—no
QQQ
QHQ
QQQ
HQQ
QQQ
QQQ
QQQ
u—d—nc
QQQ
HQQ
QQQ
HQQ

QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
QQHQQQHQQQHHQQHQQQHQHQHHHQHQHQHQHQHHHQHQ v-‘Q

 

 

QQQ
HQQ
QQQ
HQQ
QQQ
VHv-(O
QQQ
QHQ
QQQ
v—u—no
QQQ
QHQ
QQQ
QHQ
QQQ
QHQ
QQQ
QQQ
QQQ
HQQ
QQQ
QQQ
QQQ
v—cv—to
QQQ
HQQ
QQQ
HQQ
QQQ
HQQ
QQQ
HQQ
QQQ
HQQ

 

QQQ
HQQ
QQQ
HAG
QQQ
v—n—ao
QQQ
QQQ
QQQ
QHQ
QQQ
QHQ
QQQ
QHQ
QQQ
HQQ
QQQ
QQQ
QQQ
QQQ
QQQ
QHQ
QQQ
QHQ
QQQ
HQQ
QQQ
HQQ
QQQ
HQQ
QQQ
v—u—tO
QQQ

v—u—tc

QQQ
HQQ
QQQ
v—u—:o
QQQ
v-u—no
QQQ
QHQ
QQQ
QHQ
QQQ
QHQ
QQQ
QHQ
QQQ
HQQ
QQQ
QQQ
QQQ
QQQ
QQQ
QHQ
QQQ
QHQ
QQQ
HQQ
QQQ
HQQ
QQQ
v—u—to
QQQ
v—nv—ao
QQQ
HQQ

QQQ
HQQ
QQQ
v—u—to
QQQ
v—u—ao
QQQ
QHQ
QQQ
QHQ
QQQ
QQQ
QQQ
QHQ
QQQ
HQQ
QQQ
QQQ
QQQ
HQQ
QQQ
QQQ
QQQ
QHQ
QQQ
HQQ
QQQ
HQQ
QQQ
HQQ
QQQ
v—u—nc
QQQ

v—Iv—‘O

QQQ
HQQ
QQQ
v-u—nO
QQQ
v—u—nc
QQQ
QHQ
QQQ
v—u—to
QQQ
QHQ
QQQ
v—u—Io
QQQ
HQQ
QQQ
QQQ
QQQ
QQQ
QQQ
QHQ
QQQ
QHQ
QQQ
HQQ
QQQ
HQQ
QQQ
HQQ
QQQ
v—u—nO
QQQ
QHQ

QQQ
HQQ
QQQ
v—u—O
QQQ
wan—1O
QQQ
QHQ
QQQ
QHQ
QQQ
QHQ
QQQ
QHQ
QQQ
HQQ
QQQ
QQQ
QQQ
QQQ
QQQ
v-‘v—io
QQQ
v—u—nO
QQQ
HQQ
QQQ
HQQ
QQQ
v—u—to
QQQ
HQQ
QQQ
QQQ

QQQ QQQ QQQ QQQ Q
HQQ HQQ QQQ QQQ H
QQQ QQQ QQQ QQQ Q
HQQ HHQ QHQ QHQ Q
QQQ QQQ QQQ QQQ Q
HHQ QHQ QQQ QQQ Q
QQQ QQQ QQQ QQQ Q
QHQ QHQ HQQ HQQ H
QQQ QQQ QQQ QQQ Q
QHQ HHQ QQQ QQQ Q
QQQ QQQ QQQ QQQ Q
QQQ QHQ QQQ QQQ Q
QQQ QQQ QQQ QQQ Q
HHQ v—u—tc Ov—uc QHQ v-I
QQQ QQQ QQQ QQQ Q
HQQ HQQ HQQ QQQ H
QQQ QQQ QQQ QQQ Q
QQQ QQQ HQQ HHQ H
QQQ QQQ QQQ QQQ Q
HQQ HQQ QQQ QQQ Q
QQQ QQQ QQQ QQQ Q
QQQ QHQ QQQ QQQ Q
QQQ QQQ QQQ QQQ Q
QHQ QHQ QHQ QHQ H
QQQ QQQ QQQ QQQ Q
HQQ HQQ HQQ HQQ Q
QQQ QQQ QQQ QQQ Q
HQQ QQQ HQQ HQQ H
QQQ QQQ QQQ QQQ Q
HHQ QQQ HQQ HQQ H
QQQ QQQ QQQ QQQ Q
v—u—nc q—n—nc v—AOQ QQQ '—
QQQ QQQ QQQ QQQ Q
v—u—no QHQ v—tv—IQ v—u—to H!

QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ QQQQQQQQQ Q
v—u—Qv—u—n—cOv—u—cCOv—no—aOv—‘u—n—tO—t—nOQv—tO—no—u—n—ao AAOQ—IOOO—s v-:

 

150

 

 

 

wmuNQQ. wan: Q. 2 QQQ QmwQQQ QVQQQQ NmQQQQ QQQEQ
wavQQ mVNmQQ mthQQ meQQQ QQQ—Q NﬁmeQ thbQQ
bQVmQQ QSQNQ mQQmQQ manQQ :3on mmeQQ vaQQQ-
déRQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
Q.Q Q.Q Q.Q Q.Q Q.Q whom _ .Q thQQQ QELQQ 269 Q ovum Q Q Q.Q
QVNQQQ New: Q 9m _QQ QNNQQQ Q.Q- wthQQ- Qv: _Q QQQQQQ
mean—Q.Q QthNQ SQQQQ QNSQQ h: 5Q- VmeNQ QQQHQQ
WNSQQ wNQQNQ mchQQ VQQQQQ QQVmQQ VNQQQQ w-2Q
wwNQQQ NENQQ NGQQQ- QOQQQ QEmQQ SQQQQ hQVmQQ-

. QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
Q.Q Q.Q Q.Q Q.Q Q.Q QchQQ QQBQQQ- SVQQQ- QSNQQ- meQQQ

- :QQQQ QthQQ SNQNQ QQQ—Q.Q 3: _Q- Q.QQNQQ QQQmQQ
meSQ cmthQ- «HQNQQ cmwaQ VRQQQ 302 Q $wm~Q
QQQQQQ mama .Q Q3 wQQ momm— Q QwthQ :mQQQ QmwQQQ
m—SQQ Q3 _ —.Q thcQQ Q.Q- Q.Q- Q.Q ~QQQQQ —§.Q QQQQW.W
QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
Q.Q nXHQQQ humeQ. mGQ—Q ESQQ QENQQ SmVQQ QSQQQ
Q—QQQQ 5%on mQQNQQ QvaQQ mENQQ- mvmeQ QNQ~ Q.Q
QQQQQQ QQMQQQ- memQQ mVQQQQ wQZQQ ELQQ :wQ_Q
wNQQQQ- QNQQQ- wNQNQQ Q.Q- QQSNQ :NQQQ- QNNQQQ

- QQQNQQ :vQQQ wQ§.Q gem—Q $.0th mmwmQQ meQQOQQQm
QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
Q.Q NwmwQQ NQQmQQ mwmQQQ QQQVQQ hvchQ QOoQQ QQQQQQ
QmNQQAQ~QhQQAthvMQAvaQwQAQ_mmvongmhnogvmw_vogv
QQPnQQ C QQQ wBNNQQ QQQmQQ mmeQQ GSNQ mQQQQQ

- ENNQQ- wQ—wQQ vmwNQQ mvmm Q.Q mQNQQQ- QVQmQQ VQNNQQ
QWQNQAQNQVQQAQNQQQNAQQQQQQAQNVm_QAQVQVQQAQw_QmQAQ
Q£§QQQQQQQQQQQQQQQQQQQQQQQQQQQQ

Q.Q Q.Q Q.Q Q.Q Q.Q Q.Q wQNNQQ- SQ Q.Q Q Q 5Q :ZQQ NVwNQQ
- Q.Q- mmwSQ wwwmQQ- cmeQQ mthQQ mQQQQQ- mam—Q
w—BQQ- th—NQ mQEQQ- QmmwQQ QQQQQQ QE‘QQQ- QQmQQQ-
~§NQ SQ: Q QmNQQQ :bNQQ- mQNQQQ Q.Q- whthQ hwmmHQ
QmEQQ- QVQQQQ mmmNQQ c— QQQQ- mQQQQQ Q.Q Q.Q Nm‘hmwmvoom
QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ

Q.Q Q.Q Q.Q wQQﬁQ VQMVQQ- :mQQQ- wwwNQQ meﬁ .Q ~QQQQQ
thQQQ vcomQQ VNQmQQ SQQQQ- Q.Q meSQ QthQQ QQvQQQ
3.— mQQ- MW—wQQQ mQNmNQ ommme EQEQ me? —.Q meQQQ

QN— _ _.Q 23on Q QQQQ QQQQQQ NwmQQQ 059 .Q QNNVQQ
QNSQQ wwchQ QNQmQQ QQQ—Q.Q QQEQQ bow—Q.Q mme HQ

 

QQS _ .Q

Q.Q Q.Q Q.Q Q.Q Q.Q Q.Q Q.Q Q.Q Q.Q Q.Q Q.Q Q.Q Q.Q Q.Q Q.Q Q.Q Q.Q Q.Q
Q.Q NQONQ- «SEQ MQVQQQ QQQSQ awn—Q- Q.Q—2Q 3QN~Q
QQSSQ QQNQQQ mmdeQ- mem—Q QNQNQQ QWBQQQ QQQNQQ

- EQQQQ mhwzQ- mvaQQ mQhQQQ QQQ—Q thQQQ- QthQQ
vaQQ hem—Q NthQQ QSmQQ NthQQ vaanQ ~32 Q
NNmeQ QQQQQQ mhnhQQ- vaNQQ vaQQQ thmQQ thQQQ-

£3 :32 a? 822. a 22:5 05 8:3 saws; Basia -

O
O
O
I
O
O
O
O
O
O
O
O
O

~QQ ~QQ HQQ HQQ —QQ AQQ HQQ
QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ

I
O
O
O
O
C
I
O
O
O
O

v—n—no v-‘v—‘o v—u—nd v—u—IO v—n—ao

QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
HHQ HQQ HQQ HHQ QQQ HQQ HHQ QQQ

HQQ HQQ HQQ ~~Q HQQ QQQ ~~Q HQQ
QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
HQQ HQQ HQQ HQQ QHQ HQQ HQQ QHQ

O
O
O
O
O

~~Q HQQ ~~Q HQQ HQQ QHQ HQQ ~~Q
QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ

~~Q QQQ HQQ QQQ QQQ HQQ QQQ QQQ
QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ

QHQ HQQ HQQ ~QQ HQQ ~~Q HQQ ~~Q

O

O
O
C
O
O
O
C
O
O
I

~~Q HQQ HQQ HQQ ~QQ HQQ QQQ HQQ
QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ

O
O
O
O
O
O
O
C
O
O
O

HQQ QQQ QQQ HQQ QQQ

QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
QHQ QHQ QHQ QHQ QHQ QHQ QHQ QHQ

O
O
O
O
O

~~Q QQQ QQQ QQQ QQQ QQQ QHQ QQQ
QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
QHQ QHQ QHQ QHQ QHQ QHQ QHQ QHQ

v—u—ao v—u-no GAO v—u—dc v—u—Io

QQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQQ
v—nv—uv—IOv—av—n—aOv—u—u—no—IQv—aOv-u—u—to—u—n—IOv—nOv—IOv—u-u-ao'u

QQQ ~AQ Q~Q

v—:v—do

O
O
O
o
O

O
O
O
O
C
O
O

v—‘v—ﬂo

O
O
O
O
I
O
0

O
O
O

v-iQQ

QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
HQQ HQQ HQQ HQQ HQQ HQQ HQQ QHQ
QQQ HQQ HQQ HQQ QQQ HHQ HQQ QQQ
QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ
QQQ QQQ QQQ QQQ QQQ QQQ QQQ QQQ

QQQ QQQ QQQ

HQQ

 

 

151

 

Q Q QQQ
Q VG.” QQQ
QQ mQQN QQQ
QQSWQQﬁ QQQ
QQMRQQw QQQ
OONdOOO QQQ
QQdQQQC QQQ
OO'dOO§ QQQ
QQSoqqg QQQ
OONdCO.QCOO
QQNQQQ°9QQQ
QQddQQECQQQ
QQQQQQWQQQQ
QQWdQQRoQQQ
QQNQQQQQQQQ
OOﬁdOO oQQQ
QQOQQQ QQQ?
QogdooﬁoQQQ
QQmQQQﬁQQQQ
QongQQCQQQ
QQdQQQSQQQQ
OOﬁdOOOOOOO
QQVQQQVQQQQ
QQBdQQﬁoQQQ
QQNQQQOQQQQ
QQddooﬂoQQQ
QQ‘QQQOQQQQ
QQQQQQEOQQQ
QQOQQQQQQQQ
QQSNQQ oQQQ
Qquqquqqq
OOmmOOSFOOO
QQOFQQQGQQQ
QQﬁSQQQSQQQ
QQQNQQQQQQQQ
QQQQQQQQQQQ~

 

 

Q

QQ -
86o O
vQ . 3
00¢ GONG

00
00

QSQ w QchwaQQ
o m o "O o o w 0 am 0 o

6

<3\QQv-w".QQCD‘QQQSC‘!QQ YQQQdQQ

130.121
0%5
000
8
0
0
3
0
0
0.

3005 -0.05861
3n
8
0
b
0
1
0
0

O

mm§QQQ$CQQﬁooogoooﬁooothQ

0128

50

chQQEQQQQQQQEQQQ;QQQ;5QQ
OQOQOOOwoqcﬁocoonQwoOO

ooQQQMQQQo

M

05785()03

MO

-0m

NO
0 0 0m 0 O O O . O I
ﬁQQQvQQQaQQQ$QQQ QQQﬁcQQ

..dQQQQdQQQQQQQNQQQQQQQQdQQ

01969 0.07223 0

03058 0
09230

w 0 I I
ooQQQ

225170

0

1746 0.01009 0

1
0

0.17722 0.08684 0.04199 -0.04357 0.08628 0.01343 0.03651
0.00658 -0. 12145 -0.08413 -0.02417 -0.02417 0.00848 0.17315
000000000000000000000000000000000000
0.0 0.07090

0.05218 0.14246 0.10362 0.09479 0.12942 0.11250 0.09547

,QQQQQQQ.QQQﬁQQQNQQQNdQQ,QQQﬁdQQQQ
QQQQQOQQmOQQQOQQQOQQ.OQQQOQQQOQQ

oQQQ OOOmOOOOOOOOOOOO QQoo
m 85" m.. a..ﬁo..©c..g

aboogﬂoogﬁoomwoowgoom
”QQQVWQQ ”QQMWQQMNQQN~QQWWQQF~QQ
ooQQQN,°C‘§QQ,BQQQ3QQ§§QQ8QQQ8$QQWQQQQE
QQQQ?QQQ?QQQQQQQQQQQQ?QQQQQQQ°QQ.Q

Q
OI OI
CO

w—q.4§..§>
5m SQngQQQg
QQ hQQMVQQWmQQ NOOOVOOwhOOoqocmN
- " N " "QﬁQQSdQQdQ
..d ..woqqdoqqdoqq?oqq?oqqdoqq'°
0Q QQ QOORQOOvQOO QQQ¢dQQh.QdeQQ“Q
. ..m°qq~°QQ9°QQVoQQO .. ..
dthOOOQOOﬁQOONQOO QQQonQmQQdeQQEQ
'Qoqqqgoqqcoqqﬁoqq QQQﬁoQQOOQQQoQQdG
OQ dQQQdQQQQQQQQQQQdQQQOdQQQQQQQdQQwQ
°°£~qqquQQ~°QQNOQQ quonQooqqmoqq

Q
~SFOOOOOQOOWQOONQOO QOOmeOWQOOSdOO§g

QQWOQQ"
Q
OOOdOOgQ

. .. ..8°quoQQ8°
. _ d§OOQgOOC.OOSdcoﬁgooadOOﬁq
QOQQ' QQ- Q95 QQ' QQ'QQQN QQdQQQQ
QQQQ QQQOQQQ.QQQ9QQQ dOOOQOOmeOOQ
NQQON QQWooocoQQQOQQmoQQmOQQmoQQg
w QQdiQQVQ
WQQQNOQOQOCOwOOO OOO~oqu°QQ~oqqﬁ°
QQQddQQQQ

mqqgoQng
~QQowQQQS

MQOOOVOOOﬁVOOvaOoocomOOOVMOOV_Ome

w 6663;ddagddg~ddggddggddggddggddag
.OOQﬁQQQﬁQQQﬁ QQﬁQQQQQQQNQQQNﬁQQNQ

6671 0
6
1
0
0
0
l
6
0
0
7
1
0
0
1

. 8
0
0
3
2
0
0
7
1
0
0
1
1
O
0
5
2
O
0
9
2

0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
O
0
0
0
0
O

 

U2

 

BIBLIOGRAPHY

BIBLIOGRAPHY

[1] B. Widrow, “Generalization and Information Storage in Networks of ADALINE
Neurons,” in Self—Organizing Systems, Spartan Books, Washington DC, 1962.

[2] M. Minskey and S. Papert, Perceptrons, MIT Press, Cambridge MA, 1969.

 

[3] S. Grossberg, Studies of Mind and Brain: Neural Principles of Learning, Per-
ceptron, Development, Cognition, and Motor Control, Reidel Press, Boston,
1982.

 

[4] J.J. Hopﬁeld, “Neural Networks and Physical Systems with Emergent Collective
Computational Abilities,” Proc. Natl. Acad. Sci., USA, Vol. 79, pp. 2554-2558,
1982.

[5] J .J . Hopﬁeld, “Neurons with Graded Response have Collective Computational
Properties like Those of Two-state Neurons,” Proc. Natl. Acad. Sci., USA, Vol.
81, pp. 3088-3092, May 1984.

[6] D.W. Tank and J.J. Hopﬁeld, “Simple Neural Optimization Neural Networks:
An A / D Converter, Signal Decision Circuit, and Linear Programming Circuit,”
IEEE Trans. on Circuits and Systems, Vol. OAS-33, No. 5, pp. 533—541, May
1986.

[7] T. Kohonen, Self-Organization and Associative Memory, Springer-Verlag, Third
Edition, Berlin, 1988.

 

[8] DE. Rumelhart, G.E. Hilton, and R.J. Williams, “Learning Internal Represen-
tations by Error Propagation,” Parallel Distributed Processing, Cambridge, MA,
MIT Press, Vol. 1, pp. 318-362, 1986.

[9] EC. Hoppensteadt, An Introduction to the Mathematics of Neurons, Cam-
bridge University Press, 1986.

[10] D.W. Tank and J.J. Hopﬁeld, “Collective Computation in Neuronlike Circuits,”
Scientiﬁc American, pp. 104-114, Dec. 1987.

153

154

[11] RP. Lippmann “An Introduction to Computing with Neural Nets,” IEEE ASSP
Magazine, pp. 4-22, April 1987.

[12] C. Mead, M. Sivilotti, and M. Emerling, “A Novel Associative Memory Imple-
mented using Collective Computation,” Chapel Hill Conference on VLSI, pp.
329-342, 1985.

[13] NH. Farhat, D. Psaltis, and E. Paek, “Optical Implementation of the Hopﬁeld
Model,” Applied Optics, Vol. 24, pp. 1469-1475, 1985.

[14] J .P. Sage and RS. Withers, “An Artiﬁcial Neural Network Intergrated Circuit
Based upon MNOS/CCD Principles, ” American Inst. of Physics, Neural Net-
works for Computing, pp. 381-385, 1986.

[15] HP. Graf, W. Hubbard, L.D. Jackel, and P. deVegvar, “A CMOS Associative
Memory with Several Hundreds of Neurons,” American Inst. of Physics, Snow-
bird, Utah, pp. 182-187, 1986.

[16] HP. Graf, L.D. Jackel, and WE. Hubbard, “VLSI Implementation of a Neural
Network Model,” IEEE Computer, pp. 41-49, March 1988.

[17] HP. Graf and L.D. J ackel, “Analog Electronic Neural Network Circuits,” IEEE
Circuits and Devices, pp. 44-49, July 1989.

[18] W. Wike, D.V. derBout and T. Miller III, “The VLSI Implementation of
STONN,” Proc. IJCNN, Vol. III, pp. 529-534, 1990.

[19] S. Garth, “A Chipset for High Speed Simulation of Neural Network Systems,”
Proc. IEEE ICNN, Vol. III, pp. 443-452, 1987.

[20] J. Rasure, D. Hush, J. Salas, and M. Newell, “A VLSI Three Layer Artiﬁcial
Neural Network for Binary Image Classiﬁcation,” Proc. Int. Neural Network

Society (poster session), 1988.

[21] S. Kung and J. Hwang, “Parallel Architecture of Artiﬁcial Neural Network,”
Proc. IJCNN, Vol. II, pp. 165-172, 1988.

[22] CE. Atlas and Y. Suzuki, “Digital Systems for Artiﬁcial Neural Networks,” IEEE
Circuits and Devices, pp. 20-24, Nov. 1989.

[23] Y. Hirai, K. Kamada, M. Yamada, M. Ooyama, “A Digital Neurochip with
Unlimited Connectivity for Large Scale Neural Networks,” Proc. IJ CNN, Vol. II,
pp. 163-169, 1989.

155

[24] A. Moopenn, A.P. Thakoor, T. Duong, and SK. Khanna, “A Neurocomputer
Based on an Analog-Digital Hybrid Architecture,” Proc. IEEE ICNN, Vol. III,
pp. 479-486, 1987.

[25] S. Gilbert, “A Generic Architecture for Wafer-Scale Neuromorphic Systems,”
Proc. IJCNN, Vol. III, pp. 501-513, 1989.

[26] Y. Tsividis, S. Satyanarayama, “Analogue Circuits for Variable Synapse Elec-

tronic Neural Networks,” Electronic Letter, 1987.

[27] P. Miller, J .V. derSpiegel, D. Blackman, T. Chiu, T. Clare, J. Dao, “A General
Purpose Analog Neural Computer,” Proc. IJCNN, Vol. II, pp. 177-182, 1989.

[28] D. Nguyen and F. Holt, “Stochastic Processing in a Neural Network Applica-
tion,” Proc. IEEE ICNN, Vol. III, pp. 281-291, 1987.

[29] DP. Specht, “Probabilistic Neural Networks for Classiﬁcation, Mapping, or As-
sociative Memory,” Proc. IJCNN, Vol. I, pp. 525-532, 1988.

[30] A.F. Murray and A.V.W. Smith, “Asynchronous VLSI Neural Networks using
Pulse-stream Arithmetic,” IEEE J. of Solid-State Circuits, Vol. 23, no. 3, pp.
688-697, June 1988.

[31] DE. van den Bout and T.K. Miller, “A Digital Architecture employing Stochas-
ticism for the Simulation of Hopﬁeld Neural Nets,” IEEE Trans. on Circuits and
Systems, Vol. 36(5), pp. 732-738, May 1989.

[32] M.S. Tomlinson Jr, D.J. Walker, M.A. Sivilotti, “A Digital Neural Network Ar-
chitecture for VLSI,” Proc. 1.] CNN, Vol. II, pp. 545-556, 1990.

[33] J. Darnell, H. Lodish, and D. Baltimore, Molecular Cell Biology, Scientiﬁc Amer-
ican Books, New York, pp. 715-765, 1986.

 

[34] EM. Salam, “A Tutorial Workshop on Neural Nets and Their Engineering Im-
plementation,” 3lst Midwest Symp. on Circuits and Systems, St. Louis, Aug.
1988.

[35] L.O. Chue and G.N. Lin, “Nonlinear Programming without Computation,” IEEE
Trans. on Circuits and Systems, Vol.31, pp. 182-188, Feb. 1984.

[36] M.P. Kennedy and L.O. Chua,“Unifying the Tank and Hopﬁeld Linear Pro-
gramming Circuit and the Canonical Nonlinear Programming Circuit of Chua
and Lin,” IEEE Trans. on Circuits and Systems, Vol. 34, pp. 210-214, Feb. 1987.

156

[37] WE. Weideman, “A Comparison of a Nearest Neighbor Classiﬁer and a Neural
Network for Numerical Handprint Character Recognition,” Proc. IJ CNN, Vol. I,
pp. 117-120, 1988.

[38] T. Irino and H. Kawahara, “A Method for Designing Neural Networks Using Non-
linear Multivariate Analysis: Application to Speaker-Independent Vowel Recog-
nition,” Neural Computation, Vol. 2, pp.368-397, 1990.

[39] A. Guez and Z. Ahmad, “Solution to the Inverse Kinematics Problem in Robotics
by Neural Networks,” Proc. IJCNN, Vol. II, pp. 617-624, 1988.

[40] S. Hosogi, “Manipulator Control Using Layered Neural Network Model with
Self-Organizing Mechanism,” Proc. IJCNN, Vol. II, pp. 217-220, 1990.

[41] M. Kuperstein and J. Wang, “Neural Controller for Adaptive Movements with
Unforeseen Payleads,” IEEE Trans. on Neural Networks, Vol. 1, No. 1, March
1990.

[42] F. Rosenblatt, Principles of Neurodynamics, New York, Spartan, 1962.

 

[43] J. Hertz, A. Krogh, and RC. Palmer, Introduction to the Theory of Neural
Computation, Addison Wesley, pp. 89-111, 1991.

 

 

[44] M. Yu, “A Study of the Applicability of Hopﬁeld Decision Neural Nets to VLSI
CAD,” 26th ACM/IEEE DAC, pp. 412-417, 1989.

[45] SE. Fahlman, “Fast-Learning Variations on Back-Propagation: An Empirical
Study,” Proc. Connectionist Models Summer School, Pittsburgh, pp. 38-51, 1988.

[46] D. Plaut, S. Nowlan, and G. Hinton, “Experiments on Learning by Back Propa-
gation,” Technical Report CMU-CS-86-126, Dep. of Computer Science, Carnegie
Mellon University, Pittsburgh, PA, 1986.

[47] TR Vogl, J.K. Mangis, A.K. Rigler, W.T. Zink, and D.L. Alkon, “Accelerating
the Convergence of the Back-Propagation Method,” Biological Cybernetics 59,
pp. 257-263, 1988.

[48] RP. Gormann and T.J. Sejnowski, “Learned Classiﬁcation of Sonar Targets
Using a Massively-Parallel network,” IEEE Trans. on Acoustics, Speech, and
Signal Processing, pp. 1135-1140, 1988.

157

[49] Y. Le Cun, B. Boser, J .S. Denker, D. Henderson, R.E. Howard, W. Hubbard, and
L.D. Jackel, “Backpropagation Applied to Handwritten Zip Code Recognition,”
Neural Computation, pp. 541-551, 1989.

[50] Y. Le Cun, B. Boser, J .S. Denker, D. Henderson, R.E. Howard, W. Hubbard, and
L.D. J ackel, “Handwritten Digit Recognition with a Back-Propagation Network,”
Advances in Neural Information Processing Systems, Vol. 2, pp. 396-404, 1989.

[51] GE. Hinton and T.J. Sejnowski, “Optimal Perceptual Inference,” Proc. IEEE
Conference on Computer Vision and Pattern Recognition, Washington, pp. 448-

453,1983.

[52] GE. Hinton and T.J. Sejnowski, “Learning and Relearning in Boltzmann Ma-
chines,” Parallel Distributed Processing, Vol. 1, Chap. 7, Cambridge, MIT Press,
1986.

[53] K. Gutzmann, “Combinatorial Optimization Using a Continuous State Boltz-

mann Machine,” Proc. IEEE ICNN, Vol. III, pp. 721-734, 1987.

[54] S. Eberhardt, T. Duong, and A. Thakoor, “Design of Parallel Hardware Neural
Systems from Custom Analog VLSI ’Building Block’ Chips,” Proc. 1.] CNN, Vol
II, pp. 545-556, 1990.

[55] DB. Schwartz and RE. Howard, “A Programmable Analog Neural Network
Chip,” Proc. IEEE Custom Integrated Circuits Conf., IEEE Cat. No.:88CH2584—
1, pp. 10.2.1-10.2.4, 1988.

[56] Y. Tsividis and S. Satyanarayama, “Analog Circuits for Variable-Synapse Elec-
tronic Neural Networks,” Electronic Letters, Vol. 23, pp. 1312-1313, 1987.

[57] J. Raffel, J. Mann, R. Berger, A. Soares, and S. Gilbert, “A Generic Architecture
for Wafer-Scale Neuromorphic Systems,” Proc. IEEE ICNN, Vol. IV, pp. 485-
493, 1987.

[58] R. Hecht-Nielsen, Neurocomputing, Addison—Wesley, pp. 272-297, 1990.

 

[59] C. Mead, Analog VLSI and Neural Systems, Addison-Wesley, MA, 1989.

 

[60] M. Holler, S. Tam, H. Castro, and R. Benson, “An Electrically Trainable Ar-
” tiﬁcial Neural Network (ETANN) with 10240 ‘Floating Gate’ Synapses,” Proc.
IJCNN, Vol. II, pp. 191-196, 1989.

158

[61] DA. Pomerleau, G.L. Gusciora, D.S. Touretzky, H.T. Kung, “Neural Network
Simulation at Warp Speed: How We Got 17 Million Connections Per Second,”
Proc. IJCNN, Vol. II, pp. 143-150, 1988.

[62] BM. Forrest, D. Roweth, N. Stround, D.J. Wallace, and G. V. Wilson, “Im-
plementing Neural Network Models on Parallel Computers,” Computer Journal,

Vol. 30(5), pp. 32-41, 1989.

[63] R. Kuczewsk, M. Myers, and W. Crawford, “Neurocomputer Workstations and
Processors: Approaches and Applications,” Proc. IJ CNN, Vol. III, pp. 487-500,
1988.

[64] Y. Suzuki and L. Atlas,“A Study of Regular Architecture for Digital Implementa-
tion of Neural Networks,” Proc. IEEE Int. Symposium on Circuits and Systems,

Portland, May 1989.

[65] Y. Suzuki and L. Atlas, “A Comparison of Processor Topologies for a Fast Train-
able Neural Network for Speech Recognition,” Proc. IEEE Int. Conf. on Acoustic,
Speech, and Signal Processing, Glasgow, May 1989.

[66] Y.C. Kim and M.A. Shanblatt, “An Implementable Digital Multilayer Neural
Network (DMNN),” Proc. IJCNN, Vol. II, pp. 594-600, 1992.

[67] SW. Golomb, Shift Register Sequences, revised ed., Laguna Hills, CA, Aegean
Park Press, 1982.

 

[68] A.K. Jain, “Pattern Recognition,” Intl. Ency. of Robotics: Application and Au-
tomation, 1988.

[69] A Design & Test, “Behavioral Description Languages,” IEEE Design & Test of
Computer, pp. 56-68, 1990.

[70] J .R. Armstrong, “Chip-level Modeling with HDLs,” IEEE Design & Test of Com-
puter, pp. 8-18, 1988.

[71] C.J. Tseng, R.S Wei, S.G. Rothweiler, M.M. Tong, A.K.Bose, “Bridge: A Be-
havioral Synthesis System for VLSI,” IEEE CICC, pp. 261-264, 1988.

[72] G. Borriello and E. Detjens, “High-level Synthesis; Current Status and Future
Directions,” 25th ACM/IEEE DAC, pp. 477-482, 1988.

 

159

[73] M. Shahdad, R. Lipsett, E. Marschner, K. Sheehan, and H. Cohen, R. Waxman,
D. Ackley, “VHSIC Hardware Description Language,” IEEE Computer, pp. 94-
103, Feb. 1985.

[74] IEEE Standard VHDL Language Reference Manual, IEEE std. 1076-1987,
IEEE Inc., Mar. 31, 1988.

 

[75] VHDL User’s Manual, Volume 1: Tutorial, Intermetrics Inc., April 1987.

[76] R. Stansie, M. Brown, “VHDL Modeling for Analog Digital Hardware Design,”
ICCAD, 1989.

[77] R. Stansie, M. Brown, “Using VHDL as a Language for Describing the Behavior
of Analog and Mixed Analog/ Digital Systems,” VHDL Users Group, April 1990.

[78] Y. Xie and M.A. Jabri, “Analysis of the Effects of Quantization in Multilayer
Neural Networks Using a Statistical Model,” IEEE Trans. on Neural Networks,
Vol. 3, No. 2, pp. 334-338, Mar. 1992.

[79] C.Y. Maa and M.A. Shanblatt, “Linear and Quadratic Programming Neural
Network Analysis,” IEEE Trans. on Neural Networks, Vol. 3, No. 4, pp. 580-594,
July 1992.

[80] J .V. Neumann, “Probabilistic Logics and the Synthesis of Reliable Organisms
from Reliable Components,” in Automata Studies, Prinston, N .J., Princeton

University Press, pp. 43, 1956.

[81] BR. Gaines, “Stochastic Computing,” Spring Joint Computer Conf., Vol. 30,
pp. 149-156, 1966.

[82] ST. Ribeiro, “Random Pulse Machine,” IEEE Trans. on Electronic Computers,
Col. EC-16, No. 3, pp. 261-276, June 1967.

[83] P. Gupta and R. Kumaresan, “ Stochastic and Deterministic Computing with
Pseudo Random Sequence: Application to Digital Filtering,” 12th Annual Conf.
on Signals, System and Computers, Paciﬁc Grove, CA, pp. 159-163, Nov. 1986.

[84] B.W. Lindren, Statistical Theory, NY, Macmillan Company, pp. 59-75, 1962.

 

[85] Y.H. Pao. Adaptive Pattern Recognition and Neural Networks, Addison Wesley,
pp. 3-21, 1989.

 

 

 

 

 

 

 

 
 
  
 
 

 

  

 

 

 
 

 

 
 

 

. ,., .. . S
. . .. . , E
y . .. . . _ , An
4 : _ . 7 R
, . _. , B
. .. . .. T.
n .. I . ., N
. . . . 1.. U
H . . T
. Q
. . J T
. . AH]
.. I . G H
I III
H
..H.. . 3 c
_.., . . I
., . . . , H
r .. V. ...,w .9 .2
. . . .y;
H. .. .. ~. .. . .

v, ,,’Jyl

 

 

 
 

 

 

 

 

 

 

 

  

1...
..