-'{lc

 

 

ulszwnffiflx‘“; .

 

 

 

 

 

.
:sz 92‘.

I

I
‘ “$1., M
(“swqdumr 1

:1“
.
'I
w. y L . ”AW“

@4943: ism

Mai

.1425! ‘ll.l“f‘

 

.p :1 i,’

Afgrfit‘ Jr

.1" C.“

.a<\u

"3mg” :33

;
.. “43'
~ «3“.‘8'

....anruu:"‘
”an. -_--‘~«
’3‘“

 

‘0‘” « 13‘;
My} 334%,
$233153”

‘ls
- 122:“: *“ I '
f5;‘-“J'I‘:f‘ I - ‘ I .. . ~, .5
A 1" _ . w , . 3” ‘ ““29 “ml" "
~ ‘ . _..uu
a ~-u:».-u

'1
V5.2. .

n m. .
.1 JV,“ '
17w

2%?

1“.
Nik'yumeu'
,1 «ml.

l'hyili:

0
I .‘ l
r‘.’ "

{ii'lfflvmuu ‘

O

L
r '36‘ a

m... 4
.wynfiflfih

 

 

 

~ 23;: ‘ '19,“
w. : > . 1 v ,..Xn.~u in“
~ A: I anruu.~..u g
""4?“
is ' 5P1. ‘9 11’ - r;

«W #311;
E‘E'ALE “$5"?

 

3: H A 'M '25:?" V iii. '“E‘Et
I - . I ' r I. V‘ >~ ‘1’. ' '
_,.‘:4n:;':u- I . “ v‘i ‘ . fé:§€§3é;‘¥2‘ “
.3 2%: me ,_ ~ 1- " 4‘3“ {‘7’
I .. gates; ,.
(in?

4:) «us: '
”In?! .. "

 

 

134» ‘x,i." ‘ 4 u ‘
1.. ‘. INN; "*W‘iy'ffgl‘,“
5a

hv'uok'j’: - ‘ ‘
(é?

u‘éfi‘fi?‘
§

”1’!
_i.....,:. nu“
"'31.“

‘1
¢rv:-‘o-I"r.|“"

W’

 

LEE.»

_J ' 1‘ . '
“‘ELDI ‘5-

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

IIIIIIIIIIIIIIIIIIIIIIIIIII IIIIII

3 1293 00910 86I75

This is to certify that the

dissertation entitled .

Vision Guided Robotic Tractor: The
Development And Operation Of An Engineering
Prototype

presented by

Bernard Wayne Fehr

has been accepted towards fulfillment
of the requirements for

Ph.D. degreein Agric. Tech. & Sys. Mgt.

 

1% 52:12“;

Ma jor professor

Date W

MS U i: an Affirmative Action/Equal Opportunity Institution 0-12771

USNIVER ITY LIBRARI (x , _..‘

III

’5 .
—“" a" Q

 

 

 

I LIBRARY *—
Mlchlgan State
University

L_‘

 

PLACE IN RETURN BOX to remove this checkout from your record.
TO AVOID FINES return on or before date due.

M DATE DUE DATE DUE DATE DUE

‘—

5-5

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

fiIsI I

MSU Is An Afiinnative Action/Equal Opportunity Institution
omens-9.1

._. y“.

VISION GUIDED ROBOTIC TRACTOR:
THE DEVELOPMENT AND OPERATION OF

AN ENGINEERING PROTOTYPE

BY

Bernard Wayne Fehr

A DISSERTATION

Submitted to
Michigan State University
in partial fulfillment of the requirements
for the degree of

DOCTOR OF PHILOSOPHY

in
Agricultural Technology and Systems Management
Department of Agricultural Engineering

1990

é¢5~ 7/25-

ABSTRACT

VISION GUIDED ROBOTIC TRACTOR:
THE DEVELOPMENT AND OPERATION OF AN ENGINEERING PROTOTYPE

BY

Bernard Wayne Fehr

An investigation was made into the practical use of
color machine vision to guide a small tractor, expected
benefits to include improved: 1) operator performance, 2)
field efficiency, and 3) safety. A technological approach
was used to assemble a Vision Guided Robotic Tractor (VGRT)
that could follow a crop row. The VGRT consisted of a
master computer which received video imagery of the path of
travel. From this imagery, the master computer determined
a control signal used to steer the tractor. The steering
control signal was sent from the master computer, via a bi-
directional parallel interface, to a slave computer. The
slave computer monitors the functions of the tractor and
receives the control signal from the master computer. The
slave computer uses the control signal to stop or start
tractor motion as well as determine positioning for the
front wheels.

The VGRT was tested for its ability to follow well
defined lines, lines with noise, and crop rows in a field.
Line following was done in varying light and temperature

conditions. Two types of line followers were developed:

1) contrast within the color band of best discrimination,
and 2) self-calibrated to the color of the desired line.
Both methods utilized floating windows to enhance rate of
position update performance. The steering performance of
the VGRT was compared to the steering performance of human
operators on the same tractor. The development of the VGRT
even included a "learn" mode whereby the VGRT could
assimilate distance and direction information necessary to
repeat a left or right hand headland turn.

The results of the study produced a relatively
inexpensive VGRT that could follow well defined lines at
speeds up to 9.5 km/hr, and crop rows at speeds up to 4.0
km/hr. The VGRT could make either right or left hand
headland turns, locate and follow parallel lines. The
steering performance of the VGRT was comparable to that of
human operators. The operation of the VGRT was also simple
enough for inexperienced operators to master use of the

system in approximately one hour of self instruction.

Approved:

filo/4.2.44;

M r Professor

/“

/

f' A
I

it ‘ , .
73 (Z—CCL" r5487” ’1 L /. 11k
Departmeng C irperson"
\Jf/

Copyright by
Bernard Wayne Fehr

1990

DEDICATIONS

To, Susan Fehr, my Wife who was the reason for finishing
this, and getting on with life,

To, Walton and Mida Fehr, my parents who gave me the
motivation to start this, and the discipline to complete

To Mrs. Hix and Miss Beer, teachers who, many years ago,
were the beginning of all this,

To Les Sheets, who once gave me enough rope to hang myself
and we both found out it wasn’t long enough,

To Norm Reese, the one person in the Agricultural
Engineering department at MSU who is truly qualified to
hood me at graduation,

To Mary Carlson, a bright ray of sunshine at MSU
Agricultural Engineering who truly can be counted on to
pleasantly and efficiently get things done,

To the ROTC Department at MSU in the charge of Sergeant
Major Waters, the one department on campus concerned enough
to give me appropriate space in which to work,

To Bob and Joan Wendt, my in-laws, some very special people
who provide number one (see above),

To Brian and Brenda Robertson, Bill DeGraff and Dr. Linda
Donoghue, very good friends who helped keep this whole
thing in perspective,

To the Millers; Henry, Betty, Carol, and Elaine, some very
special people who gave me back the way of farming that I
have known and loved,

To Norm Shoopman and Chuck Walters, who gave me my first
inspiration in electronics,

To Mr. Ferree who instilled in me a deep desire to know my

significant digits, and to Mr. Geshiwlm who taught me the
"nuts and bolts" of things.

Soli Deo Gloria

ACKNOWLEDGMENTS

I would like to thank Dr. Gerrish for serving as
advisor on my Ph.D. committee and also to Dr. Surbrook from
Agricultural Engineering, Dr. Shanblatt from Electrical
Engineering and System Sciences, and Dr. Stockman from
Computer Science for serving on my graduate advisory
committee.

Many thanks are also due to the following:

Norm Reese, Dr. Surbrook, Dr. Welch, and Frank
Galbavi, for the use of their lab space,

Russ Rivet, Sergeant Mgr. Waters, and Tom Campbell,
for testing space from outside the Agricultural Engineering
Department,

Detroit Edison for their contribution of the electric
tractor and extra attachments,

Massey-Fergeson for the initial funding of this
project,

Dr. Brown (USDA) MSU Agricultural Engineering for the
loan of equipment for use on this project,

Bary Darling for use of MSU farm fields for testing
purposes,

Dr. Bud Belcher for use of his specially drained field
for testing purposes.

Solvitur ambulando

vi

TABLE OF CONTENTS

LIST OFTABLESOOOOOOOOO..OOOOOOOOOOOOOOOOO0......O... X
LIST OF FIGURES.......................... ..... . ...... Xi

LIST OFABBREVIATIONS...’00.0.0.0...OOOOOOOOOOOO... Xiii

CHAPTER
I. INTRODUWIONOOCOCOOOOOOOOOO..OOOOOOOOOOOO ..... l
1.0 INTRODUCTIONOOOOOOOOO0.000....O .......... 1
1.1 GOAL OF THIS RESEARCH.................... 5
1.2 OBJECTIVES OF THIS RESEARCH.............. 7
1.3 LIMITATIONS OF THIS RESEARCH...... ..... .. 7
II. LITERATURE REVIEW. ............................ 8

2.0 INTRODUCTION............................. 8
REVIEW OF GUIDANCE IN AGRICULTURE........ 9
VISION GUIDANCE IN AGRICULTURE........... 16
NON-AGRICULTURAL VISION GUIDANCE -

ROAD FOLLOWING........................... 19
EDGE DETECTION........................... 23
COLOR DESCRIPTION AND PROCESSING......... 26
RANGE FINDING............................ 28

MN”
0 o o
UMP

NNN
00.
GUI-b

III. METHODOLOGY....... ............................ 33

INTRODUCTION............................. 33
TECHNOLOGICAL VS. ENGINEERING APPROACH... 33
CONCEPTUAL MODEL......................... 36
OPERATIONAL MODEL........................ 40
THE TEST TRACTOR......................... 41
SOLID STATE VIDEO CAMERA AND CALIBRATION. 45
SLAVE COMPUTER........................... 48
.1 SLAVE COMPUTER HARDWARE................ 49
.2 SLAVE COMPUTER SOFTWARE................ 51
MASTER COMPUTER.......................... 55
.1 MASTER COMPUTER HARDWARE...... ..... .... 55
.1.1 CUSTOM POWER SUPPLY..... ..... . ....... 56

uuUUUUUUUUUU
000000000000
\JQQO‘GGU‘fiUNPO

vii

IV.

3.7.1.2 CUSTOM I/O BOARD..................... 59
3.7.1.3 VIDEO DIGITIZING BOARD............... 60
3.7.2 MASTER COMPUTER SOFTWARE............... 62
3.7.2.1 INITIALIZATION OF CONTROL PARAMETERS. 64
3.7.2.2 STEERING CONTROL..................... 69
3.7.2.3 COMMUNICATIONS SOFTWARE.............. 71
3.7.2.4 ROW CENTER, OR EDGE DETECTION
SOFTWARE............................. 75
3.7.2.4.1 CONTRAST MODE ROW FOLLOWER......... 79
3.7.2.4.2 COLOR DISCRIMINATION MODE ROW
FOLLOWER........................... 83
3.7.2.5 HEADLAND TURNS....................... 87

RESULTS AND DISCUSSIONOOOOOOO..OOOOOOOOOOOO... 88

4.0 INTRODUCTION............................. 88
4.1 CONTINUOUS LINE TESTS.................... 92
4.1.1.1 FIXED STEERING GAIN TESTS............ 92
4.1.1.2 VARIED STEERING GAIN TEST............ 97
4.1.2 CONTINUOUS LINE STEP TESTS............. 98
4.1.2.1 RESULTS OF RIGHT AND LEFT STEP.......100
4.1.2.2 RESULTS OF COMPARISON STEP TESTS.....104
4.1.3 RUNOUT TESTS...........................105
4.1.3.1 RESULTS OF RUNOUT TESTS. ..... ........106
4.1.4 VGRT TILT TESTS AND RESULTS............108
4.1.5 PIXEL WIDTH OFFSET TESTS AND RESULTS...109
4.2 INTERMITTENT LINE TEST...................110
4.2.1 INTERMITTENT LINE TEST RESULTS.........111
4.2.2 FUZZY ROW TESTS AND RESULTS............112
4.2.3 CROP-ROW-FOLLOWING TEST AND RESULTS -

CORN...................................114
4.3 HEADLAND TURNS AND RESULTS OF TESTS......116
4.4 POSITION UPDATE RATE TESTS AND RESULTS...117
4.5 PERFORMANCE OF VGRT AS A SYSTEM..........118
4.5.1 RESULTS OF PERFORMANCE TESTING.........121
4.5.2 INEXPERIENCED OPERATOR PERFORMANCE

TESTS..................................122
4.6 CAMERA TESTS.............................124
4.6.1.1 LENS VARIATIONS......................131
4.6.1.2 HUE, SATURATION, AND INTENSITY

VARIATIONSOOOOOOOOOOO00......O...0.0.131
4.6.2 CAMERA TEST RESULTS....................132

SUMMARY, CONCLUSIONS, AND RECOMMENDATIONS.....134

5.0 INTRODUCTION.............................134
5.1 SUMMARY..................................134
5.2 CONCLUSIONS........... ................... 135
5.3 RECOMMENDATIONS..........................138

viii

APPENDICESOOOOOOOOOOOOOOOOOCOOOOOOOOOOOOOOOO0.00I.00.140

1.
2.
3.
4.

Inexperienced Operator Driving Test.......l40
Custom AT DC to DC Power Supply...........142
Custom I/O Board..........................144
Slave computer............................l46

LIST OF REFERENCESOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO ..... 150

ix

LIST OF TABLES

Page
Results of camera calibration......... ..... ... 48
Steering Variability - 8 Updates/sec.......... 94

Steering Variability
Steering Variability
Steering Variability
Steering Variability
Steering Variability

Steering Variability

Step Comparison Tests.

Step Comparison Tests.

Step Comparison Tests.

4 Updates/sec.......... 94
2 Updates/sec..... ..... 95
l Update/sec........... 95
0.5 Updates/sec........ 96
Human Operators........ 97
Varied Steering Gain... 98
Speed 0.6 Km/hr. ...... 104
Speed 1.8 Km/hr.......105

Speed 4.0 Km/hr ....... 105

Results of Runout Tests.......................107

VGRT Tilt TestOOOOOOOOOOOOOOO0.00.0000........109

Results of Pixel Width Tests..................110

Steering Variability - Intermittent Line......111

Steering Variability - Fuzzy Row..............114

Position Update Rates vs. Delay Value.........118

User Friendly Test Times......................124

Camera High Light Level Tests.................133

LIST OF FIGURES

Figure Page
3-1 Side View Of VGRTOOOOOOOOOOOOOOOOOO.... ....... 34

3-2 Block diagram of VGRT......................... 38

3-3 Top-down view of linear actuator placement.... 43
3-4 Front view of wheel angle potentiometer ....... 44
3-5 Electrical diagram of transistor buffers ...... 46
3-6 8751 port listing............................. 50
3-7 Electrical diagram of custom power supply ..... 58

3-8 Block diagram of the custom I/O board......... 61

3-9 Side view of VGRT. Note camera nadir is not
perpendicular to ground plane ...... . .......... 65

3-10 Flow diagram of guidance control parameters... 66

3-11 Tangent curve approximation ................... 67
3—12 Top-down view of VGRT ready for operation ..... 68
3-13 Diagram of triangle used to determine

steering angle................... ............. 70
3-14 Procedure for mapping Y coordinate value ...... 72
3—15 Procedure for mapping X coordinate value ...... 73
3-16 Procedure to determine steering angle... ...... 74

3-17 Flow chart of.communications exchange......... 76
3-18 Communications procedure with slave computer.. 77

3—19 Communications procedure with slave computer,
continued.......OOOOOOOOOOOOOOOOOO 000000000000 78

xi

Flow chart of executive loop.. ...... .......... 80
Contrast mode row following............ ....... 82
Color discrimination mode row following. ...... 86
View of second camera placement, used to

record tractor path of travel, and distance

"tiCR" marks.......OOOOO......OOOOOOO.....0... 90

Graph of steering response, right and left
step. Gain = 0.5....................... ...... 101

Graph of steering response, right and left
step. Gain=1.0.0.000000000000000000... ..... 102

Graph of steering response, right and left
step. Gain=2000000000000000000 0000000000000 103

Fuzzy row test, simulated crop row and weed
bIObOOOOOO.........OOOOOO 0000000000 ... 00000000 113

View of camera placement on tractor and
incident high level light problem.............126

Two color camera test paper............. ...... 128

Placement of camera test paper to simulate
up-sun field lighting conditions ...... . ....... 129

Bimodal distribution used in histograms to
rate camera resolution improvement methods....130

xii

LIST OF ABBREVIATIONS

adc = analog to digital converter

cm = centimeter

d = dimensional

df = damping factor

eprom = erasable programmable read only (memory)
F (OF CAMERA) = F/STOP

HSI = Hue, Saturation, Intensity

m = meter

k = kilo (byte)

km = kilometer

nf = natural frequency

mm = nanometer

NTSC = National Television System Committee
po = percent overshoot

ram = random access memory

rt rise time

sd standard deviation

VGRT = Vision Guided Robotic Tractor

xiii

CHAPTER I

INTRODUCTION

1.0 INTRODUCTION

The rapid advance in technology is starting to catch
up to agriculture. As the market for farm products becomes
worldwide and more complex, and profit margins decrease
with the rising cost of machinery, farmers find that they
have to "farm smarter," not just "farm harder." In order
to earn more money, a farmer will try to farm larger
amounts of land requiring an increased capital investment
in tools. These tools would certainly include larger farm
machinery, but would also include greater use of farm
management services and possibly even a home/farm computer.
Agriculture is rapidly entering the let century and
farmers need to keep pace with computerized advance.

Unfortunately, new technology has yet to reach the
point where it can protect the farmer from the dangers of
operating large powerful equipment. The safety
aspect of farming continues to be of vital importance. As
an occupation, agriculture continues to rank as the most
dangerous in the nation; the operation of agricultural
machinery is the most dangerous component (Doss, 1988).
Operating larger equipment requires increased amounts of

concentration, but the attention span of the human operator

remains the same. New technology should be used to improve
safety by aiding the operator in basic field operations.
Murphy (1988) expounds on the development of tractors and
mechanization in agriculture and the quest for a guidance
system to augment human capabilities.

A recent farmer survey has shown that human error is
one of the main reasons for errors which result in
accident. Fatigue was listed as having the greatest
adverse effect on operator performance (Smith, 1987). A
tired operator may make an error in judgment which would
not be made under more alert operator conditions. Perhaps
technology can be used to reduce the stress of operating
equipment so that the operator can remain alert longer and
make fewer dangerous mistakes. Technological aids could
also free the operator from monotonous functions so that
more time could be allocated to efficiently managing the
field operation. Palmer and Matheson (1988) estimate that
as much as a 10% reduction in cost of crop production could
be realized by improving the navigation accuracy of field
equipment. It would seem that farming is ready for
improved technology. But, is improved technology ready for
farming?

Goody (1982), in his book "The Intelligent
Microcomputer" likens the activities of the microprocessor
and peripheral chips, which are the heart of today's

computers, to the basic sensory and decision-making

functions of a human. Perhaps a computer, which can mimic
certain functions of a human, could be useful in helping
the farmer to more safely and effectively operate farm
equipment. The computer would seem a natural device to
take over some of the monotonous tasks that now belabor the
farmer. Chief of these tasks is the repetitious and highly
demanding operation of field machinery. Driving a tractor
back and forth across a field requires constant attention
which has to be split between steering the tractor and
monitoring the equipment the tractor is powering. A study
done by Kaminaka, Rehkugler, and Gunkel (1981) shows the
degradation in steering accuracy that results from
monitoring a towed implement. A useful technological aid
based on a computer could relieve the tractor operator of
the monotony of steering the tractor.

One of the advantages a computer has over a human is
that the computer can repeat the same operation many times
for extended periods with no degradation in performance.
The human operator, while poor at repetitious tasks, is
better than the computer at making complex and relatively
infrequently occurring decisions. By applying the theory
of speed-load stress, Becker, Shoup, and Sinden (1983)
showed that guidance may be the single most significant
load on the mental facilities of an operator performing a
field operation. Allowing a computer to take over the
demanding repetitious tasks, such as steering, while

leaving the complex but less frequent tasks to the human

operator would offer an improvement in both field
efficiency and safety for the farmer.

Many guidance systems have been developed for tractors
which have an inherent defect preventing them from being
truly successful in a variety of situations. Most of the
present guidance systems sense the tractor position from a
point that is in direct proximity to the tractor. That is,
the sensing mechanism determines the steering response of
the tractor from a point that is somewhere near the front
of the tractor, or in one extreme case, from behind the
tractor (TRI-R Innovations, 1988)*. These systems may be
contrasted with the human operator who looks ahead of the
tractor to determine the steering response. This ability
to "look ahead" provides an anticipatory mode of operation
which is inherently stable.

Using an anticipatory guidance method, the operator
can discern whether the tractor is going to deviate from
its intended course before it actually does so to any great
extent. The human operator can anticipate that, at the
present heading, the tractor will start to leave its proper
course. On the other hand, when determining the steering
response from a point too near the tractor, the tractor
actually has to leave the proper course before the sensing

device can determine that a steering error has occurred.

 

* Any mention of a brand name throughout this paper does

not necessarily constitute an endorsement of that product.

Fehr (1986) discussed the effect of moving the steering
reference point and the improvement in accuracy realized by
providing an anticipatory mode of operation.

One solution to the problem of providing an
anticipatory guidance system for agricultural tractors lies
in using machine vision. A video camera positioned to
provide a forward field of view coupled with a computer
could mimic the function of the human operator. Gerrish
and Surbrook (1983) studied the potential accuracy of using
machine vision guidance to steer a tractor. They showed
that present technology for a camera, and computer, is
accurate and fast enough to provide steering information
for a row-crop operation. Based on theoretical work it
would appear that vision technology is indeed ready for

agriculture.

1.1 GOAL OF THIS RESEARCH

The goal of this research effort is to evaluate vision
technology in the agricultural environment by fabrication
and testing (as opposed to mathematical analysis and
modeling). The success of this research project will be
measured more by the performance of a working prototype
rather than by analysis on paper.

A performance-oriented approach will be used to
develop a test tractor to follow high-contrast lines or

crop rows using computer vision as a sensor. An important

portion of the research effort will be directed at
developing a system that is reliable. This system will be
used to evaluate the ability of machine vision to guide a
tractor. The main thrust of the research will not be to
develop new algorithms to distinguish lines (hereafter
lines may refer to constructed test lines or actual crop
rows in a field), but to develop a discrete-control method
of following a well-defined line using feedforward
techniques. The ability to follow a line will be evaluated
by introducing "noise" into the line, by "ringing" the
system (using a step input), and by comparing tractor
performance under computer control with performance under
human control. To further emulate the agricultural
setting, the ability of the tractor to negotiate headland

turns will also be studied.

1.2 OBJECTIVES OF THIS RESEARCH

To accomplish the above goals the following objectives
were developed to direct the study. This research project
will be completed when the following objectives are
undertaken and accomplished or proved unfeasible.
1) Procure an affordable vision system and install it on
the test tractor.
2) Develop the system to follow a well defined line.
3) Develop the system to undergo a 180 degree turn, locate
a parallel line and follow it.

4) Make the system "user-friendly".

1.3 LIMITATIONS OF THIS RESEARCH '
Restricted funding and equipment availability limited
the research to development on an electric lawn tractor.
This reduced the testing of the unit in actual field
conditions. The short wheelbase of the lawn tractor
limited field speeds to about four km/hr. Test conditions
were created both indoors and out of doors to evaluate the

system.

CHAPTER II

LITERATURE REVIEW

2.0 INTRODUCTION

The amount of literature relating to vehicular
guidance is becoming voluminous, an indication of the great
amount of effort being applied to this field of study.

This research venture is yet another step in the direction
of complete autonomous vehicle operation. At this stage of
technological development, it is felt that guidance should
be employed to assist an operator. The human operator
would still be on the vehicle but could now free up his
attention to devote more time to other tasks. The
advantage of having an operator on board is reflected in
guidance controllers of a reduced complexity, and human
intervention in the event of a guidance system failure.

The automatic guidance system developed for this study
was further simplified by the nature of its intended use.
Following crop rows or crop edges implies a relatively
great deal of ’a priori’ knowledge. The crop rows or crop
edges are relatively straight, of a high contrast nature
(vegetative material and soil background), and are the
intended path of travel for the vehicle. The vehicle in
this case would be a row-crop type tractor or harvester

such as a combine.

2.1 REVIEW OF GUIDANCE IN AGRICULTURE

Young in 1976 published a monograph which summarizes
the developments relating to guidance of an agricultural
tractor to that date. The monograph began with an
overview of different methods of guiding the tractor.
Young's survey of guidance methods is still a definitive
work.

To summarize, Young (1976) provided the following

outline of guidance methods:

Indirect vehicle guidance by man

a. Guidance by master vehicles
b. Guidance by radio control

Vehicle guidance on directrixes

a. Guidance with a directrix generated by the
previous operation

1) Guidance by means of mechanical contact

2) Guidance by means of non-contact scanning

3) Guidance by means of mechanical contact or
non-contact scanning and an additional angle
measurement

b. Guidance by a directrix installed in the field
1) Guidance by a mechanical conductor device
2) Guidance by a conductor cable using
alternating current
3) Guidance by a light beam

c. Guidance by a directrix calculated with the aid
of fixed points

Vehicle guidance by image recognition

Vehicle guidance by man with semi-automatic steering
(P- 3)

10

As Young explained the different systems that were
possible for automatic guidance, he indicated that the
methods involved were to be an aid to the tractor operator,
not a replacement for the operator. More than 10 years
later Young's work seems amazingly prophetic. He predicted
that with the advent of increased computer power in smaller
packages, image recognition guidance will be feasible.

This type of guidance would not only be able to follow the
directrix, but would also be able to find it as well. This
ability would be a significant improvement over previous
methods that could only follow the directrix.

Since 1976 much new work has been accomplished in the
category of agricultural vehicle guidance. In his 1981
study of a robotic lawn tractor, Squires discussed the
classification of guidance systems into three basic types.
These three groups were: semi-automatic, automatic command,
and automatic preset.

The semi-automatic system could remove the operator
from a hazardous location, but did not decrease the
attention required for the steering task. The semi-
automatic system could include radio control and master-
slave arrangements. The automatic command system required
highly accurate component systems to provide steering
control from a remote location. The automatic command
system resulted in a relatively expensive system that
required off-vehicle location of control systems. The

third guidance system, automatic preset, had the advantage

11

of a steering control system that was completely contained
on the vehicle. Automatic preset also had the advantage of
relieving the operator of some of the steering task.

McMahon (1982) listed four reasons to justify the
development of automatic steering: 1) reduced fatigue, 2)
increased vehicle speed, 3) increased machine output by
allowing the operator more time to monitor machine
performance, and 4) reduced wasteful implement overlap.
These are all factors improving the operator’s field
efficiency.

In his development of an automatic steering controller
for an apple harvester, McMahon (1982) also detailed the
major components of what should comprise an automatic
steering control system. The steering system was broken
down into two main parts, a controller and sensing system.
The controller received guidance signals, processed the
signals, and provided a controlling signal to the machine.
McMahon further explained that controllers could be of two
types, analog or digital, and that the sensors also could
be of two types, contact or non-contact.

In his explanation of sensing systems, McMahon
discussed the advantages and disadvantages of the two main
categories of sensors. Typical contact-type sensors were
mechanical, which means their successful operation required
periodic adjustment. Moreover, contact-type sensors wore

and required periodic replacement of parts to continue

12

proper operation. Contact from the sensor to the object
may also damage the object to an unacceptable degree. In
some extreme situations the amount of contact force may
prevent a contact-type sensor from even being considered
for use, such as in sensing cereal grains.

In contrast, McMahon explained the advantages of using
a non-contact-type of sensor. By its very nature of
maintaining some distance between itself and the object
that it is sensing, the non-contact sensor avoided most of
the problems associated with the contact-type sensor. The
potentially more complex non-contact sensor can be provided
with better mechanical protection than can the more
vulnerable contact sensor.

In my masters thesis (Fehr, 1986) I added a new
concept to automatic guidance systems by developing a dual
mode guidance system for a row crop planter. Previous
guidance systems in agriculture typically utilized only one
mode of guidance. If this single mode failed to detect the
directrix, the vehicle became confused and required
operator intervention to restore the steering control
signal.

The guidance system developed by me utilized two modes
of guidance. If one mode (the primary mode) lost its
guidance reference a second mode was available to provide
guidance information. The operator need not be aware of
which system was providing guidance information, only if

the event occurred where both systems were confused. This

13

guidance system also used a combination of contact and non-
contact sensing showing practical applications of both
methods.

My thesis, also described how I used a microprocessor
in a multi-tasking mode. Use of programmable peripheral
chips changed the role of the microprocessor to more of a
supervisory mode of operation. Instead of the
microprocessor being directly in charge of a few tedious
tasks that resulted in poor use of the power of the
microprocessor, it delegated these tasks to other parts of
the computer. With this supervisory mode of operation, the
microprocessor’s power was much more efficiently utilized
resulting in a more powerful guidance computer with only a
small increase in complexity.

Murphy (1988) offered some interesting insight into
the evolutionary development of the guidance technology
used in agriculture. First, he explained the developments
that increased the physical abilities of the operator,
chiefly through the use of hydraulics. Later in the
development of agricultural technology the emphasis shifted
to enhancing the operator's mental capabilities. By
providing engine speed controls, draft and depth controls,
the operators field efficiency was improved. Ultimately
this has led to the development of guidance systems for
agriculture. With the help of a guidance system, the

operator can now control large implements at high speeds,

14

another way to improve his field efficiency.

In his development of an ultrasonic-based guidance
system for a tractor, Murphy brought out other interesting
aspects of guidance in agriculture. An index relating to
the difficulty of tracking on a row has been devised to
quantify the degree of diligence required in performing a
given field operation. Studies of this sort have shown
that after operator fatigue, one of the greatest
contributors to poor steering accuracy resulted from the
operator having to look behind to monitor the implement and
look ahead to steer the tractor. This awkward, but
necessary, arrangement caused a reduction in speed as well
as a decrease in steering performance.

As Murphy progresses to a discussion of guidance
specifics he first mentions the aspect of safety. He
discusses some established guidelines that relate to
protecting the operator, the implement, the vehicle, and
the surroundings. Automatically guided vehicles were
classified into two types, one type with operator assist
and one type that would have no operator on-board.
Regardless of the type of system, the vehicle should have
the capability for manual guidance in public areas or where
automatic guidance provides too great a risk. In any case,
the vehicle should also have the ability to revert to
manual control and/or shut down in the event of a failure.
Safety would require that the steering wheel remain

stationary when in automatic mode, and always be available

15

to the operator in an override mode for emergencies.

Murphy then discussed steering control algorithms.
These algorithms would include speed-dependent and non-
speed-dependent aspects of controlling the tractor
implement combination. Work was also done on different
types of steering arrangements. This would include, front
axle steering, rear axle steering, and articulated
steering. The consensus is that all three types of
steering could be adequately controlled for field
operations.

Murphy also contrasted proportional hydraulic steering
control with digital hydraulic steering control. Murphy
used the digital mode of steering and arranged a set of
networked microprocessors to provide the control signal.
This network of microprocessors was another method of
distributing the processing load. One microprocessor acted
as the master of the system, the other microprocessors were
slaves which were controlled by the master.

In summary, Murphy drew the following relevant
conclusions about automatic guidance:

1. Safe and effective control of agricultural vehicle

guidance is possible using automatic methods.

2. A microprocessor provides the speed and
programmability required for automatic guidance
control in agricultural operations.

3. Non-contact sensors with no moving parts are
desirable for automatic guidance control because

they are less fragile than mechanical contact
sensors. (p. 11)

16

It would seem that the necessary parts of an automatic
steering controller have been well established, as well as
the advantages and disadvantages of various sensing
methods. It now remains for the development of
sufficiently complex controllers which will take advantage
of the best sensing methods to provide a reliable steering
control signal. An overview is given of these theses as
the best way to present the previous works relating to
guidance in agriculture in an expeditious manner. For more
in-depth information these theses can be studied as well as

the numerous citations that generated the discussions.

2.2 VISION GUIDANCE IN AGRICULTURE

The first investigation of the use of a vision system
to provide a steering signal for a field operation is from
Gerrish et al. (1983), Gerrish, Stockman, Mann, and Hu,
(1986). This study indicated that then state-of-the-art
technology in video cameras, video digitizers, and
computers were fast and accurate enough to provide
sufficient control signals to steer a tractor in a row-crop
application. It now remained for the reduction of this
theory to practice. Fehr and Gerrish (1989) produced an
operational Vision Guided Robotic Tractor (VGRT) that could
follow corn rows at speeds up to four km/hr.

Reid and Searcy (1988) worked on a parallel effort to

Gerrish et al. in developing an operational vision guided

17

tractor. Reid (1987) used a slightly different approach in
his vision-based steering control system. He used a
monochrome camera with an optical filter to reduce the
computing overhead. The optical filter limited the light
received by the camera to the near-infrared portion of the
light spectrum. The received light was in the 850 nm
wavelength portion of the spectrum with a pass band of 100
nm. Reid's reasoning was to take advantage of the
increased reflectance properties of soil and vegetative
material in this light wavelength.

Reid then proceeded on a method to distinguish the
crop material from the soil background from which to then
extract a control signal. A Bayes classifier for a two
class problem was used to find the optimum threshold to
segment the image into object (plant material) and
background (soil). Run length encoding (RLE) was used to
reduce the memory storage requirements of a thresholded
image. The transition points between runs represented
boundaries between plant material and soil. The start and
stop points of the runs (corresponding to the location of
plant material) were averaged to provide a representative
crop canopy center.

Reid then used one of two methods to quantify the line
associated with the crop row. The Hough transform (Reid
and Searcy, 1986) was used to define the line of the row,

or a method of row clustering was used. Either method

18

produced a set of lines corresponding to the crop row in
the ground plane. From these lines Reid then developed a
control signal for steering the tractor. Like the method
used by Gerrish and Surbrook, Reid defined steering error
in two terms, heading error and offset error. By comparing
the heading of the tractor and the offset position of the
tractor to the lines formed from the crop rows, which are
extended to the vanishing point, a steering heading was
determined for the tractor.

Reid's vision guidance system had some similarities
and many differences with respect to the vision guidance
system described in this study. He used a monochrome
camera in a slightly different portion of the light
spectrum and placed the camera on the front of the tractor
instead of over the rear axle. Reid's method of
determining the steering control signal for the tractor
also has a much higher computational overhead than what was
used in this study. The vision guidance system in this
study never quantized heading error and offset error as two
separate entities. Both Reid’s guidance system and the
vision guidance system developed here used two computer
systems in a master and slave arrangement (Brandon, Searcy,
and Babowicz, 1989).

From the many examples of automatic guidance systems
found here and in the literature it would seem that
robotics, in the form of automatic guidance, is firmly

entrenched in agriculture. As future efforts continue in

19

the development of robotic guidance systems it may be well

to remember the words of Isaac Asimov (1950):

THE THREE LAWS OF ROBOTICS
1) A robot may not injure a human being, or through
inaction, allow a human being to come to harm.
2) A robot must obey the orders given it by human
beings except where such orders would conflict with
the First Law.
3) A robot must protect its own existence as long as

such protection does not conflict with the First or
Second Law.

Handbook of Robotics
56 Edition, 2058 A.D. (p. i)

2.3 NON-AGRICULTURAL VISION GUIDANCE - ROAD FOLLOWING

Outside of agriculture, many other efforts are being
undertaken to study vision guidance. The equipment and
type of algorithms involved in these projects differ
markedly from those used in this agricultural related
research project. One similarity, however, is that these
non-agricultural vision guided vehicles use vision
equipment that is self-contained on the vehicle. A
representative sample of some of the research being
conducted is given here.

In the Federal Republic of Germany, research has been
done on a vision guided high speed vehicle (Dickmanns and

Zapp, 1987). This vehicle was to operate in the relatively

20

controlled road situation of an autobahn. The vision
system used was two forward looking black and white CCD
cameras. One camera had a wide angle lens, the other
camera a telephoto lens for looking farther ahead. These
cameras were mounted on a movable base. The computer
system involves the use of multiprocessors to split the
image processing up among as many as 14 processors.

To follow a road, Dickmanns, and Zapp (1987) outlined
five necessary functions that the autonomous system had to
perform. The functions were:

1) Detection of the road and its parameters like
curvature, lane width, surface state, presence or
absence of obstacles (static objects).

2) Recognition of relative position in the lane and of
position and speed relative to other moving
objects.

3) Correct interpretation of traffic information like
road markings (e.g. lines, interrupted or not),
traffic signs (e.g. speed limits) and local

warnings.

4) Safe and efficient control of the vehicle depending
on the actual state resulting from 1 to 3 above.

5) Easy human interface for both switching from human
to autonomous control and back and for human
decision taking on request of the automatic system
(warning functions). (pp. 1,2)

This system was placed in a van and fully equipped for

automatic control. A test road scene was projected in
front of the vehicle at simulated speeds of from 30 to 60

km/hr.

Road following work has also been done at Carnegie

21

Mellon Institute which includes the use of adaptive color
classification and shape tracking (Wallace, 1987). In this
research a van was equipped with a color camera mounted on
a movable base that used vision to track the shape of the
road through the image sequence. The basic problem was to
segment the image into road (the navigable portion of the
image in front of the vehicle), and shoulder (all other
non-navigable portions of the image).

To operate a road following vision guided vehicle
successfully in spite of all the color and shape
inconsistencies encountered in a road situation the

following assumptions were developed (Wallace, 1987):

1. Local Ground Plane. We assume that the roadway
around the vehicle lies in a plane, or at least
nearly so much so that the difference is
unimportant for steering in situations we
encounter. Furthermore, we assume that the one-to-
one mapping from image points to ground plane
points is known.

2. Obstacle freedom. The road in front of the vehicle
is assumed to be obstacle free, so that the
location of the road is all that is necessary for
the robot to move forward without collision. In a
practical system, the detection and avoidance of
obstacles will be left to other modules running
concurrently with the road following module.

3. Shape constancy. Although we allow the colors and
textures of road and shoulder to vary wildly, the
basic shape of roads and shoulders remains
constant. Also, the program described here has no
knowledge of intersections or road shapes other
than the simple case of a locally straight road
bounded by parallel lines.

22

4. Shape continuity. The position of the road changes
only slightly from one scene to the next. More
precisely, if the robot senses scenes of the road
arbitrarily close together in time, then the
difference in road position between scenes becomes
arbitrarily small.

5. Dead reckoning. From one image to the next, an
estimate of the vehicle’s motion is known. This
estimate is assumed to be bounded.

6. Colors distributed normally. The measured color
features of objects are assumed to be statistical
phenomena modelable by multivariate normal
distributions. (p. 259)

A test situation was video taped of a one km curving
asphalt roadway. The roadway had sections in direct
sunlight and sections in the shade of trees. The imagery
was further confused by pot-holes, color gradations in the
road, and seasonal variations such as snow. A description
of the success of the vehicle in following the road is
somewhat vague. Failure modes of the vehicle were
classified when a successful completion of the test section
of the road was not completed.

This vehicle has since been developed to include the
use of artificial intelligence in the form of neural
networks to provide steering feedback control (Touretzky
and Pomerleau, 1989). These neural networks use a method
of back propagation to "learn" the proper algorithm to
solve the presented problem and associated solution. The
problem and its solution are presented, over and over, to

the neural network which alters its weights when it reaches

the correct solution. The neural network method has the

23

advantage of being able to attempt to resolve problems
which it has never seen before, but which are similar to
problems with which it has experience.

These non-agricultural autonomous vision guided
vehicles work in an outdoor environment similar to the
environment of the vision guided tractor presented in this
dissertation. Similarities would include changing light
conditions as well as color variations in the scenery
surrounding the intended path of travel. The major
differences would probably be in the computer systems,
involved in the imaging systems and the types of algorithms
used to solve the path following problem. The road
following vehicles have more complex computer systems and
use much more complex algorithms then does the tractor

guidance system.

2.4 EDGE DETECTION

Of all the computer vision operations used to attempt
to follow some target path such as a road or crop row, one
Operation is basic to most methods. The operation of edge
detection is usually used to note when something in the
scene changes which might be significant in detecting the
desired path of travel. An edge can be described as a
discontinuity or abrupt change in grey scale value or color
in the scene (Levine, 1985). This abrupt change in grey
scale or color could be the difference between the grass at

the edge of the road and the asphalt of the road, or the

24

difference between the soil of the field and the vegetative
material of the crop row.

Edges can be used as a clue to the shape of the object
and the object may contain edges in any orientation. The
orientation of the edge and the significance (rate of
change of grey scale or color through the edge) of the edge
can also be clues used to gain information from the edge.
Edges that are nearly parallel to the supposed direction of
travel could be a good clue as to the edge of the road,
similar to that of the broad edge of the crop row in a
field. Edge detection may be necessary to locate these
areas of interest to determine a steering response.

Edge detection usually consists of scanning a matrix
that contains the edge operator across the pixel array of
the image. The edge detection matrix may either be one or
two dimensional depending on the complexity of the target
edges and their orientation. The dimensions of the matrix
may also be varied to alter the size of the edge that is
being viewed for as well as to reduce the effects of noise
in the image.

Perhaps the most simple of edge detectors examines the
difference between two pixels. A simple two element matrix
is scanned across the image. At each new location the
difference in grey scale or color between the two pixels is
noted. When the difference is great, it is a good

indication of the presence of an edge. Such a detector

25

works well for very ideal edges, but ideal edges are rarely
found in natural scenes. As mentioned, the size and
orientation of the matrix may be changed as well as the
detection operation to improve the response of the edge
detection to less ideal situations.

Edge detection can be further enhanced by first
preparing the image. The usual method is to segment the
image into object and background. With a two class system
it is very easy to locate and derive information from the
edges as there is little or no noise in the image.
Thresholding is an easy method used to segment the image.

To threshold an image, a histogram of the light
intensity levels must first be taken. Each pixel in the
image has an associated light intensity value. The
intensity levels represent each discrete level of light
intensity that contains the range from cut-off (no light)
to saturation (maximum light). One value per pixel if it
is a black and white image, this would be its grey scale
value. In the case of a red, green, and blue color image
there would be three intensity values, one corresponding to
each color value in the pixel. Thresholding can be based
on one intensity value in the case of a grey scale image,
or on one or more in the case of the color image.

The histogram represents a frequency of occurrence of
each intensity level for the image. For an image that
contains a clearly defined object and clearly defined

background, the histogram should have two modes. One mode

would
The n
more
segue
on tt

lies

All i
to cu
be se
class
image
shoulu
image
Which
(LGVi:

26

would correspond to the object and one mode the background.
The more clearly defined the object and the background, the
more definition these two modes have in the histogram. To
segment the image, an intensity threshold is selected based
on this histogram.

The threshold is selected from an intensity value that
lies between the two peaks of the modes in the histogram.
All intensity values below this threshold value can be set
to cut-off (black) and all values above the threshold can
be set to saturation (white). Now the image has only two
classes in it, object and background. If the original
image has good contrast, the object in the segmented image
should bear the outline of the object in the original
image. The segmented image now has clearly defined edges
which can be examined for further guidance information

(Levine, 1985).

2.5 COLOR DESCRIPTION AND PROCESSING

The apparent color of an object is due to the
reflectance and absorption of portions of the visible light
spectrum from that object to the human eye or video camera.
Different colors are the combination of different
wavelengths and intensities of portions of the visible
spectrum. For most humans the visible spectrum ranges from
400 nm (violet) to 700 nm (red) (Searcy and Reid 1989).

Typically color is described by one of three methods.

One 1
to dt
valu:
This
to rn
invo
prim;
blue
the c
colo:
appee

repre

mode
Yell:
Order
Where
adds
by tr
is tr
proje
Subtz
film
amoun
and d
by th

allow;

27

One method, commonly used in digitizing video equipment, is
to describe the color based on the respective intensity
values of the red, green, and blue components of the color.
This method lends itself to being used in an additive mode
to recreate the original colors. The additive mode
involves the use of adding specific amounts of the three
primary colors, red (R)(650 nm), green (G)(530 nm), and
blue (B)(460 nm) (Kantowitz and Sorkin, 1983), to recreate
the original color. With the use of these three primary
colors all colors can be represented. The human eye
appears to work on the additive mode of color
representation (Lillesand and Kiefer, 1979).

The photographic process (film) uses a subtractive
mode to capture colors. The primary color complements,
yellow, magenta, and cyan, are subtracted in a specific
order to maintain the integrity of the original color.
Whereas the additive process contains a light source and
adds the primary colors which then are emitted and received
by the human eye, the subtractive process is passive and it
is the reflected light (or transmitted light in the case of
projected film) which is seen by the eye. In the
subtractive process ambient light must first strike the
film layers of the complementary colors. Depending on the
amount of remaining complementary colors (after exposure
and developing) the light that is not absorbed (subtracted)
by the complimentary colors is reflected or passed and

allowed to travel to the eye were it is registered as a

28

color (Lillesand and Kiefer, 1979).

The third method of color processing and rendition
involves the use of hue (H), saturation (S), and intensity
(I) to recreate the original image colors. Hue refers to
the fundamental color, saturation defines the strength of
that color, and intensity is the light level intensity
associated with the color. This HSI method of reproducing
color is probably more accurate than the RGB intensity
levels method (Williams, 1988).

The RGB intensity method is actually three
monochromatic values used to recreate the original color.
This method can result in some ambiguity as to the original
color because each monochromatic color (R, G, or B) must be
treated with a different weight based on the response of
the human eye. No matter how it is dealt with the RGB
intensity method is only three grey scale renditions of the
original color at different frequencies. The HSI method
portrays the original color through the use of color
descriptors (HSI) that can recreate more exactly the
original color based on more factors than just grey scale

intensities of three color frequencies.

2.6 RANGE FINDING
Range finding can be a non visual method to determine
distances to objects in a given range of detection. This

distance information can be used to both guide a vehicle

29

(Dunlay and Morgenthaler, August 1986) as well as avoid
obstacles (Dunlay and Morgenthaler, October 1986).
Typically, range finding would use light or sound waves as
the mechanism to determine the distance, since tactile
sensing would be at a disadvantage in a mobile robot.

Range finding may have an advantage over intensity
images in unstructured work environments. It can be argued
that agricultural field settings are fairly structured and
may not need range finding. This is true until the
function of obstacle avoidance is deemed necessary for safe
and successful operation of an automatically guided
tractor in a field. Range finding may also provide help in
locating rows or cut crop edges due to the dramatic change
in height from crop to soil.

Whittaker, Turkiyyah, and Hebert (1987) list some of
the elements of an unpredictable work environment that may
also be relevant in an agricultural setting where range
finding may offer an advantage.

Unstructured environments lack regular, obvious
patterns that can be exploited to classify, recognize,
and understand the surroundings. Cognition in these
environments requires inferences based on intrinsic
and often subtle characteristics of objects and their
surroundings. Robots must discover structure in the
environment rather than impose preconceived models and

adjust their parameters as in rationalized
environments such as factory work cells.

30

Dynamic environments evolve over time: they are
discovered incrementally as task execution proceeds.
The perceived world model rarely converges to a
preconceived model (though some preconception is
important as a starting point in perception). Such
environments defy preplanning and present the need to
continuously manage potential contingencies.

Achieving useful goals in these environments
requires complete three-dimensional maneuvers without
constraints such as planar actions or fixed
orientations, hence the need for spatial reasoning
capabilities not usually required in factory
environments. It is also common that forceful
interactions with the environment are required, ...
(p. 1991)

Unlike visual intensity images of a scene which may be
confusing due to the difficulty of extracting 3D
information from a 2D representation (camera’s image plane)
of the scene, range finding utilizes the one physical
phenomenon of spatial importance, that is, the geometry of
the physical scene. Range data can provide information
relevant to the following geometric items: angularity,
orientation, extent, curvature, shape, and surface texture
(Whittaker et al., 1987). Some of these items could be
very useful in an agriculture setting.

Whittaker et al. (1987) further goes on to explain
some of the more appealing advantages of range data.

Many scenes derived from unstructured work
environments are not "interesting" in the sense that
no one object is directly apparent to a viewer. The
most expressive way of describing such scenes is by
prescribing their geometry. Soil in an excavation pit
is a good example of shapeless geometry where depth
maps can be readily constructed from range images, but
where intensity, color and regular geometry add little

information. Depth maps can represent terrains and
object surfaces efficiently.

As t1
becon
supp]
Land

Gremk

waves
The 5
range
fihdi
Water
Obsta.
resell
(Waite
improv
that m

1978).

31

The inherent three-dimensional spatial geometry
of most environments where robots work and the
distinguished importance of the near field make
desirable the use of a sensing model that provides
direct, voluminous, dense, and accurate data for
building models and planning actions. Emphasizing the
near field and discounting the far is a natural
consequence of finite-range sensors and provides a
good way of separating the background from foreground
objects most relevant to robot modeling and planning.

Many robots plan their actions based on distances
in the environment (between objects, from surface to
tool tip, etc.,). Range data enable fast, surface-
relative tactics such as "fly-over", i.e. traversing
over a surface while maintaining an objective standoff
distance to cope with unanticipated surface
variations. (p. 1992)

As the speed of receiving range data increases, it may
become more useful as a guidance aid in agriculture as a
supplement to a vision system such as in the Autonomous
Land Vehicle (ALV) of Martin Marietta (Turk, Morgenthaler,
Gremban, and Marra, 1987).

As previously mentioned range finding may use sound
waves for detecting spatial information around a robot.
The sound waves are generally in the ultrasonic frequency
range and may be pulsed or continuous. This use of range
finding (specifically known as SODAR, SONAR is for use in
water, Polaroid, 1982) would probably take the form of
obstacle avoidance due to the difficulty in achieving high
resolution between small objects in the environment
(Walter, 1987). However, techniques do exist to help
improve the ability of ultrasonic sound to resolve objects

that may make it more useful in obstacle detection (Gross,

1978). Range finding of an extreme nature may even use the

32

Global Positioning System of satellites for position
information (Larsen, Tyler, and Nielsen, 1988).

Optical range finding, which can also be used for
autofocusing, is of two basic types: active systems
which require an active source of illumination and passive
systems which do not employ their own source of
illumination but rely on ambient illumination. There are
disadvantages and advantages to either type of range
finding (Wolpert, 1987).

Active systems have the advantage of being able to
work well in situations of total darkness, or situations of
less than optimal lighting conditions. They do however
have a problem when objects, such as a window, are between
the ranging system and the object. Passive systems are
unaffected by windows but do have performance problems in
low light situations, could not work at all in total
darkness, and with objects that have poor contrast. A good
ranging system may have a combination of both methods

(Wolpert, 1987).

CHAPTER III

METHODOLOGY

3.0 INTRODUCTION

The purpose of this chapter is to describe the
development of the Vision Guided Robotic Tractor system
(VGRT) (Figure 3-1). The VGRT started with a conceptual
mode of development which then proceeded to an operational
mode of development. The conceptual mode deals with the
development on paper of a prototype VGRT from available
hardware. The operational mode is the functioning result
and is composed of a hardware component and a software

component.

3.1 TECHNOLOGICAL VS. ENGINEERING APPROACH

The technological approach to solving the problem of
using machine vision to guide a tractor is both a pencil
and paper approach as well as a hands on applied approach.
It is more specifically the use of both of these methods on
a continually alternating basis. This approach would be in
contrast to the classical engineering approach of using
pencil and paper until a complete path can be seen from
problem formulation to solution. After completion of the
engineering design phase, the project is then typically

turned over to the applications approach and an attempt is

33

QC

 

 

MASHHQ
COMPUTER

 

34

CAMERA
\\\ SLAVE COMPUTER
I [L l

 

 

 

I

\gji

 

 

 

Figure 3-1.

 

L
as r

/

 

 

4 COMMUNICATIONS LINK

 

Q LINEAR ACTUATOR
TRACTOR K\\¥

Side view of VGRT. This is a representative

view of the completed VGRT showing the location of the
major components.

35

made to verify the design by testing a prototype.

Each approach, technological or engineering, can be
made to appear superior to the other. Engineering uses
mathematics and other abstract processes as tools to solve
the problem. This may produce a fine solution on paper
that is impractical when it comes to actually trying to
build a working model. The engineering design approach
may, however, reveal fatal limitations that save the
expense of building something that will quickly show the
same fatal flaw. The engineering approach may produce a
prototype that proves that a solution is feasible. The
operation of the prototype however may not be very robust
if it has been designed only for proof of concept.

The technological approach would use appropriate
technology in a step-by-step approach to solve the problem.
The big problem would be broken down into several smaller
parts. These parts would be solved, recombined, re-
evaluated and then presented as a solution to the big
problem. As a smaller part is solved it can be used as a
determining factor to decide if the potential still exists
to solve the whole problem. With this approach, if a fatal
flaw is found, some components may have to be discarded;
.but if a partial section works, then that part of the whole
is already completed. The technological approach should
produce a prototype that has a relatively high degree of

robustness, this robustness being necessary to indicate

36

that the technology used did indeed solve the problem.
Whereas prototype failure can be forgiven in an engineering
proof-of-concept exercise, there is no such forgiveness in
the technological exercise where the working prototype is
the only proof admitted. By this definition of what the
technology approach must accomplish also indicates that it
must provide real-time position information to get the
prototype to operate at real field speeds. The engineering
approach may not provide this.

The technological approach is the solution of choice
in this research project. The reason for choosing
technology instead of engineering to solve this vision
guidance problem is based on the author's current degree
program and background which are both technology oriented.
The question is: "Can technology solve the problem of using
machine vision to guide a tractor?" It was believed from
the inception of this project (1985) and to the present
that available "off the shelf" technology would be
sufficient to produce a working prototype of a vision
guided tractor. This technology must be combined in a
creative way to provide a solution, as opposed to creating

new technology to solve the problem.

3 . 2 CONCEPTUAL MODEL
The problem of using machine vision to guide a
'tractor, and developing a working prototype was first

*viewed as a problem which can be broken down into component

37

parts. The conceptual solution to this problem would have
a tractor with two computer systems arranged in a master-
slave configuration (Figure 3-1, 3-2). The master computer
would control the machine vision system and the slave
computer would control the functions of the tractor. At
the outset, I wanted to leave the master computer as free
as possible to do image analysis of as yet undetermined
complexity. The master computer can then spend its time
deriving a steering control signal from the field imagery.
The master computer would also be the source of software
development for both computer systems.

The camera would be placed over the rear axle and
mounted above one of the drive wheels (Gerrish and
Surbrook, 1984). Such placement should give the camera the
most lateral stability and still give it a forward view of
the field scenery. The master computer’s control signal
would be presented to the slave computer, which would deal
with the tedium of turning the steering wheels and/or
starting or stopping the tractor.

In summary, the conceptual model of a working

;prototype VGRT would have the following characteristics:

Master computer:
1) Provide a supervisory role for dealing with failure
modes.
2) Provide fast clock speeds for timely control

signals (at least two position updates per second).

38

 

 

 

 

3&8 E38
8 ..E on

 

 

 

 

 

 

 

mozfimalj
was: 025mm;
52an ya 2 l_ R:
m H a < \ teal.
...i- iii! moEmEzH
$5.128 mid
x . -- x7!
55.5 22-2% mmEémmmww
mmmmimémséom
E5985 3 ”I38 a3

 

 

 

 

mm; mommam TIL

 

 

Lmdéab \ @828 films:

 

 

 

mMNCHoHD omDHH/ITT

 

/ filmmmz<o mOJOQ

I

 

 

This block diagram

f the various components of the

Block diagram of VGRT.
teraction o

in

shows the

Figure 3-2 .
VGRT.

3)

4)

5)

5)

7)

8)

9)

10)

39

Provide means for developing software.

Provide capabilities for converting visual image
information to a steering signal.

Provide a learn mode for headland turns.

Provide the overseer mode for starting or stopping
the tractor.

Provide the interface (communications link) to
slave computer.

Derive its power directly from the tractor
batteries.

Consist of an Intel 80286 microprocessor based
system.

Provide access to operational parameters (LCD).

Slave computer:

1)

2)

3)

4)

5)

5)

Provide direct control of direction and movement
of tractor.

Provide distance traveled information.

Derive control of starting and stopping tractor
only from master computer.

Provide interface (communications link) to master
computer.

Derive its power directly from tractor batteries.
Consist of an Intel 8751 microprocessor based

system.

40

Tractor:
1) Provide convenient use for this purpose (readily accepts
computer control).
2) Facilitate testing (can be operated indoors or
out of doors).
Camera:
1) Provide wide dynamic range.
2) Operate in color.
3) Utilize rugged solid state circuitry.
4) Produce an NTSC signal output.
The next phase of development will convert the conceptual

model into the operational model.

3.3 OPERATIONAL MODEL

An operational model based on the conceptual model was
developed using a technological approach to determine the
feasibility of using machine vision to steer a small row-
crop type tractor. This development was broken down into
the following tasks: 1) acquire a tractor, 2) develop and
install a slave computer system, 3) install video
digitizing board in similar (to master) computer to test
and develop vision algorithms, 4) acquire a master
computer, develop DC to DC power supply for the master
computer, install video digitizer board in master computer,
and install master computer on the tractor, 5) develop and
install communications link between master and slave

computers, 6) develop system software, and 7) test

41

performance of system.

The operational vision guided tractor development is
divided into two parts, the hardware and software portions
of the completed prototype. Hardware refers to either
mechanical, electrical, or electronic components while

software refers to the programs developed.

3.4 THE TEST TRACTOR

The tractor used in the development of this VGRT was a
GE ElecTrac lawn tractor donated by the Detroit Edison
Company. The tractor has a three speed manual transmission
with no reverse gear. Each mechanical gear has four
electrical speeds to provide a total of 12 forward speeds,
reverse is limited to two electrical speeds per gear.
Reverse is obtained by reversing the polarity of the field
winding of the drive motor. This tractor is a small scale
model of a typical large field tractor and has many
functions which make it comparable to a full size tractor.

There are many advantages to using a tractor of this
size and type for development work. Being smaller, it is
easier to work around. Having an electric motor for its
motive force instead of an internal combustion engine
allows for its use indoors, and due to its smaller size,
smaller test areas can be used. The ability to test
indoors safely is a major advantage as testing and

development could proceed year around.

42

The initial testing of the vision system need not have
waited for the procurement of an expensive full size
tractor. If the system proved successful on the smaller
tractor, then there would be sufficient reason to move up
to a full size tractor.

From the work of Squires (1981), some of the necessary
conversion components were already installed on the
tractor. These included a linear actuator to steer the
front wheels (Figure 3-3), the relays to actuate the linear
actuator, and a metal box located on the hood of the
tractor in which to mount the slave computer. The steering
wheel had been removed for safety. LManual steering was
done through a double-pole double-throw switch.

A 1K ohm wire-wound 270 degree potentiometer was
installed on the rotating member of the front left steering
king-pin (Figure 3-4). This potentiometer was implemented
as a voltage divider to provide a change in voltage on its
wiper arm that was directly proportional to the angle of
the front wheels.

A toothed gear was added to the brake shaft of the
transmission to provide distance pulses to a reluctance
pickup. The brake shaft provided a direct link to the rear
axle that was independent of transmission gear range. The
result is a series of pulses that were directly related to
rear wheel revolution.

The reluctance pickup used was a DI-MAG model 58424.

This model provided a digital output that was compatible

43

LINEAR
ACTUATOR

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

EAT ERR
WHEEL ROD FROM STEERING ROD wHEEL

WHEEL

Figure 3-3. Top-down view of linear actuator placement.
The linear actuator is used to turn the steering wheels.

44

/ WHEEL ANGLE POTENTIOMETER

 

 

 

 

 

 

 

 

 

 

FLEX- i I
COUPLER gym :3
\ Er I
. I MOUNTDKSBRACKET
I} TI I! i
i
1 a” : I >
! é ' : AXLE \
i ii T? E , 8
g 1; ii , I '1 KING—PIN
i £1 1! I
i it ;!
I f! f ,_____I\_///
’ é!
;: J!
M ii FRONT
% fl LEFT
W N WHEEL
l 1 L

 

 

 

 

Figure 3-4. Front view of wheel angle potentiometer. The
potentiometer provides steering wheel feedback to the slave
computer.

45

with the input to the counter in the slave computer.
Instructions with the reluctance pickup indicated that it
should be mounted as close to the gear teeth as possible
for best sensitivity. This proved to be a problem as the
sensitivity was so great that vibrations in the mount used
to hold the reluctance pickup caused false distance pulses.
This problem was remedied by desensitizing the reluctance
pickup through simply increasing the space between it and
the teeth of the gear.

Another advantage of the electric tractor was the ease
with which the slave computer could be placed in control of
the tractor. Transistor buffers were used to increase the
current gain of the microprocessor to the point where the
current was sufficient to operate the relays that
controlled the tractor. The last stage of the transistor
current amplifiers (buffers) was an open collector wired OR
configuration. This type of interfacing allowed for the
control of the tractor by the stock speed and direction
controls or by the slave computer (Figure 3-5). The slave
computer could be either on or off for manual control of
the tractor. With this configuration, the tractor could
still be operated even if the slave computer was removed

for development work or if a component failed.

3.5 SOLID STATE VIDEO CAMERA AND CALIBRATION
An Hitachi MOS type color camera (model VK-C3400A) was

used as the video sensing device. This camera provided a

446

pmcsmlaopm \ um¢u>ucia¢¢3¢ou \ monumm o»
nHHHVT.

nHHHUY:

Shmumuc NOOHO
...-CO
_ I

 

 

 

 

[IONIL

firm >¢JU¢

 

 

zanmmusauou cupJuu umuoz

sous u>uhoaoxu ..za 4
«ohmnmuc
cosmumuc

W,

>CN

&U [n )‘N...

80 PMHWUE

PHDOKHO u>H¢O >¢Jw¢
PHSOQHU cunts “OhMHmZCK...

hww acumnmus

ngaco

 

20 Pm Hmwc
“OHMHWUE
‘ uvfi In” VN'FLiA;

‘_ _ r Carmumflfl

.56

Alt
0

02 H bu Him JW>UJ

Electrical diagram of transistor buffers.

These buffers provide an interface between the slave

computer and the tractor.

Figure 3-5 .

47

2:1 interlaced, internally synchronized NTSC video image of
at least 280 lines of horizontal resolution. The video
output of the camera was coupled directly to the

digitizing board. To provide the camera with the greatest
lateral stability, while still providing sufficient field
of view, the camera was mounted on a platform directly over
the right rear wheel. A quartz-halogen light, which
augments ambient light, was provided on the same platform
to aid the camera in low light settings.

The Hitachi camera had an F of 1.2, and a variable
zoom from 10.5 to 65mm. The camera was calibrated
corresponding to the fixed zoom values as imprinted on the
camera lens assembly. For calibration the camera was
mounted 120cm up from the floor and at a distance of 178cm
from a 122cm by 244cm sheet of 12mm plywood. The camera
was placed so that its nadir was perpendicular to the plane
of the plywood sheet with the X-axis parallel to the
horizon. This arrangement produced a test setup that
contained the full view of the camera on the plywood for
all zoom angles.

For each imprinted zoom number the width of the field
of view was measured for both the X and Y axes. The

results of this calibration are listed in Table 3-1.

48

 

Table 3-1 shows the results of calibrating the
Hitachi camera at the specified zoom angles.

Tahle_3:1I__Besult§_2f_samera_salibratignl
292m 23129 31921 zldesreesl Ylsml, e ees
10.5 122 34 94 23
15 as 25 7o. 21
20 65 20. 53 17
25 51 16 42 13
3o 41 13 32 1o.
45 28 9 23 7
65 20. 6 16 5

 

The calibration of the camera was necessary to map the
camera image plane coordinates onto the real world ground

plane coordinates.

3.6 SLAVE COMPUTER

The slave computer is a single chip computer. The
Intel 8751 controller type microprocessor has four eight-
bit input/output ports, onboard 4K byte erasable
programmable read only memory (EPROM), and 128 bytes of
random access memory (RAM), and two 16-bit programmable
counter/timers. The ports can be configured in any
combination of in or out and are also bit selectable as to
«direction. If a bit is set to a high level, it is at a

high impedance state. In this high impedance state, an

49

outside signal can drive the bit high or low and can be
read as such by the 8751. This is how bits within a port
can be set in or out. Generally the 8751 will drive a
device with an active low level since the 8751 can source
much more current than it can sink (conventional electron
flow). The slave computer has port 0 set in, port 1 set
out, port 2 bi-directional, and port 3 set primarily in
with one bit set out. Port 3 is a special port. It can be
used as an eight-bit bi-directional port or as the
connection to special internal functions within the 8751.
These internal functions include the timer/counters,

interrupts, and serial I/O (Figure 3-6).

3.6.1 SLAVE COMPUTER HARDWARE

Port 0 of the 8751 microprocessor is devoted to an
eight-bit analog to digital converter (ADC). The ADC is
connected in a free-running mode with its output buffers
always on. The analog signal input to the ADC is connected
to the filtered output of the steering angle potentiometer.
With such continuous free-running operation of the ADC, the
8751 is assured of a constant stream of new steering wheel
position updates, as many as 2000 per second.

The potentiometer is positioned so that approximately
straight travel corresponds to an ADC output of 124.
Values below 124 indicate turns to the right and values

above 124 indicate turns to the left. The complete

50

 

 

P8] \
Form . 8—BIT A TO D WHEEL ANCLE
I PUB /
P17 RICHT TURN
P15 LEFT TURN
,_ P15 SEAT BYPASS (DEAD MAN)
/ P27 FORll P14 N.C.
8-BIT . P13 SPEED 3
PARALLEL - PORT2 P12 . SPEED 2
INTERFACE ' gig ESEWLARlD/REVERST
\ P22 ' C
P37 ROTARY SWITCH (HIGH BIT)
P35 ROTARY SWITCH (LOW BIT)
P35 DISTANCE PULSES
PORT3 P34 N.C.
P33 . NC.
2%? 3 ANNETTE A
z . 1
NET 8751 P32 HANDSHAKE TO 8751

 

 

Figure 3-6. 8751 port listing. This figure provides a
diagram of the various port connections of the 8751
processor used in the slave computer.

51

turning range of the front wheels is from 96 to 145 with
one change in count corresponding to approximately a one
degree change in steering angle.

Port 1 is used as an out port which controlled the
seat dead-man switch; steering relays; speeds one, two , or
three relays; and the forward or reverse direction drive
motor control relay. Port 2 is used in a bi-directional
mode of operation as the parallel communications link with
the master computer. Port 3 is used on an individual bit
basis. One bit is the input to counter 0. A second bit is
the input for interrupt 0. A third bit is used as the
communications handshake signal line received by the slave
computer. A fourth bit is used as the communications
handshake signal line which is sourced by the slave

computer.

3.6.2 SLAVE COMPUTER SOFTWARE

Two Assembler programming languages are necessary for
the VGRT. The control oriented instructions for the slave
computer differ to a great degree from the much more
flexible instructions for the master computer. The
Assembler for the master computer also has many more
features that make it rival a high level programming
language in some respects.

The first software written for this VGRT was Assembler
language for the slave computer. Test programs were

written to see if the computer hardware and tractor were

52

functioning properly. These tests included starting and
stopping the tractor in forward and reverse, turning the
front wheels, changing the speed of the tractor, and
counting distance pulses.

Two problems had to be solved in the software used to
control the tractor. A dead band had to be determined for
the steering wheels that would provide accurate steering
but would prevent the wheels from chattering. A dead band
of approximately two degrees provided a good level of
steering accuracy but had some chatter. Increasing the
dead band to approximately three degrees gave the tractor
good steering accuracy, but with virtually no chatter.

The second problem deals with stopping the tractor
while in reverse. Starting the tractor into motion
requires closing a SPST relay that provides power to the
shunt wound electric drive motor. Directional control of
the motor is accomplished by a second DPDT relay that
controls the polarization of the field windings. For
reverse travel, this DPDT relay has to be energized to
reverse the field polarity. The problem stems from this
DPDT relay being in series with the SPST relay (in regard
to the field winding, field and armature are in parallel);
both relays could not be de-energized at the same time.
The SPST had to first be de-energized, then the DPDT. If
both relays were de-energized at the same time, inductive

kick produced a large arc on the DPDT relay damaging the

53

contacts. A time delay between releasing relays of even
0.5 seconds is sufficient to prevent the problem.

This also brings up two more considerations when using
computer control of the tractor. 1) The stock tractor
controls have a built in delay to prevent too rapid a shift
between forward and reverse. When the computers are
operating the tractor, this built-in delay is bypassed and
has to be provided by the software. 2) When using computer
control, limits have to be provided to prevent the linear
actuator from turning the steering wheels hard against
their limits. The linear actuator has sufficient force to
jam the wheels at their travel limit where they may remain
mechanically stuck and require operator intervention to
return to an in-between position.

After it was determined that the slave computer could
operate free of interference produced by electrical noise
from the tractor, work could continue on development of
communications with the master computer. A communications
exchange can only be initiated by the master computer.

Only the master computer can start tractor motion. This
prevents accidental tractor movement before both computers
are up and running.

A characteristic of the 8751 is that at "reset" the
counter/timer registers are set to zero. This is not
a problem except when an error occurs in a communications
attempt and the master computer resets the slave computer.

A "reset" would have zeroed the accumulated distance and

54

give the master computer a false distance reading at the
next communications interchange. This problem was remedied
by reserving two RAM addresses as a scratch pad in which to
save the two bytes of distance information. During
initialization after a reset, the data in these two RAM
addresses is now written into the low and high byte
registers of the distance counter.

At power up, the distance values are random which is
allowable since only elapsed distances are used. The
distance information is updated in these RAM addresses
during each subsequent communications interchange. When a
reset occurs, the current distance information is reloaded
into the counter allowing the distance count to survive the
default reset to zero.

The slave computer executes a loop after
initialization that provides for steering wheel response or
communications interchange. The normal mode of operation
for the slave computer is simply to monitor Port 2 for
steering wheel positions. This mode is indicated by an
inactive handshake line. At the beginning of its program
loop, the slave computer checks for an active handshake
line. If the line is not active, it reads the data from
Port 2 which will tell it the setting to which the front
'wheels are to be turned. If a handshake line is active
then the slave computer jumps to the communications

interchange subroutine.

55

3.7 MASTER COMPUTER

The master computer is a general purpose, commercially
available MS DOS based, IBM AT type system. The system
chosen was a 6 or 12MHZ Intel 80286 based system using the
Zymos "personal computer on a chip" (POACH) set. This type
of system was chosen because it had a dual purpose. The
computer was used asthe master computer in the vision
guided tractor; and was also used for software development.
Essentially the computer could be operated independently of
the tractor as a conventional personal computer. This was
a great advantage as it was not necessary to procure a
separate development system and programs could be stored

and loaded from a floppy disk.

3.7.1 MASTER COMPUTER HARDWARE

The only significant modification to the computer was
the removal of the standard 110v power supply and the
installation of a custom built DC to DC converter power
supply which allowed the computer to receive its power from
the tractor batteries. Holes were also machined in the lid
of the master computer to facilitate air movement for
improved cooling.

The master computer contains a Hercules monochrome
graphics adapter board, a floppy/hard disk controller, a
(mistom built parallel interface board, 1M byte of RAM, and

a: video digitizer board. The 1M byte of RAM is split into

56

two 512K byte areas. One 512K byte area is used for
program operations, the other 512K byte area serves as a
virtual disk. Programs as well as editors, assemblers, and
linkers can be loaded once from a floppy disk to the
virtual disk and executed many times at a great improvement
in speed. This was a great convenience factor for
developing software.

The master computer was also used to develop and
cross assemble the software for the slave computer. A
separate custom built 8751 EPROM programmer was connected
to the printer port of the master computer. The program
code was transferred through the printer port to the

programmer and then "burned" into the 8751.

3.7.1.1 CUSTOM POWER SUPPLY

A DC to DC power supply was built for the master
computer to reduce waste power consumption from the tractor
batteries. This power supply also supplies a filtered +12V
for the solid state camera. The power requirements for the
computer and camera were +12V at 750mA, +5V at 6A, -12V at
less than 100mA, and -5V at less than 100mA. Series linear
power supplies were used for the two positive supplies and
'the two negative supplies used inverting mode switching
supplies.

This custom power supply was built so that it would
fit.in the same location in the master computer case as the

YIIOV'AC power supply it replaced. The master computer case

57

was fitted with two cooling fans, one fan to draw ambient
air into the case and one fan to exhaust hot air from the
case. The slave computer only requires +5V at less than
200mA so a series linear power supply was used. All power
supplies were fitted with ample heatsinking to reduce
component heat levels.

Both the master and slave computer power supplies were
subject to a great deal of battery supply voltage noise.
This noise was the result of either starting or stopping
the drive motor, or various relays engaging or disengaging.
Proper isolation from the regulated to the unregulated
sides of the power supplies had to be provided to prevent
the noise from disrupting the computer’s operation.

Two precautions were taken to improve isolation. A
diode was placed in series with the positive supply lead
from the batteries to the input of the power supply. This
prevents the power supply filter capacitors from
discharging back into the batteries when battery voltage
momentarily drops due to drive motor startup. The second
precaution was to use a modified pi-R filter after the
diode (Figure 3-7).

The pi-R filter uses two filter capacitors, one on
either side of a series resistor to reduce voltage
fluctuations. The two capacitors source or sink current to
'try to maintain a steady voltage while the series resistor
Tgives the voltage fluctuations a place to develop. The

large filter capacitors were each strapped with a small

 

58

IOHhUWhOCfl
wUGhJO> ¢W>O

 

      

>J¢lDM
KUIOL UZMIUhHIM
woo: UZHPKN>ZH

      

>J113M
KUIOE UZHIUHHIM
MOO! 01~h¢U>ZH

  

  

 

mltq n

lObCJDUUfi
“CMZHJ WUHEWW

 

 

*JtLDW KUIOL 1Ub¢U>ZOU 00 Oh 00 k0 ICKUCHO XUOJO

 

 

nflv L AW.
_ aao _ use use _muEJHL
ulnnmcmpm

 

 

 

 

 

 

 

so I!
acmxomu

 

 

 

mtfd M
GOPGJDOWK

 

QGU
UZHEACEFW
EUhJHk
“mm
“man KOhmHWWK
KUIOA UZHXUOJN
II I ll
ECU — ECU — ECO _ EGO — mUhJHk
UZHEECEPW Ulfllkdflrm
“NFJHL
Emu
UWDk EOhMHMWK

 

 

   

 

KCUZHJ MMHKUm

 

 

 

EWIOL

   

 

f custom power supply.

This power supply allowed the master computer to work

directly from the tractor batter

ies .

Electrical diagram 0

Figure 3-7.

59

capacitor to reduce the effects of the external series

resistance associated with the large capacitors.

3.7.1.2 CUSTOM I/O BOARD

The custom I/O board provides three basic functions
for the master computer system. One function is to sense
the temperature inside the master computer case. The
internal case temperature is monitored in the unlikely
event that a component failure might cause a drastic and
possibly harmful rise in temperature. Another function is
the provision of a four digit display of user selectable
information. This information could pertain to any of the
operating conditions of the VGRT. During normal operation
of the VGRT the monochrome monitor is removed, leaving the
four digit display as the only means to view operating
conditions.

The third function of the custom I/O board is the
parallel communications link to the slave computer. This
is an eight-bit bi-directional link with two handshake
lines to the slave computer. To insure that the slave
computer is operating in proper response to the master
computer or in the event of a failure in communications, a
slave computer reset signal line is also provided from the
I/O board. To initialize slave computer operation or
recover from a failure in the communications mode, the
master computer can reset the slave computer to re-

establish proper operation of the VGRT.

60

For ease of communication protocol and construction, I
chose a parallel type interface for the communications
link. A parallel communication operation would be less
susceptible to errors and would provide faster data
transfer rates than would a serial link. None of the clock
frequencies in either of the two computers are temperature
compensated. Therefore there was some risk, due to the
potentially wide temperature range over which the VGRT
might operate, that baud rates could change enough to cause
data transmission errors in a serial link. With a parallel
link that has handshake lines, there is no danger of this
occurring.

The parallel link consists of an eight-bit bi-
directional data bus, two handshake lines, and a ground
reference bus. The handshake lines are used by one
computer to indicate to the other computer that it is ready
to send data or to acknowledge that data has been received
(Figure 3-8). Each computer sources one handshake line and

monitors the other.

3.7.1.3 VIDEO DIGITIZING BOARD

A 256 by 256 pixel, color NTSC video digitizing board
(ATT ICB) provides the video input to the master computer.
The RAM that contains the digitized images is located in
the memory map of the master computer (the board is placed

in the backplane of the master computer). By placing the

61

 

SW2 4-DIOIT LCD

2% TEMPERATURE
DEGREES PAHRENHEIT
/\ INSIDE AT CASE

 

 

 

 

 

 

L_._I.__.._—... ...—A

 

: t HANDSHAKE IN
\1, {V 4, I > HANDSHAKE OUT

€———> BIT 7
%——> BIT 6
€——-—> BIT 5
e—é BIT 4
€———9 BIT 3
I/O PARALLEL INTERFACE < 9 BIT 2
é———9 BIT 1
6—9 BIT O

4 > GROUND

 

 

 

 

 

 

a

 

 

L > 8751 RESET

 

 

PC AT P

 

Figure 3-8. Block diagram of the custom I/O board.

62

video RAM in the computer’s memory map, more Assembler
programming instructions are available to manipulate the
data. Each color band (red, green, and blue) is digitized
on a scale ranging from 0 (cutoff) to 31 (saturated). Each
pixel is represented by three values, one value for each
color band. Due to the camera’s recording format, the
video image actually uses 256 by 200 pixels (4 by 3 aspect
ratio) which requires 104K of RAM per digitized image.

This RAM is physically located on the digitizing board and

is available for memory access in two 64K blocks.

3.7.2 MASTER COMPUTER SOFTWARE

One of the major requirements involved in working with
the VGRT is having sufficient computational speed to
provide timely position information. Attaining maximum
speed entails having as fast a computer clock speed as
possible along with the most efficient form of programming.
In this VGRT, one handicap is the 6MHZ limit on clock speed
in the master computer, as imposed by the video digitizer
board. (When trying to process the large amounts of video
information 6MHZ is none too fast a clock speed.) In order
to get the maximum performance from this clock speed, all
programs are written in Assembler language.

Assembler language may be tedious but it offers the
most compact, efficient programs and fastest execution

times possible. The difference in execution time between

 

63

Assembler language and a high level language may be as much
as three or four times depending on the language and how
well it is compiled, (in one extreme situation the
difference in execution time was two orders of magnitude of
time slower for the same histogramming program written in
Quick-Basic). Assembler language also has an advantage in
dealing directly with the bit manipulations that are
required for the video information and I/O control to the
slave computer.

The software development for the VGRT proceeded in a
logical manner from first learning how to control the video
digitizer board to actually processing images in a near
real time mode. When a high enough degree of confidence in
programming skills was established, the video digitizing
board was installed on the test tractor. Programming then
continued with developing the VGRT.

The camera image plane contains a 2D map of the 30
real image. The 20 image is sent to the video digitizer.
The image is mapped by X, Y coordinates for each pixel.

The pixels can be considered to be in the first quadrant of
a Cartesian coordinate system with the lower left corner of
the image as the origin. Using this coordinate system
permits easy mapping of the virtual coordinates of the
image (as derived from the camera’s image plane) to real
coordinates as found on the real-world ground plane.

To derive control signals by which to steer the

 

64

tractor, the camera’s image has to be mapped to real world
coordinates. Since the camera is positioned so that it
provides a low oblique view of the field image, its nadir
is not perpendicular to the ground plane. This causes a
perspective of the image that has to be corrected in order

to derive real distances and angles (Figure 3-9).

3.7.2.1 INITIALIZATION OF CONTROL PARAMETERS

In order to convert virtual distances in the camera’s
image plane to real distances on the flat ground plane, the
VGRT had to be calibrated based on the camera position.
During testing, the constants associated with camera
position were a user input so they could be changed for
different tests. The master computer will query the user
for the camera’s height, angle of tilt below the
horizontal, and the zoom angle. At this point, the master
computer uses this information to determine the near and
far edges of the camera’s field of view in real distances
(Figure 3-10). For all mapping of image plane coordinates
to real world coordinates, only the tangent (or arctangent)
function is used (Figure 3-11). The user can then select a
control point distance (forward of the front axle) and
enter it into the master computer. The master computer
will then query for a steering gain value and a center
pixel line value. The VGRT is now ready for operation
(Figure 3-12).

The control point is used to determine where the

65

m02<hma x<n_

1

3m; Lo 04mm

82.: m5 Joxwzoo

ll

mozlfimua z<mLz

 

I’l

/

m302<,zoo~ mmm:<o

 

mmFDLZOQ
mmeSL

 

 

 

 

TAPES mid /

<zmz<o

59$ «52%
”W N 50.2 St
II (Emu—.30

 

R

Note camera nadir is not

18

When the camera nadir

not perpendicular to the ground plane a perspective error

Side view of VGRT.
perpendicular to ground plane.
troduced which must be corrected for.

Figure 3-9.
is in

66

---- FLOW DIAGRAM ----

Initialization of constants for mapping image plane to
ground plane. Only the tangent function is used.

This

is executed once at beginning of program.

1. Enter camera height.
2. Enter camera tilt angle (below the horizontal).
3. Enter camera zoom number.

4.
5.
6.

11.

12.

13.

14.

15.

Look up corresponding X and Y zoom angles.

 

CAMERA
(:\\ ADI NEAR CENTER FAR
\_~\ J ‘ I e
33535;;~-ITHETA : L_5 5‘?
\\ ‘1‘ ~‘\\ ' l
1 »\__ :0PP. 70R: opp:
PEJGHT ‘\\;\\\v\\‘ I‘T‘~~~- ’ '
, \\\ \\\ : IT“ ‘~L I
J .\\ \T\\N59fi? i‘}~~1‘~“ 1
‘ GROUND ‘\. : \\\L 9/ “‘~~- :
_ “'- _ —’ _' "" '— —\K\—- — -—‘/\l~/—. —- —. .... ‘_‘“<:_
\\\ 4\/ ~\\
SIDE VIEW ‘\ /' 1323 PIXELS

Find tangent of theta (camera tilt angle).

Opposite side equals camera height.

Using camera height determine adjacent side.
(camera height / tan of theta = adjacent side).

Now know opposite and adjacent sides in cm.

Determine nadir length (hypotenuse), in cm.
(Pythagorean theorem to determine hypotenuse).

Determine nadir length in pixels.

Opposite side now equals Y axis / 2 or 100 pixels.

Theta now equals Y zoom angle / 2.

(100 pixels / tan of theta = adjacent side).

Now know opposite and adjacent sides in pixels.
(adjacent side = nadir).

Now have a direct correlation of camera position
constants in both real units (cm) and image
(pixel) units.

Alternately add and subtract Y zoom angle / 2 to get
near and far edge of window on ground plane.

The adjacent side is the respective near and far
distance.

16. Enter a control distance between these two values.
17. Enter steering gain.

18. Enter lateral center value.

19. VGRT is now calibrated and ready for operation.

Figure 3-10. Flow diagram of guidance control parameters.

67

4ST GRAPH OE TANGENT VS. DEGREES

I A ,//
35 2 /%//// SEG.3
; /// Y-1/3X4-12
-E ,//
i /

/
25-7 , '/ SEG.2

x1 Y-3/7X-+5

DEGREES

15

5 _L ///’ Y=23/42x

/ _I I I I I I I 1 I I

I I I T T I fix I

@113 23 3O 48 SO BO 73 88 SD 133

TANCENT X 188 (RANGE 8 TO 45 DEGREES)

Figure 3-11. Tangent curve approximation. The tangent
curve from 0 to 45 degrees is approximated by three
straight lines for ease in computing tangent functions.

68

 

 

 

I I I
HEDOF"” I T"
YEW I I I
I I
I T I
I I 4;
I 1 5 FMR I
I . I DISTANCE
i {I I
I —I‘— ‘7!— I
I I cmmm I
I I ‘ DISTANCEI
L I I I ;
; 5 I
SLAVE I I I I I I
N
COMPUTER i DESIANCE I I
I I ' I I I I
WI I FL)! \I I \I
t I '———r . I , \I I

 

I_:

 

.I______.

Iszfi-J
III’
II

I

_nucmR II;
msmm ; I

\ EU

.I IrL—
I i IL
II_I: CAMERA
I ,
T

I

 

II
II
I
I
J

I
t
i
l
l

DIRECTRIX

Figure 3-12. Top-down view of VGRT ready for operation.
The various camera position constants have been entered and
the near and far field of view values have been determined.

69

vertical center of the window will be in each image.
The window will move horizontally back and forth as
necessary on the pixel line that passes horizontally

through the control point.

3.7.2.2 STEERING CONTROL

The control point located in the image serves as a
reference to determine the angle to which the front
steering wheels should be turned. All steering angles are
either to the right or to the left of a center line, or if
the tractor is traveling directly along the line, straight
ahead. The center line is the vertical row of pixels that
is parallel to the major axis of the tractor. If this
pixel line is directly over the center of the row, or edge
of the row, the tractor is traveling straight ahead with no
steering correction required. When this pixel row and the
center of the row start to diverge or converge then a
steering correction is required to bring the tractor back
onto course.

The angle at which to turn the front wheels is
determined from one of the angles formed by an imaginary
triangle (Figure 3-13). The three corner points of this
triangle are: 1) the point at the intersection of the
center pixel line and the front axle, 2) the point on the
center pixel line that is the assigned control point
distance ahead of the front axle, and 3) the control point

as determined from the row edge or row center finder. The

70

FIELD OF VIEW

AI I
I Irv .
LU
éLIcmmmflBMWE
I N
L‘E/ fI \P s
2 ’ '
w I
I Q,/ gj// I

IF\IF\\\‘\7I /L§
LNRLLIJAM’

Figure 3-13. Diagra
steering angle. Thi
situation and the resu
the angle at which to turn

FRONT AXLE LINE

DIRECTRIX

 

m of triangle used to determine
5 view shows the VGRT in an off-row
ltant triangle formed to determine

the front wheels.

71

angle formed by the center pixel line and the line from the
control point distance on the pixel center line to the
control point is a right angle. The angle formed by a line
drawn from the axle point to the control point on the
center pixel line, and the line drawn from the axle point
to the control point determined from the row or edge point
is the angle at which to turn the front wheels. In essence
the front wheels are always aimed at the control point
determined in the row or edge finder. When the steering
gain is one, the wheels are actually aimed at that control
point, when the gain is more or less than one than the
wheels are turned more or less than that angle (Figure 3-14
to Figure 3-16). The amount of error from this method of
mapping image coordinates to real world coordinates was

less than 10 percent, based on a calibrated scene.

3.7.2.3 COMMUNICATIONS SOFTWARE

A communications interchange always follows the same
pattern. Having only a single communications interchange
procedure greatly simplifies the communications protocol.
No information has to be exchanged to see which procedure
to follow and this reduced the number of handshakes and the
associated chance for error. To further reduce complexity,
the master computer is programmed to be the only computer
that can initiate a communications interchange. The

interchange always starts with the master computer

72

---- FLOW DIAGRAM ----
Steering angle, mapping Y coordinate value.
Only the tangent function is used.
This is executed once each position update.
Map image coordinates to real-world coordinates.

1. Run contrast mode or color discrimination mode row
edge or center locator.

2. Now have X and Y coordinates of row (use Y value).

3. If above center subtract 100 to get absolute value of
Y.

4. Determine tangent of theta.
(Pixels above center / nadir in pixels = tan theta).

CNfiRA
’\..
\ \
x! ffé ‘ ~
I \\\~\\:““‘
\ \ ‘ ‘~ . ~ ‘ \ THE-TIA
HEIGHT \ \ u.\ 7- \ TARGET PIXEL ON:
I \\ =._ ‘~L\ .DMGEPLME
I X _ x“ / GROUND PLANE
_ N (fiflfifl ‘\ Nmmexx 3~~L I
_ _ _ " "‘ Ax — "\/' _. ‘_ _
x \ \ Z / \ ...
SIDE vzgw \ \ ’IDD PIXELS
CAMERA IMAGE PLANE ‘ J\

5. Determine angle in degrees.
(Theta in degrees = arctan theta).

6. Convert this angle to real world angle.
(Camera tilt angle - theta = new theta).

7. Now have theta of real world angle to determine actual
distance of located row point in front of front
axle.

8. Determine real world distance.

(Camera height / tan of theta = distance in cm.).
(Subtract 99 cm to convert distance to front axle).

9. If Y is below center add instead of subtract 100 to
get absolute value of Y, and add resultant angle in
degrees to camera tilt angle.

10. Now have Y value in real world units.

 

[\~>¥ mama
\“~‘ L—‘T ..L-I
\\\~‘_~t‘ Tl-ETA I I
\\ \\:‘C‘ OPP I
Madn' ‘\ ‘\ \~-“ i TMWETPDELON:
I x \\ up I /IMAGE PLANE
I \\ \\ \x / GROUND PLANE
GROUND \ NADIR\~\ 7"~- I
_. _. _. _. _ _. .\ ._ 7.. _. ‘1. __ _.
\\\ \l/ \
SIDE VIEW ‘\ Iaa Prxas
(mnaADMqH1ME J\\

Figure 3-14. Procedure for mapping Y coordinate value.

73

---- FLOW DIAGRAM ----
Steering angle, mapping X coordinate value.
Only the tangent function is used.
This is executed once each position update.
Map image coordinates to real-world coordinates.

 

 

1. Have X image coordinate value from run of row mode
locator.
2. Find absolute value of X from center.
3. Get camera zoom angle / 2 for X axis.
4. Determine tangent of this zoom angle.
5. Effectively move camera to ground plane to determine
full width (127 pixels) of X row at this Y distance.
(Y length cm. * tan theta = full width in cm.).
6. Now know full width of X row in real world values.
7. Determine real world distance of X pixel coordinate
value.
8. Use proportion to find actual X value in real world
units.
((pixel offset * full width in cm) / 127 pixels)
= displacement from center in cm.
9. Now have X and Y coordinates of row edge or center in
real world units.
10. Use these units to determine steering angle.
”~l TARGETFfiXEL
_..——-"’"‘fl‘ “““““ l/::-
CAMERA d,,a—"’:: ......... 1
,.——*:Z ****** ' ZKFTXELS FROM CENTER
; 32:13:: ______________ .NADIEL_—_T_'
L—————‘ “‘~~--I5_ZOOM ANGLE L‘
__ P 128 PIXELS
IOP VIEW “‘-L )

Figure 3-15.

———__..

X ROW INTERSECTING
GROUND PLANE

Procedure for mapping X coordinate value.

74

---- FLOW DIAGRAM ----
Determination of steering angle.
Only the tangent function is used.
This is executed once each position update.

1. Know now the X and Y values for the row center or edge
in real world cm. values.

2. Use Y value in reference to the front axle.

3. Y value is adjacent side of triangle.

4. X value is opposite side of triangle.

5. The angle formed by the adjacent side and hypotenuse
of this triangle is angle at which to turn front
wheels when steering gain = 1.

6. Determine tangent of this angle.

(X value / Y value = tan theta).

7. Determine this value in degrees.
(Theta in degrees = arctan theta).

8. Multiply this degree value by steering gain constant.

9. This degree value is either to left or right of
center.

10. Now have angle to which to turn front wheels.

CONTROL DISTANCE

 

L3} EDGE OR CENTER
2 l /
UJ I I
CE I /
LlJ I I
LL I /
LU I I
Q: I I
LU l /
I— I I
UZJ: ,' HYP.
0! ,’

. \/ THETA

I

 

I

I

| l

F—TTT
l

I
, l
, l

VGRT I
a, / CONTROL DISTANCE REFERENCE

I

FRONT RIGHT WHEEL.

 

Figure 3-16. Procedure to determine steering angle.

75

sending a control word to the slave computer. This control
word tells the slave computer to start or stop the tractor
and how fast to move the tractor. The slave computer then
responds and sends to the master computer the two bytes of
distance information.

One weakness in this communications interchange is
that no verification is made by one computer to guarantee
that the data are received correctly by the other computer.
However, use of the system has shown that if both computers
successfully complete their respective transmission and
reception of data, then no error had occurred. "Watchdogs"
are used in the master computer to monitor the handshake
line from the slave computer. If an error occurs in the
communications protocol, a watchdog is activated causing
the master computer to stop the present communications
interchange and reset the slave computer. At this point
the master computer initiates another communications
interchange. Redundancy in the communications attempt,
when an error is detected by a watchdog, is the method used

to resolve errors in communications (Figure 3-17 to 3-19).

3.7.2.4 ROW CENTER, OR EDGE DETECTION SOFTWARE

Two types of row or line follower software were
developed. The first type uses contrast within the image
to determine an edge to follow. This method was fairly

simple to implement and provides a means for developing the

76

( TO SLAVE )

 

 

 

 

 

\1/
SAVE CONTROL
BYTE
L I
SET ERROR CI
CNT-O j

 

 

 

RESET ERROR I /RESET SLAVE7

 

 

 

 

 

 

 

 

 

 

PLAG 1 1* COMPUTER /
I ”I
I RELOAD CONTROL
\y BYTE
/\/\ /'[\
/ NO
/ DO /7 /////L\\\\
/ COMM ;/ // n \\YES
/AL\(,AL\ \\\\GNT-5?’/>—-—--7
\L \\\ /// :
CHECK ERROR >f7' I
FLAG I

 

 

 

 

IINCREMENT
LERROR COUNT j

I
i A
K ‘i
/'/ \JES I

<g\\: TRUE? /, ~
\2/ fi
1/ / SET DISPLAY "

 

 

 

 

\

 

 

 

 

 

 

NO [ACORN ERROR
SAVE E I
[HSTANCE COMM OK I

I I
EXIT R

Figure 3-17. Flow chart of communications exchange. This
flow chart shows the redundant communications attempts if
one attempt is not successful.

77

(START COM)

/SEND T7
CONTROL :
BYTE /*

/
;/SET 7 SLAVE COMPUTER WAITING
HANDSHAKE/ TO RECEIVE BYT:

 

 

 

\L

READ
HANDSHAKE
Z J

2L 1W3
<\EffiéHAKEi>*—-%K\\ WATCHDOO::>;E§1_7
I
;E'YE8 \‘\//// S
5

/SET /

Z/ HANDSHAKE’ SLAVE COMPUTER READY
J, TO SEND BYTE

READ /
HANDSHAKE/< »
TNO

 

 

 

 

 

 

/READ 7 ;
DISTANCE 3
L HI BYTE ,

IAT W

 

Figure 3-18. Communications procedure with slave computer.
This flow chart shows the order of the data exchange and
where the watchdogs are used to resolve missing handshakes.

78

r \l/A IB
/SET :7 T
/ HANDSHAKE/’
/ /

READ ST7
HANDSHAKE/<

A lN

/ \. NO /' ‘ YES I

HANDSHAKE/ WATCHDOC/ _A
/ \ //
YES V

V

 

 

 

 

/
.4‘ /

 

READ
Z DISTANCE /
SLAVE COMPUTER READY
LOW BYTE/ To SEND BYTE

 

 

‘ 7
L/EIENTDSHAKE /

READ 7
HANDSHAKE //< T
NO
,//)L\\ ///\
\\ NO / YES
HANDSHAKE ;>——%K<Qdfi:fj389 —‘9

YES

(EXIT >< ERROR FLAG <—-——'

I

 

 

 

 

 

   

 

 

 

 

Figure 3-19. Communications procedure with slave computer,
continued.

79

steering control software. The second method actually
calibrates itself to a particular color and finds the
average center of the row or line to follow. Whereas the
first method was used to test the basic performance of the
guidance system and develop the steering control software,
the second method increases the robustness of the system
and provides a means to actually follow a crop row (Figure

3-20).

3.7.2.4.1 CONTRAST MODE ROW FOLLOWER

The contrast row follower requires a bi-modal
distribution in the image histogram. A histogram is taken
of the window area to find the color band with the best
discrimination. A smoothing filter is passed over each
color band’s histogram to reduce noise and accentuate
trends. The filter adds one-half of the target value plus
one-fourth of the previous value and one-fourth of the
successive value. The sum of these values is placed in a
new histogram at the same respective location to prevent
the new values from confounding the old values.

After smoothing, each new histogram is tested to find
which color band has the lowest valley between the two
peaks. The color band with the lowest valley is selected
as the color band to use for locating the edge. The color
band with the lowest valley should provide the best
contrast between the row (object) and dirt (background)

(Levine, 1985).

8

 

POWER LP /
RESET

 

INITIALIZE SYSTEM

J/

/IwmwmmaA
/ CONSTANTS

/

\L
MAP IMAGE PLANE
TO REAL PLANE

\I/

DETERMINE TARGET
CG.0R VALUES (1)

\L

‘ BEGIN TRACTOR
MOTION

 

 

 

/
/

 

 

 

 

 

 

 

0

ELOWCHART OE
EXECUTIVE LOOFJ

 

STOP TRACTOR ’
- MOTION

7

\V

DETERMINE IF GAP
(R ED OF ROW/LINE

T

 

 

 

 

 

     

_.___...‘_.__4-.—.— _——_—————-——

GM
\ /
‘r10
31

 

 

 

 

FIND ROW /
LINE

 

 

 

     

NO

DID IT FIND
ROW/LINE?

 

 

I ,%/ READ KEYBOARD //
/

 

 

 

f

DO LEFT OR RIGHT
HAND TURN

 

 

 

 

 

 

YES
CONTINUE

N0

 

STU’I EXIT

F

Figure 3-20. Flow chart of

executive 100p. This flow

chart shows the order in which guidance control software is

executed.

81

The color scale corresponding to the valley is used as
the threshold to classify pixels into object or background.
Values on one side of the threshold are object, values on
the other side are background. The important issue is not
which side of the threshold is object and which is
background, but rather to find the boundary between them.

Next a raster scan is used in the window area to
locate the edge of the row. Scans go from left to right
and stop when a pixel is located that is on the opposite
side of the threshold from the first pixel in the scan
line, or the other side of the window is located (Figure 3-
21). For each scan line that crosses the threshold, the X
coordinate of the pixel is saved to help in determining the
location of the edge. After the scans are completed for
the window (typically 16 scans), all the X values from scan
lines that crossed the threshold (but not scan lines that
stopped at the opposite side of the window) are added
together and an average is calculated. This X value,
coupled with the Y value corresponding to the center of the
window, provides a single coordinate point of the row edge
(left side of the row). This point is used both as the
control point for steering the tractor and as the center of
the next window. When the coordinate point is used as the
control point for steering it is on the real, ground plane
and when it is used as the center for the next window it is
on the virtual camera image plane. This method allows the

window to float with the steering control point.

 

 

 

 

“99 VCRT ‘ A99 “99

 

 

 

 

 

 

 

 

 

 

 

II'igure 3-21. Contrast mode row following. View of VGRT
33d use of scan lines in contrast mode to determine row
e ge.

83

The contrast edge follower is fast and easy to use
because no calibration is involved. During normal
operation, a new histogram is taken for each new window.
This provides a threshold that is consistent with the light
levels in the window portion of the image. Weaknesses of
the contrast follower require a relatively high degree of
contrast between object and background and that the
contrast only occurs at the boundary between the two.
Contrast that occurs elsewhere in the image is considered
to be noise and, in large amounts, can confuse the edge
follower and provide a false control point. Noise can be
in the form of shadows that fall across the edge or
variations in the color of the edge and background. One
precaution to note: no significant portion of the histogram
can be either in cut-off or in saturation. This last item
is principally dependent on the light adaptation and

dynamic light range of the camera.

3.7.2.4.2 COLOR DISCRIMINATION MODE ROW FOLLOWER

The color discrimination row follower more fully
utilizes the advantages of having three color bands than
does the contrast edge follower. Whereas the contrast
follower uses only one of the three color bands at a time,
the color discrimination follower uses a combination of all
three bands all of the time. In a row situation, color can

be a simple clue in identifying the plant row from the

84

background dirt. The added complexity of having a color

video digitizer may be offset by having relatively simple
row detection algorithms than would be the case with the

inherently simpler monochrome video digitizer.

The color discrimination row follower is based on a
calibration sequence that determines the percentages of
red, green, and blue in the target color. When
initializing the program, a still image is taken in which
the target color must be located in the lower center of the
field of view. First a histogram is taken of the lower
area of the field of view. A threshold is determined as
previously described and the background is set to a zero
value. An average color intensity is determined for each
color band based on the remaining target color. These
average intensities are then converted to a percentage of
the whole color. The target color is determined by a mix
of percentages of the three colors that compose it. By
converting to a percentage basis, changes in light level
have a reduced effect on altering the apparent color of the
row. This can reduce the confusion that shadows cause to
the VGRT.

Tolerance limits of plus and minus ten percent (of the
average percent value) are set for each color band based on
the percentages determined in the calibration sequence.
Each pixel that is tested in an image is converted to a
Percentage of its composite colors. These percentages are

tested to see if they fall within the tolerance limits of

85

the calibration colors. Each color band can have a score
of one or two. A one indicates that it is completely above
or below the calibration limits, a two indicates that it
falls within the tolerance limits. For all three color
bands, a total score can range from three to six with a
score of six indicating a very good match.

Another improvement in the color discrimination row
follower is the method of locating the control point.
Instead of locating the edge of the row, a new method of
finding the approximate center of the row is used. Scan
lines at the upper and lower edge of the window start at
the outside edges and scan in towards the center of the
image (Figure 3-22). A scan line stops when it finds a
pixel that has a score of five or six. This indicates that
the pixel color matches the calibration reference colors
well. If no color match is found, the scan lines stop when
they meet in the center. All four scan lines have to find
a valid color match in their respective pixels for the
coordinates of these four pixels to be considered valid in
determining the control point. The four X values are
averaged to provide the X value for the control point. The
Y value is based on the center of the window.

There is more computation time involved with the color
discrimination row follower than the contrast edge row
follower. Yet, color discrimination is generally less

Susceptible to noise and provides a control point that is

86

 

RDA ,
’,,,___££ Hancrvmw
RAMS 53 Cd:i‘\\\\\\\ /

/

l “““ "" /
Q9!) ______ 4);; CL _ _ _ CGJTRQ. DISTANCE
T

 

 

 

:1
J
EEEEi:] ,

 

[_____

%%%%%%%%%%%%%%€9%
%>
_%j

%€9%%%€9%% .

 

 

 

Figure 3-22. Color discrimination mode row following.
View of VGRT and use of scan lines in color discrimination

mode to determine row edge.

87

more immune to undulations in the row edges. Because the
color discrimination row follower "looks" for a specific
color, it is less prone to confusion by spurious areas of
contrast. Varying light levels also do not confuse it

because colors are converted to percentages. In the case
of shadows, as long as they are not completely black, the

row can be discerned despite them.

3.7.2.5 HEADLAND TURNS

Dead reckoning is used to guide the tractor through
headland turns. Each turn can consist of one, two, or
three segments. The operator must first teach the tractor
the turn; then the tractor will repeat the turn whenever
the operator wishes it to do so. Left and right hand turns
are taught separately.

In the "learn" mode, the master computer sets the
distance/direction arrays to zero and about once each
second records steering angle and distance information in
the array. When a turn is played back, the master computer
monitors distance information from the slave computer.

When a distance is reached that matches one found in the
array, the front wheels are turned to the angle found at
that distance. This continues until the length of the turn
is completed. At the completion of the turn, the tractor
stops and waits until the tractor operator initiates

further action.

CHAPTER IV

RESULTS AND DISCUSSION

4.0 INTRODUCTION

Tests of the Vision Guided Robot Tractor (VGRT) were
designed to evaluate the performance of the system as a
whole. Tests started with the VGRT following a continuous
line with varying forward tractor speeds, control
distances, and position update rates. The continuous line
tests were considered to be the easiest scenario for the
VGRT and would establish what should be its best
performance. As new, more difficult tests would be
attempted, any changes in VGRT performance could be
evaluated against the original benchmark performance tests.

Testing progressed to greater levels of difficulty by
modifying the line which the VGRT followed. Spotted lines
were used to test for performance following a less than
ideal line. A step was introduced into a continuous
straight line to test the time response of the system.
Human operators' manual steering performance was used as a
comparison for evaluating VGRT performance. The VGRT was
tested on "fuzzy" lines, lines that could be compared with
crop rows in a field. A "user friendliness" test was done
With operators unfamiliar with the VGRT to test the ease

‘With.which they could master the system and operate it.

88

89

And finally, the VGRT was taken to the field to see if it
could actually follow a crop row.

To record VGRT performance, a second video camera was
placed at the back of the tractor and positioned so that it
could record a downward-looking view of the right rear tire
and the area on the ground directly behind the tire. When
the tractor was in motion, the camera was set to record
this view which contained the line the tractor was intended
to follow (directrix) and distance marks which were made on
the tire (Figure 4-1).

The tread face circumference of the right rear tractor
tire was divided into thirds at which points white stripes
were painted on the tire. These stripes passed within the
field of view of the second video camera providing a fixed
distance interval of 61 cm of tractor travel. To further
expedite the measurement process of recording undulations
(lateral deviations from the desired path) in the tractor’s
path of travel, a meter stick was mounted in the field of
view. This meter stick was rigidly attached at the rear of
the tractor and positioned 2.5 cm above the floor
perpendicular to the line the tractor followed. With this
arrangement, the second video camera could provide a record
of the path the tractor traveled which would contain a
"tick" mark at every 61 cm interval of tractor forward
travel. At each mark the lateral position of the VGRT

Could later be read from the meter stick.

90

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

T——1 i TRACTOR
I
a l I
SECOND CAMERA/g _ T
II: T ‘
/ \ ‘
FIELD OF VIEW/E — ”I I i
’ ' STRIPES
/ O I
I .
“REAR WHEEL
METER STICK“
\ ,
/ E27

Figure 4-1. View of second camera placement, used to
record tractor path of travel, and distance "tick" marks.

9O

 

 

 

 

 

 

 

 

 

 

 

 

 

F——n- i ; TRACTOR
g p l l
SECOND CAMERA/? __ T
7;:
/ \ :
FIELD OF VIEW/E +- 4 i i
a T
’ STRIPES l
l O A,

 

 

 

 

/

METER STICK\,

\ ,/
/ £14

~REAR WHEEL

Figure 4-1. View of second camera placement, used to
record tractor path of travel, and distance "tick" marks.

91

At a later date the standard 1.3 cm VHS video tape was
played back to review the tractor's guidance performance.
Every time a white stripe on the wheel would pass within
the field of view a written record was made of the position
of the line that the tractor was following with respect to
the meter stick. These resultant meter stick values were
plotted to give a visual representation of the tractor’s
performance. To quantify the tractor’s performance, a
standard deviation of the lateral tractor travel was
determined from the meter stick readings.

Further testing was done on some of the individual
components of the VGRT system to evaluate their effect on
total system performance. The amount of free play in the
steering system was determined. Factors affecting the
position update rate were evaluated. Special attention was
paid to the camera’s ability to resolve colors under
different light conditions. The last aspect of the VGRT
testing, performance, is also one of the most significant
for this research effort.

The other VGRT tests dealt with particular aspects of
‘the total system performance; the last test was actually a
‘test that continued throughout the entire evaluation of the
‘VGRT. This tested the performance of the system as a
TMhole. Was the system robust, or did it take a lot of
adjustments to get it to work for one test, then more
adjustments for the next test and so on.

As is naturally the case with most engineering

92

prototypes, there was some software and hardware
development, and the subsequent elimination of bugs, as new
tests were devised. The important question was: during a
given test (or stage of development of the VGRT) was the
performance consistent? Could the person conducting the
test expect the system to work in spite of dramatic changes
in ambient temperature, vibrations, repeated power-up

cycles, and even handling by inexperienced operators?

4.1 CONTINUOUS LINE TESTS

A continuous line 3.8 cm wide by 46 m long was used as
a basis to test what would be the best performance of the
system. The test track was indoors and under fixed
lighting so that the only variables in the VGRT system
could be controlled at the tractor. This test was used to
establish the basic performance characteristics of the
VGRT. As later tests became more complex, comparisons in
;performance could be made to evaluate the ability of the
‘VGRT to perform the more complex guidance tasks. All
(zontinuous line tests used the contrast edge-following mode

of VGRT guidance .

‘4.lnl.l FIXED STEERING GAIN TESTS
The first series of tests involved running the tractor
art four different speeds, four different control distances,

five different position update rates, and a fixed steering

93

gain. This test was to indicate the effect that control
distance and position update rate would have on steering
stability for a given speed. The speeds selected for the
test were 0.6 km/hr, 1.8 km/hr, 4.0 km/hr, and 9.4 km/hr.
These speeds gave roughly a doubling of speed for each
increment in speed and were compatible with the gear ranges
on the tractor. They also covered the speed range of the
tractor from its slowest speed to its fastest speed.

The control distances selected were one, one and one-
half, two, and three meters (ahead of the front axle). The
position update rates were eight, four, two, one, and one-
half per second. The results were divided by category into
different position update rates. The VGRT was run once for
each trial situation until it failed to follow the line the
full distance. For a trial run to be successful the VGRT
had to follow the line for the full distance. The results
for computer control appear in Tables 4-1 to 4-5. Each
result value in the table represents the standard deviation
Yof the tractor’s lateral displacement about the line in

<3entimeters, (each datum was accurate to 0.5 centimeter):

94

 

Tables 4-1 to 4-5 indicate the change in VGRT line
following performance when the position update rate is
varied. N - at least 70 samples per test run. Lateral
position accuracy to +/- 0.5 cm. The table results use
standard deviation as the means to express variability.
A "-” sign indicates an untried test.

 

 

 

— ° ' - s s c.
QQDEIQI KRLDI
W In).
Qlé llfi ilQ iii
1.0 1.8 cm 103 -
1.5 1.6 2.0 3.0 3.9
2.0 2 6 2.2 4.5
3.0 2 2 3.5 4 9
I;pig,4-;3 Steering Variability - 4 Updateszsec.
QQDLIQI BELL:
Distance (m)
QLQ ng 2L9 2_g
1.0 1.7 cm 2.1 -
1.5 1.9 3.2 2.5
2.0 1.3 1.2 4.7

 

Table 4-3- “W

 

Centre]. mm:
mm

M 14.5. 1.9—Q 9.4
1.0 2.6 cm 2.4 -
1.5 1.5 2.3 6.5
2.0 2.4 4.2 -
3.0 2.4 3.1 9.6

 

 

IQDL§,4'4- .Steering Variabiiity - l Upda§e1sec.

 

93111219.]. 1mm;
DLELEDEQ lfll

0.6 1.8 4.0 9.4
1.0 1.8 cm - -
1.5 1.9 - -
2.0 5.3 8.0 -

 

96

 

Table 4-5. Steering Variability - 0,5 ngates[see.

Centre]. mun:
Distanss m

(2,6 be 3.0 9.4
1.0 - - -
1.5 5.1 cm - -
2.0 6.6 - -
3.0 6.2 - -

 

As previously noted, a test run was considered successful
only if the VGRT could travel the full distance. A blank
space in tables 4-1 to 4-5 indicate that the VGRT was not
able to travel the full length of the line. Failure to
follow the line was usually a result of the VGRT going out
of control and losing the line as its control signal.

For comparison, five human operators were asked to
drive the tractor over the same course. Each operator was
told to drive the tractor with the line under the right
front tire and to follow it as best as could be done. The
Ihuman operators drove the tractor at each of the same four
:speeds and were allowed to select their own control
(distance. A steering wheel was installed on the VGRT for
‘this purpose. The operators had no practice or minimal
Practice in steering the tractor and used their normal

‘Lision for steering, see Table 4-6 for results.

97

 

Table 4-6 provides a comparison between human
operators and computer operation of the test tractor. N =
at least 70 samples per test run. Lateral position
accuracy to +/- 0.5 cm. The table results use standard
deviation as the means to express variability.

-6 ' V - 0 t s.

993123.91: mu;

Qlé 115 £12 213
1 0.7 cm 0.9 1.1 1.5
2 1.0 1.5 2.3 6.1
3 0.5 1.0 0.9 2.0
4 0.6 0.6 0.9 2.3
5 0.9 1.0 1 O 3.8

 

4.1.1.2 VARIED STEERING GAIN TEST

To test the effect of steering gain on tractor
steering performance, the control distance and position
update rate where held constant. The tractor was run at
each of the four speeds with five different steering gain
‘values. The following steering gains where used: four,
two, one, 0.5, and 0.3. For each position update the
steering error was determined in degrees then multiplied by
‘the steering gain factor. The results of the test are as

‘follows in Table 4-7:

char
cons
pos:
stax

0.5

0.3

4.1

C011

Per

nat
sta
div
the
Sec

C011-

98

 

Table 4-7 shows the VGRT's change in performance for a
change in steering gain. The position update rate was held
constant. N a at least 70 samples per test run. Lateral
position accuracy to +/- 0.5 cm. The table results use
standard deviation as the means to express variability.

 

Table 4-7. St e ° V ° -
Varieg Sbeering Gain.

gain 1934111;

.0.._§ LLB. .4_.Q 94
4.0 0.7 cm 0.7
2.0 1.1 1.7 1.5
l 0 1.5 2.0 3.0 4.0
0 5 8.7 8.7 9.7 9.2
0 3 12 12 15 17

 

4.1.2 CONTINUOUS LINE STEP TESTS

After exhaustively testing the VGRT system on a
continuous straight line to establish basic parameters of
jperformance, a step of 30cm was introduced into the line to
"ring" the system. This step input would determine the
natural response of the system and indicate the degree of
stability in the steering control. The step tests were
(divided into three series, the first tested the ability of
‘the VGRT to respond to both a right and left step and the
Second tested the ability of the VGRT to respond to a

constant step at different speeds. The third step test,

test

CORE

int:
dete

eitr

cor:
dete
affe

to 1

com;
ope]
woul
res;

Opel

were
Cri1
Crii
P011
time

fact

Cent

99

tested the ability of human operators to respond to a
constant step at different speeds for comparison purposes.

The purpose of the first step test, that of
introducing a left or right step into the line, was to
determine if the VGRT responded in a similar manner to
either step. This would give an indication of the effect
of camera placement on VGRT response to steering
corrections in either direction. The concern was to
determine if an offset camera placement would have an
affect on steering response that would differ from the left
to the right side.

The second and third tests were to provide a
comparison between VGRT steering response and human
operator steering response. The results of these tests
would indicate if the VGRT could provide a steering
response to a change in course similar to that of a human
operator.

To describe the results of these tests, four criteria
were used. For a given test only one or more of the
criteria may be used depending on the applicability of the
criteria. Criteria are used sufficient to indicate the
point of the comparison test. The four criteria are: rise
time (RT) (10% to 90%), percent overshoot (P0), damping
factor (DF), and natural frequency (NF).

For the purposes of this research effort and
simplicity the VGRT is considered to be a second order

control system. Due to the emphasis of this research (non

math
the I
deci
"rin
VGRT
resp
orde
syst

comp

4.1.

upda
stee
0.6

dist

gair

100

math intensive) no mathematical verification is given for
the decision to consider this order of control. This
decision is based on observation of the results of
"ringing” the system. The plotted results of ringing the
VGRT system with a low steering gain (0.5) indicate a
response that could be adequately described by a second
order system (Figure 4-2). It is felt that a higher order
system is both unnecessary and would needlessly over

complicate the mathematical analysis of the results.

4.1.2.1 RESULTS OF RIGHT AND LEFT STEP

For the left versus right step test the VGRT’s speed,
update rate, and control distance were held constant. The
steering gains used were: 0.5, 1.0, and 2.0. The speed was
0.6 km/hr, eight updates per second, and 1.5 meter control
distance were used. The results for the three steering

gains are as follows (see Figures 4-2, 4-3, and 4-4):

101

STEP—FUNCTKNJRESPONSE

STEP - 0.3m: GAIN - 0.5

.1)
C.)

 

 

LATERAL DISPLACEMBIT (c m)

 

 

 

 

50 -I ,/\.
/ \.. Ifi/‘K‘i .
. . / \x I, ‘4. I,
,*** \
o.« 7‘ / .. \\ f/
w T .I / S. \. / /\
.4 / ‘ \. f I»
/ / \ \ q /
f I \ \ f / l
30 490000000003/0ojooooootfoooooofioooooooo/ooooooooood'
f / ‘x xx, \\ //
l / ‘x—r" '\ /
20 "I f/// \__/
{/’
10 - fi/V
/
0 I!"
‘10 T I I T I I l I l T T T T I
0 4 8 12 I6 20 24 23
TRAVELin meters
~——-U3TSEP + RBHTQBD o HWDWEDPHH

Figure 4-2. Graph of steering response, right and
left step. Gain = 0.
Test run left: DF = 0.
Test run right: DF = 0
N > 40.

DF 2 Damping Factor, NF = Natural Frequency,
P0 = Percent Overshoot.

NF = 0.07HZ, P0 = 50%

5
2:
.3, NF = 0.09HZ, P0 = 40%

102

 

 

 

 

 

bTEP—FUNLTION RESPONSE
TIER - 0.75m: '3‘!” - 1.0

60 ~

50
'E- .8
.. -+u
U
V //—'—'—\.
I: /' \\.
6] ' .20- \
5 30 >99999999999!¢?$00069ooooooooooooooo00000000000006
H .14 W """"""""""
3’ l
r‘
E A
5 20 4 1%
3" /
“ 1
U
‘- Olfl -
3 'J //;7

/'y+
0 4/
L*
'10 I I I I I T I r I I I I I I
O 4 8 12 16 20 24 28
TRAVEL in meters
-——LHWSEP I- mmnsny o INEMIOPMH

Figure 4-3. Graph of steering response, right and

left step. Gain =
Test run left: DF =

1.0
0 3 RT = 24 seconds

’ I
Test run right: DF = 0.7, RT = 19 seconds
N > 20.

DF =

Damping Factor, RT = Rise Time.

LATERAL- DISPLACE MI;

103

STEP-FUNCUON RESPONSE

SEP-Oamcnm-2D

 

 

NT (cm)

 

 

 

50

I
50 4
40 "I

r \r
/
/’
3D +990¢99999990’Ooooooooooooo9999099999999999999990<
IAI I .A'
A!"
20 d f
717
I0 1 a}
x/ x
n y{:L 1
- .r"”‘+4

‘10 T r I T T I I T I I I I I

O 1. 3 I2 16 a.) 2+ :8

TRAVEL in meters
———LHTSEP I- WQOSEP <> WENEDPMH

Figure 4-4. Graph of steering response, right and
left step. Gain = 2.0

Test run left: DF = 0.7, RT = 20. seconds

Test run right: DF = 0.7, RT = 20. seconds

N > 20.

DF = Damping Factor, RT Rise Time.

 

4“-

tpcpr

104

4.1.2.2 RESULTS OF COMPARISON STEP TESTS

The VGRT was run at three different speeds with
position update rate (8 per second), control distance
(1.5M), and steering gain (1.0) held constant. For
comparison, five human Operators were asked to operate the
tractor over the same 30cm right step. The results are as

follows in Tables 4-8 to 4-10:

 

Tables 4-8 to 4-10 show the results of the test
tractor being rung with a right step wave. The tables
provide a comparison between human operators and computer
control. N = at least 16 samples per run. Lateral
position accuracy to +/- 0.5 cm. DF = damping factor, RT =
rise time, and P0 = percent overshoot.

Table 4-8. Step Comparison Tests.

Qesreter SEEEQ 216 3min;
22 RT 29

VGRT 0.7 15.35 3

H1 0.7 8.55 3

H2 1+ 11.95 0

H3 1+ 13.65 0

H4 1+ 17.05 0

H5 1+ 15.35 0

 

H2

HI

H!

K?

105

 

- s s
QREIQLQI SREED llfi 8541;
DE BI 29

VGRT 0.7 4.88 3

H1 1+ 3.08 0

H2 1+ 3.65 0

H3 1+ 6.03 0

H4 1+ 7.25 0

H5 1+ 6.05 0

 

 

Table 4-10. Step Compabisob Tesbs.

Qesreter S2332 £19 Emit;
2: RT 29

VGRT 0.7 2.35 3

H1 1+ 1.55 0

H2 0.4 0.85 22

H3 1+ 4.05 0

H4 1+ 2.85 0

H5 1+ 2.05 0

 

4.1.3 RUNOUT TESTS

A steering runout test was done on the VGRT to show

that it could not have "accidentally" followed the line

cons:
prev
in t
runo
usin
the
unti
the
was
the
st0]

p08

4.1

sec
km/
PEI
dev

The

106

consistently and with the type of accuracy indicated in the
previous tests. Along with the runout tests, the free play
in the tractor’s steering system was also determined. The
runout test consisted of starting the VGRT on the line
using the vision guidance system to align the tractor to
the line. The tractor was allowed to travel down the line
until it reached the beginning of a three meter section of
the line. At the beginning of the test section the camera
was shut off and the tractor allowed to travel blind for
the three meters and then the tractor was stopped. After
stopping the tractor the lateral offset of the tractor’s

position from the line was measured.

4.1.3.1 RESULTS OF RUNOUT TESTS

Six test runs were made over the three meter test
section of the line. The tractor was traveling at 0.6
km/hr with a 1.0 meter control distance at eight updates
per second. The results are in centimeters of lateral
deviation from the line to either the left or the right.

The results (Table 4-11) are as follows:

107

 

Table 4-11 shows test tractor runout test results.
Runout indicates that the tractor could not have
accidentally followed the three meter line.

Ifihlfie4’11- RE§ELE§_Q£_BQDQBE_I§§£§;

Bun 931193.120 5.1515
1 64cm Left
2 57cm Right
3 54cm Right
4 57cm Left
5 55cm Right
6 52cm Right

 

By measurement the VGRT was found to have 20 to 22
degrees of steering wheel play, i.e. 20 to 22 degrees of
rotation of the steering wheel to take the slack out of the
steering system. This steering wheel play is with the
front wheels locked against rotation. The front steering
wheels have three to four degrees of free play when locked
against the steering linear actuator. This steering free
play is a reflection of the amount of play in the steering
gear, ball joints, front axle bearing, etc. Stop to stop
travel of the front wheels (50 degrees) represented one
turn of the steering wheel. The linear actuator could move
the front wheels lock-to-lock in two seconds. While the
human operators had a steering system that had more free

play, they also had the advantage of both a variable rate

108

of steering response and variable control distance.

4.1.4 VGRT TILT TESTS AND RESULTS

To determine what effect camera platform tilt has on
steering heading, for a constant VGRT heading and offset, a
jack was used to tilt the VGRT. The VGRT was aligned with
a continuous line and then not moved while a jack was used
to alternately lift one side of the VGRT or the other side.
As the VGRT was raised the steering angle (heading in
degrees) was noted. The steering gain was set to 1.0 and
the control distance was 1.5M. Table 4-12 indicates the

results of this test.

109

 

Table 4-12 shows the change in heading for a specified
tilt of the VGRT about its major axis.

IAh12.12121__¥§BI_1112_I§§§1

5195331550 12331193221111: 8559159

Left 0 128

2 128

5 128

8 128

12 128

0 128

Right 0 128

2 128

5 128

8 129

12 129

0 128

 

4.1.5 PIXEL WIDTH OFFSET TESTS AND RESULTS

A determination of the effective pixel width was made
at the 1.5M control distance. The VGRT was set to eight
position updates per second, 0.6 km/hr, 1.5M control
distance, and steering gain was 1.0. The VGRT was

positioned to follow a straight, continuous line and the

110

lateral position with respect to the line was measured for
different center positions.

The width of the camera’s field of view in pixels is
256 pixels. The center of the field of view at the control
point is usually set to 128. This center value can be
varied to alter the VGRT’s offset with respect to the row
or line. Four different center values were used and their

respective offsets measured, see Table 4-13.

 

Table 4-13 gives an effective pixel width at the real
ground plane and VGRT offset for different center values.

TabieL4-13. Results of Pixel Width Tests.

 

Center gaine Offset leni
128 5 - 8
130 10 - 13
132 13 - 15
134 18 - 20

 

This would indicate that a pixel has an effective width of

two to three centimeters at a control distance of 1.5M.

4.2 INTERMITTENT LINE TESTS
The next phase of testing was a start in the VGRT
testing that would begin to approximate a row crop

situation. An intermittent line was used to investigate

111

the change in VGRT performance to a less than ideal line
following situation. The same contrast mode edge-following
program was used but now the line had gaps placed in it.
The gaps in the line simulated the gaps between plants in a

crop row.

4.2.1 INTERMITTENT LINE TEST RESULTS

For the intermittent line test, tractor speed, update
rate, steering gain, and control distance were held
constant. The 46 m test line was made up of five-by-eight
cm rectangles of paper (Post-It notes). The papers were
placed with their major axis in line with the line and the
gap was measured from the end of one paper to the beginning
of the next paper. The tractor speed was 0.6 km/hr, the
control distance was 1.5 m, the position update rate was
eight per second, and the steering gain was set at one.

The results (Table 4-14) are as follows:

 

Table 4-14 shows the performance of the VGRT when
following a line with gaps introduced into it. N = at
least 70 samples per test run. Lateral position accuracy
to +/- 0.5 cm. The table results use standard deviation as
the means to express variability.

Table 4-15. Sbeening Vaniabiiiby - Intennittent line.
MIME). G_ap_sl.i_rIer_n1
Q 15 Z; 29 2Q

O

0.6 1.4 (cm) 2.7 4.0 3.8 4.

 

112

4.2.2 FUZZY ROW TESTS AND RESULTS

A row resembling an actual crop row was painted on
concrete to further test the VGRT in a situation more
closely resembling that of an actual crop row. The row was
green in color and made of "leaves" that were approximately
13 to 15 cm long by five to eight cm wide. The row was
23 m long (Figure 4-5). The two types of row followers,
contrast mode, and color discrimination mode, were used to
guide the VGRT on the row. "Weed" blobs were added to the
row to test the response of the two types of row followers.
The weed blobs were the same green color and were
approximately 71 cm long by 23 cm wide.

The tests involved different placement of the weed
blobs with respect to the row. For each placement of weed
blobs a test run of the VGRT was made using contrast mode
following and then color discrimination mode following.

The first two test runs used each type of row follower with
no weed blobs for a clean-row comparison control. For both
types of row followers the steering gain was set to 2.0 to
provide a greater indication of the different responses of
the row followers. For all test runs the VGRT speed was
0.6 km/hr, eight updates per second, and a 1.5m control
distance.

The test results were compared using the standard

deviation (SD) of the VGRT's lateral deviation as

WEED
BLOBS

\ \

113

Q CREEN IN COLOR.
Q Q PAINTED ON

“a
e
a
m
p
8
Rd
fl\

\
g9 SIMULATED

fl/ CROP ROW

99/

Figure 4-5. Fuzzy row test, simulated crop row and weed

blob.

114

it traveled the row and the maximum deviation (MD) from the
row center. The results (Table 4-15) are as follows (all

results are in centimeters):

 

Table 4-15 shows the test results of the VGRT
following a simulated field crop row. Results are
expressed in variability about the line (standard
deviation) SD and maximum deviation from the line MD. N =
at least 30 samples per test run. Lateral position
accuracy to +/- 0.5 cm.

- S ' Va 1 - 2 Row
Test Edge m:
E. HE S_ MD
Clean row 0.6 2 1.1 2
One blob on each side 2.8 8 0.6 2
Two blobs same side - together 4.7 14 2.0 6
Two blobs same side - apart 3.9 9 1.0 2

 

4.2.3 CROP-ROW-FOLLOWING TEST AND RESULTS - CORN

On June 20th, 1989 the VGRT was taken to a corn field
for actual row-following tests. The results of the test
would either he go or no-go, that is, the VGRT would either
follow a corn row or it wouldn’t. The tests took place on
a partly sunny day from 1200h to 1430b EDT. The corn rows
were oriented east - west on 76 cm centers and the corn
plants ranged from 20 to 36 cm tall. The color

discrimination mode row follower was used with a control

115

distance of 1.5m for the VGRT speeds of 0.6 km/hr and 1.8
km/hr, the control distance was 3.0m for the VGRT speed of
4.0 km/hr. An ND4 neutral density filter, an ultraviolet
filter, and a linear polarizing filter were placed on the
camera.

The VGRT was driven out into the field and manually
placed such that it straddled one row and had the right
rear tire near the edge of the next row. The VGRT was able
to follow a corn row at the three test speeds of 0.6 km/hr,
1.8 km/hr, and 4.0 km/hr. A visual estimate of the lateral
deviation of the VGRT about the row at any of the speeds
was in a range from zero to five centimeters. The VGRT
seemed to exhibit ample steering stability when following
rows, very similar to what could be expected from a human
operator.

The major problem with the VGRT in an actual row
setting involved the response of the video camera to
sunlight. Under direct sunlight, the camera’s ability to
resolve the difference between green corn plants and brown
dirt was greatly reduced. The severity of this problem was
reduced by placing light filters on the camera. The second
problem was the poor response time of the camera in
responding to a change of light level. A typical change
was from direct sunlight to cloud shielded sunlight. The
slow response time of the camera would sometimes cause the

VGRT to travel blind for a distance of one to three meters

116

until the camera had adjusted to the new light level.

4.3 HEADLAND TURNS AND RESULTS OF TESTS

Headland turn tests were performed on both grass (dry
and wet) and concrete to see if wheel slip would differ
enough to affect the accuracy of a turn. The results of
the headland turns are somewhat subjective as the VGRT will
repeat a turn no better than what was originally taught.
Therefore an effort was made to teach the VGRT accurate
turns that followed approximately the same arcs. A three
point turn was used as it should have introduced the
greatest error. The distance between rows was three
meters.

A turn was evaluated by the VGRT’s ability to
successfully find and follow the target row after making a
turn. After making a turn, a visual inspection was made of
the VGRTs front right wheel position with respect to the
line. The VGRT would then be set in motion to see if it
could find and follow the line. Five headland turns were
made for each test situation, each set of five turns was
based on one learned turn.

In each of the three different test situations, the
VGRT could repeat a turn and successfully find and follow
the row all five times. Since the main criterion for
success of this test was go or no-go, only approximate
position offset and heading errors are given. This is to

indicate the variability of the VGRTs position at the end

117

of a turn with respect to the target row. At the end of a
turn the position offset was consistently less than 15 cm

and the heading error was never more than five degrees.

4.4 POSITION UPDATE RATE TESTS AND RESULTS

As previously mentioned, all software for the VGRT is
written in Assembler language to ensure the fastest
possible execution times. This is critical since there are
some inherent limitations in the vision system. Due to the
type of video digitizer board used, the main computer bus
speed is limited to 6.0MHZ, not particularly fast by
today’s standards. The other main inherent limitation in
speed is the forced delay of 17 milliseconds to let the new
image settle in the video digitizer RAM storage. Each time
a new image is acquired this delay is required to insure
that the image has been fully loaded into RAM, guaranteeing
that the correct video data is present.

As methods were tried to increase the position update
rate, it quickly became apparent that bus speed and time
delay, primarily the latter, were the most significant in
controlling the position update rate. For example, adding
more scan lines (than the original two) in the color
discrimination mode follower would not decrease the update
speed as expected. An increase of one scan line would only
decrease the update rate from eight per second to seven,

two additional scan lines would only decrease it to six per

118

second.

For comparison, the 17 millisecond delay (8000H) was
decreased to see what effect this had on position update
rate. The resultant position update rates as run on the

VGRT were as follows in Table 4-16:

 

Table 4-16 shows the resultant rate of position
updates for different image acquisition delay values.

- 03' 'o v e V ue.
Dialer 141.1115 U_p__date r5125
8000B 10 per second
6000B 12 per second
4000H 18 per second
2000H 31 per second
0010H 198 per second

 

Writing more efficient Assembler code could increase the
position update rate to some extent, but it would be
difficult to achieve the rates that shortening the delay
can achieve. The position update rate can not be increased
by decreasing the acquisition delay without a sacrifice in

image integrity.

4.5 PERFORMANCE OF VGRT AS A SYSTEM

One very important aspect of the testing of the VGRT

was its performance. Performance could be defined as the

 

119

ability of the VGRT to work consistently and robustly.
Robust in the sense that the VGRT would be free of minor
software and hardware glitches that would necessitate
frequently having to rewire hardware and rewrite the
software. This performance aspect of the VGRT testing was
given careful consideration throughout the development of
the system.

It was important to decide, through the development of
the electronic systems of the VGRT, when to purchase an
electronic sub-system and when to build a custom electronic
sub-system. Tradeoffs involved in this decision were costs
vs. convenience of existing "off the shelf" electronic
systems. Either a readily available component could be
easily modified and put into the VGRT system or it would
require a degree of modification that might have
compromised its reliability. If much modification would be
involved, it may have been easier to design and build a
custom electronic sub-system. This custom sub-system can
be built with specific functions in mind and with a
sufficient degree of robustness for the VGRT system.

0f the VGRT electronic's, the slave computer,
interfaces to control the tractor, communications link
between the master and slave computer, and both computer
power supplies were all custom-built electronic sub-
systems. To reduce costs and yet provide the greatest
degree of robustness, custom sub-systems were deemed to be

the best alternative in this engineering prototype. As

120

regards the master computer, a stock, commercially
available, Intel 80286 MS 008 based system seemed to be the
best choice. In this instance, based on cost, ruggedness,
and flexibility, the commercially available unit had the
advantages.

During construction of the VGRT electronics great
attention was given to the stability and noise rejection
capabilities of the master and slave computer power

supplies. The electric tractor created a severely noisy

 

environment that could easily cause computer malfunctions
through an improperly designed power supply. Noise
radiated through a poorly designed power supply is probably
one of the most common ways for hardware malfunctions to
occur. These types of malfunctions can be particularly
hard to deal with as they may occur only at odd intervals
in conjunction with some rare event that is hard to
reproduce.

In consideration of the performance aspect of the
VGRT, a self-contained power supply should provide the
greatest degree of robustness. A self-contained DC to DC
converter power supply could work directly from the tractor
batteries reducing the number of components the system
would require just to get power. Another method would be
to us a DC to 110v AC inverter to provide power to the
stock master computer power supply. A 110v inverter would

have added components to the master computer power supply

121

which would have increased the chance for a failure. Since
the entire VGRT system works from batteries (which by their
nature need to be recharged and do not supply an endless
source of energy), the inefficiency of the 110v inverter
would have reduced the length of operating time reducing
the performance of the VGRT system. A custom power supply
could provide the greatest noise immunity with the fewest
parts which would improve the robustness of the entire
VGRT.

Heat can be as significant as electrical noise in
causing hardware malfunctions. The VGRT was subject to a
wide range of operating temperatures, from zero to 35
degrees Centigrade. This could create problems for the on-
board computer systems. During the design of the power
supplies for both computer systems, the current (I) loads
were measured to determine the size of the heatsinks for
the voltage regulators. Two fans were also installed in
the box housing the master computer. One fan was set to
blow cool air into the master computer case, the other fan
to exhaust hot air from the case. Both of these fans were
ducted to the outside of the box housing the master

computer to ensure the best air circulation.

4.5.1 RESULTS OF PERFORMANCE TESTING
Throughout the entire test of the VGRT (more than
eleven months) there was not one single component failure

in any of the electronic systems added to the tractor.

122

Extremes of test conditions included temperatures from zero
to 35 degrees Centigrade (master computer internal case
temperature as high as 60 degrees Centigrade) and ambient
conditions from bright clear sunlight to heavy overcast and
even rain. And, at no time during any of this test period
could a single malfunction be attributed to a power supply
dysfunction. With this type of robustness, any problems in
VGRT operation could usually be attributed to the tractor
or the video camera. Incorrect software operation or an ‘

incorrect design in a hardware component of the system were

 

seldom the cause of problems.

The two major problem areas in the VGRT were the
tractor itself and the video camera. Due to its age and
lack of previous maintenance, the tractor required a small
amount of constant upkeep. The main problem with the
tractor was relay contacts. The relay contacts were
subject to arcing and pitting which either caused them to
stick together or not make contact at all. Filing the
contacts usually temporarily remedied the situation. The
main problem with the camera was its lack of dynamic range
under varying light conditions. As regards the VGRT
system, the poor response of the camera was the single

greatest contributor to poor performance.

4.5.2 INEXPERIENCED OPERATOR PERFORMANCE TESTS
One difficulty in evaluating total VGRT system

performance is that it is a somewhat subjective rating.

123

With the high degree of familiarity, such as what the
person doing the development work of the VGRT would have,
it would be easy to overlook lapses in good performance,
make a small adjustment, and go on. If other operators are
brought in to run the VGRT that are not familiar with it, a
lapse in performance will present a major stumbling block
to their effective use of the system.

Five people were asked to operate the VGRT in a
simulated field situation. These five people all had some
previous experience operating farm tractors but they had no
experience with the VGRT. They were given a farm scenario
in which to test drive the VGRT (see Appendix 1). They
were evaluated on how quickly they could learn to operate
the VGRT and complete the test farm situation without
outside help. All five operators read the instructions and
completed a successful operation of the test. The times
required to read the instructions and successfully complete

the test are listed in Table 4-17.

124

 

Table 4-17. Shows the times necessary for the test
operators to familiarize themselves with VGRT operation and
successfully complete the test scenario.

I 11 I-12 H E . i] I ! I'

TEES QREIQLQI Time Iminutssl
H1 48
H2 64
H3 47
H4 64
H5 52
Average 55

 

The test operators indicated that they were pleased with

the ease with which they could learn to operate the VGRT.

4.6 CAMERA TESTS

Throughout the testing of the VGRT, one of the
greatest contributors to poor system performance was the
video camera. A poor response at low light levels could be
improved by additional lighting, but resolving the high
light level problem was more difficult. The high light
level problem was most noticeable in actual row crop field
conditions. The problem was manifest in the camera having
a problem of being unable to resolve the difference in
color of green plants and brown soil. To the human eye,

the plants had a nice green color and the soil was very

125

brown. To the camera, the plants were yellow-green and the
soil was yellow-brown.

This problem occurred in both direct sunlight and
moderate cloud cover. One factor contributing to this
problem was that the camera placement permitted incident
solar radiation to strike the camera lens (Figure 4-6).
Another factor contributing to the problem was that, in the
visual spectrum, green plant leaves and brown soil are
actually very close together. Since the camera placement
was fixed by necessity, another solution had to be found to
solve this high light level problem.

Three different types of optical filters were placed
on the camera to try to reduce the excessive light problem.
Reducing excessive ambient light would hopefully increase
the camera’s ability to resolve the colors brown and green.
Another approach was to alter the hue (H), saturation
(S), and intensity (I) constants in the video digitizer
board such that a greater apparent difference in color
would occur between brown and green. Both methods,
separately and in combination were tried to improve camera
performance.

A test situation was developed to quantify the effect
these two methods had on improving camera performance. A
22 cm by 28 cm piece of brown paper was used as the test
"soil" color. On this piece of brown paper 1.3 cm wide
stripes of green paper were glued to approximate the crop

leaf color. This produced a test image that was 50% brown

126

INCIDENT LIGHT STRIKINC CAMERA

 

\

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 4-6. View of camera placement on tractor and
incident high level light problem.

127

and 50% green (Figure 4-7). Stripes were used to enhance
the noise problem associated with the brown-green
interfaces. This test paper was placed so that the
camera's view of it was similar to a field situation with
up-sun viewing (Figure 4-8).

To evaluate the different test conditions, five
histograms were taken of each view. For a given histogram,
the red, green, and blue components were analyzed
separately. The three separate scores were then added

together to provide a composite score for each histogram.

 

The histogram test area consisted of 9900 pixels (220
pixels wide by 45 pixels high) that cut across five of the
stripes (the stripes were oriented vertically) in the
camera’s field of view. Ideally each color band would have
a bimodal distribution in its histogram. The better the
camera could "see" the two color stripes as separate
colors, the greater the distinction between the two peaks
in the histogram. As the camera's ability to distinguish
the two colors as separate colors decreased, the separation
between the two peaks in the histogram would decrease
(Figure 4-9).

The scoring method involved quantifying the
distinction of the two peaks in the histogram. The
separation of the two peaks is viewed as a function of the
depth of the valley between the two peaks. This is

calculated as what percent the valley is of the low peak

128

flwsasflmw MIMWHNMH_
- ..... I. -s- .. ---zmwmbi

 

s:-axommm-;-.wwww-s-ssszsss-

III..- II iII III. III . II IYIIIIII. IIII IIIIIiA.

YI! III-I II II III I .IIIIIIL III III .IIIIIIIIIII

. 280mm,---

Test paper used

 

III I I I ...III.-- I .

I! II 1.1 . 1| III- ..i-IIII.III|IIIII.IIIIIII’IIIIIA

 

-Iszyomm-s.

 

 

 

2mmmo
flee-E.--- M- --H ”...-w..- I
T ..--s-.- - --- - - ..--Ewan...
-..-CE- I - --
.- ...-.... --IE-s--.Is.zww.mb.-I
26-5 w. -..-...- ...... L
I... - ----ELEM...-
-.-..E.Lm - :.-------..-....-
I- is! - - -....sfimEi
I. 230% II ----Ii-----.I-
II III -I. - zmmmo

 

Two color camera test paper.

 

Figure 4-7.
to simulate brown and green field colors for camera tests.

129

 

 

 

 

 

 

 

 

 

 

 

Figure 4-8. Placement of camera test paper to simulate up-
sun field lighting conditions.

130

 

 

 

 

 

X- I
'11
fig - COOD MODE SEPARATION I.
E I
m I
Z I
O I
..<
(3 ’ I
m I /
O l I I
o / -
0 s ' I
c .
:0 , I
ITI ' I
Z . f
O I I
m I 3
8L :4
g 31
INTENSITY
TIXI—T
:0 I I
n1 , !
E . I
g '2 POOR MODE SEPARATION
:3 I I
"< I
o
.- I
o i
O I
O I
C: 1
30
2‘3
fi% // \ i
8 L_ L.‘
O 31

INTENSITY

Figure 4-9. Bimodal distribution used in histograms to
rate camera resolution improvement methods.

 

131

and what percent the valley is of the high peak. These two
percentages, six in total for a given test histogram (red,
green, and blue) were added together for a total score for
a given histogram. A lower score indicated that the camera
was better able to resolve brown and green as two separate

colors.

4.6.1.1 LENS VARIATIONS

The first attempt to improve camera performance in
high light conditions involved obtaining advice from local
knowledgeable camera and video equipment retailers and
repairers. Their advice was almost unanimous in that lens
filters would help reduce the effects of too high a light
level. Three different filter types, singly or in
combination, were recommended; a neutral density filter, an
ultraviolet filter, and a linear polarizing filter.
Filters of these types were purchased and tested for their
ability to improve the high light level color

discrimination of the camera.

4.6.1.2 HUE, SATURATION, AND INTENSITY VARIATIONS
Initially, the video digitizer board had the
recommended (default) values of hue (H), saturation (S),
and intensity (I), for reproducing acceptable color
renditions of digitized video scenes. The second attempt

to improve camera response involved altering these values

132

to try to increase the discrimination between the colors
green and brown.

The original values were H62, 8100, and 157. After
viewing the test image and altering the values a new set of
values was determined. These values of H94, 8100, and I46
appeared to give a slightly better rendition of the green

and brown colors on the test paper.

4.6.2 CAMERA TEST RESULTS

Using the aforementioned test set up (Figure 4-8), the
two camera improvements discussed above were tested. The
camera response with no filters and with the original HSI
values was used as reference. The following combination of
methods was evaluated: original HSI values with no filters,
each filter added individually (polarizer filter turned
horizontally and vertically), and all filters together: the
same sequence was applied to the new HSI values. All
combinations were tried on days with direct sunlight and
days with complete cloud cover. The results are summarized

as follows in Table 4-18:

133

 

Table 4-18 provides a comparison of the test results
for different high light level test conditions of the
camera. A lower number indicates that the camera was
better able to discriminate between the two test colors.
HSI original values (0), new values (N), and sunny (S)
or cloudy (C)].

These values are based on five replications.

- Lev T ts

£11m wmmm

Qzfi Ezé 9:9 322
No filters 585 - 555 295 - 255
U.V. filter 585 - 555 315 - 235
ND4 filter 600 - 600 295 - 280
P01. vert. 575 - 525 445 - 185
Pol. hor. 600 - 600 380 - 405
All filters 585 - 535 350 - 395

For the "all filters" test the polarizer was set to
vertical. (Vertical or horizontal refers to the position
of the index mark on the rotatable polarizer ring).

 

After viewing the ambient conditions, this table can be
consulted to determine the best possible combination of
camera filters and HSI values to get the best camera color

discrimination.

CHAPTER V

SUMMARY, CONCLUSIONS, AND RECOMMENDATIONS

5.0 INTRODUCTION

This chapter is a summary of the use of vision
guidance as a system used to guide a robot tractor. Could
present technology guide the tractor from a row or line,
and did the performance compare favorably to a human
operator’s performance? The robustness of this system was
also an important consideration in the system design. Did
the Vision Guided Robotic Tractor (VGRT) system perform
reliably and consistently through test after test in a
variety of ambient situations? These tests had to be
sufficient in scope and provide a base of knowledge to
indicate the strengths and weaknesses of the VGRT system.
Does the resultant performance of the VGRT warrant testing
this vision guidance system on a larger tractor or combine

in real field situations?

5.1 SUMMARY

As an engineering prototype, the vision guided robotic
tractor (VGRT) was developed and tested on an electric lawn
tractor. The VGRT used a color camera to sense row or line
imagery which was digitized and analyzed by a master

computer. The master computer determined a corrective

134

135

steering heading which was read by a slave computer. The
slave computer provided direct control of the electric
tractor, directed the tractor to turn its steering wheels,
and also could start or stop the vehicle.

Tests were performed on the VGRT to evaluate its
performance. These tests evaluated the basic performance
of the VGRT for comparison with the steering performance of
human operators. One very important aspect of VGRT testing
was the assessment of overall robustness of the VGRT
system.

The goal of this research effort was to evaluate
current technology and its ability to be used to create the
VGRT. This goal was to be realized by building and testing
as opposed to mathematical analysis and modeling.
Limitations were that the hardware would have to be
inexpensive, less than $5,000, and that the technology

should be available in the present retail market.

5.2 CONCLUSIONS

Four objectives were developed as guidelines in
realizing the goal of this research effort. The extent to
which the objectives were achieved is a measure of the
success of the entire project. The results of the project
in terms of those objectives are as follows:

1) The first objective was to procure a color vision
system and install it in conjunction with a controlling

computer system on a small tractor. The completion of the

136

tests in Chapter IV indicate that this vision based
guidance system could provide usable steering control of
the tractor. The installed system was completely self
contained on the tractor with the exception of the video
CRT. This system was self contained to the extent that it
was also the means of developing the software that was used
to operate the system. The length of time that the VGRT
system was used and the number of tests successfully
performed would indicate a successful completion of the
first objective.

2) The second objective was to develop the VGRT to
actually follow well defined lines or rows. The ability of
the VGRT to do this was evaluated by two methods. One
method was to quantify the variability in the lateral
deviations of the VGRT when it followed a controlled line
or row. The second method dealt with comparing the results
of VGRT steering with the results of human operator
steering on the same tractor (tractor temporarily converted
to manual steering) on the same course. The results of the
tests in Chapter IV indicate that the VGRT could
successfully follow a controlled line at speeds up to nine
km/hr or an actual crop row at speeds up to four km/hr.

In comparison to human performance the VGRT's performance
fell within the range of human operator performance.

3) The third objective was to develop the VGRT such

that it could travel from the end of one line or row,

137

make a headland turn, and locate and follow a parallel line
or row. The VGRT was provided with a learn mode in which
the operator would "teach" the VGRT how to make the
headland turn. The VGRT would remember the pattern of the
headland turn and reproduce the pattern when requested to
do so at the end of a line or row. Headland turns brought
the VGRT within 15 cm of the target line in all attempts on
several surfaces. To accommodate closely spaced rows, the
VGRT turn capabilities included up to three arcs of travel.
The results of headland turn tests in Chapter IV indicate
successful completion of this objective.

4) The final objective was to produce a VGRT that was
"user-friendly". People familiar with farm equipment but
not necessarily experts in this technology should be able
to operate the VGRT with appropriate instruction. A test
farm situation was created in which each of five people not
involved with the development of the VGRT mastered its
operation in less than one hour. The results of this test
in Chapter IV indicate successful completion of this
objective.

Overall, the results of the tests in Chapter IV
indicate a successful development of a prototype VGRT. The
success of this project is strong evidence that the

technology is ready for adoption to agriculture.

138

5.3 RECOMMENDATIONS

I recommend that this research effort be continued, to
include testing on a full size tractor or combine in actual
field situations. This research effort has discovered
basic methods of providing a steering control signal from
visual images. Future work could improve on this by
developing a steering system that would have a variable
rate of turning the steering wheels.

To reduce the complexity of the system one computer
could be used. This single computer could be used in an
interrupt mode to monitor the condition of the tractor and
to provide steering correction information directly to the
steering control system. If two computers are used in a
new VGRT system, a method of communications verification
should be used for communications exchanges between the two
computers. A faster computer would be useful to increase
position update rates. An effective increase in processor
speed could also be achieved with the existing PC-AT unit
by going to a non-NTSC video system. The faster computer
system would allow for more robust following schemes that
could extract a control signal from less ideal imagery.

One improvement that should be made with an improved system
would be to float the color references in the color
discrimination row follower. As the plant color undergoes
a slight color change in different parts of the field, the
VGRT could adapt to this change and not become confused.

Future work could also include testing other camera

139

types for improved visual response to field light
conditions. Other camera types may include cameras that
are sensitive in the near or infrared portion of the light
spectrum. Another improvement in color vision would be to
replace the RGB digitizing video system with a system that
utilized the H81 components of the video signal.

With larger sized tractors would come an increased
aspect of safety consciousness on the part of the VGRT.
Improved safety could be in the form of obstacle avoidance
and fail—safe modes of tractor drive and engine controls.
Further improvements in tractor performance for the VGRT
could come from methods to traverse gaps in the crop row,
such as a dual mode of guidance, as well as discrimination
between weeds and crop. Along with improved algorithms in
determining the crop row could come the ability to vary the
forward speed of the VGRT to the speed at which it could

tolerate the complexity of the crop row scene.

APPENDICES

APPENDIX 1

 

IY.!".‘M '. " a.”

fir»..—

140

WERATIN OF THE VISIOI GJIDED TRACTCR

CAUTION - Read thromh all instructions at least once before
atteuptim to operate tractor! Read both pageal

WENEVER TlRNING KEY SNITCH on OEARSHIFT LEVER ALUAYS STAND
TO SIDE OF TRACTG IN CASE OF ACCIDENTAL WENT OF "MUN
* TRACT“ CAN ALMYS BE STWPED DY TIRNING KEY SUITCN OFF *

First walk around the tractor aid failiarize yolrself with
the location of the following controls or itm:
1) Start key - upper ridit side of dash
2) Gearshift lever - in front of seat on floor
3) Woff switch for wall coaputer - toggle switch on
grey box on tractor hood
6) Woff switch for big conwter - toggle switch
located above left side of fan in front door of
green box (door towards center of tractor)
5) Lens cover for canera
6) Big conputer keyboard and floppy drive - inside
back door of green box - to rear of tractor
7) Locate sanll blue box with ribbon cable - inside
front of green box and place face up on green box
8) Catputer reset for big collputer, just to left of
floppy drive
To start the tractor:
1) Remove lens cap from canera hood
2) Turn key switch off
3) Put gearshift in neutral
I.) Turn on small couputer (grey box)
5) Turn on big coaputer (green box)
6) Push big caIputer reset button
7) Wait SOsec and a '1234' should amear on display in
blue box mtarily followed by a '5678'
followed by an '8888' (other rubers will flicker
pest, ignore them)
8) IF ALL DISPLAY NlllBERS HERE CORRECT STAND T0 SIDE
OF TRACTal and put tractor in first gear and turn
on key switch, tractor should remain stationary
To operate the tractor:
The space bar will stop the tractor - '8888' on display
'Y' will start tractor in drive aromd mode - '5555'-'3333'
'D' will exit drive aromd node - should see '8888'
when in learn mode, or drive around node first push an up
or down arrow to start tractor in proper direction then use
left or right arrows to steer tractor, the tractor will not
steer mtil it is moving
'6' will start the tractor mder auto guidance
'0' will start tractor in learn node, left turn 'MO' '4641'
'P' will start tractor in learn mode, right turn '4640' '4641'
'0 to initiate auto left turn - (repeat the saved turn)
'>' to initiate auto rigit turn - (repeat the saved turn)
* Always look for an '8888' between nodes of operation *

141

TEST DRIVING THE TRACTOR

Due to the size of the tractor and nature of the cuputer
installation, you will 'drive' the tractor from the rear of the
tractor using the big comuter's keyboard.

The object of the test is to drive the tractor fro- the lab outside
to the test 'field' area. You should coaplete at least 5 reads
uider auto guiance md teach the tractor both a left hand and
ridit hard haadlmd turn. These turm should allow the tractor to
go down one row and return on the other row.

First, take the tractor outside and practice 'driving' it aromd a
little bit to get the feel of the steering. The steering is
somewhat sluggish so be careful. Push the left or right arrows
once. Then let the meels turn, then push the arrow again if
necessary mtil desired turn is coupleted.

To enter drive aromd mode push 'Y' and a '5555' (then '3333')
should smear on the display. Use the q) arrow to go forward, the
down arrow to back (4:. Use the left or right arrows to steer the
tractor. Hit the space bar to stop tractor to change direction.
Enter a 'B' to stop driving arourld.

The display should always show '8888' before starting a new mode of
operation.

To enter auto-guidance mode, position the tractor so the tractor's
front ridit tire is just to the left of the row (2-3 inches is
fine), you should be able to see the row close to the center of the
camera viewer. Enter a '6' to start tractor moving. Push the
space bar to stop. then the tractor is near the end of the row (3
to lo feet). hit the space bar to stop and either teach it a turn
or, if it already knows the turn, let it do the turn. when it is
in position on the next row, just enter a 'G' to start up again.

You my position the tractor to do turns to the right. Do some
rolnds, than position it so it will need to do turns to the left.

To teach the tractor headland turns men it is stopped at the end
of a row, enter a '0' or 'P' and use the arrow keys as before in
drive around mode to operate the tractor. Stop the tractor when it
is next to the row (row is just to right of front right tire). If
you wish to redo a turn, stop the tractor and just put it back into
the learn node, '0' or 'P'. The old turn will be erased and the
new turn entered (you will of course need to redo the turn).

To have the tractor automtically do the turn, stop the tractor at
the end of the row and just enter a '<' or '>', (that's using the
less or greater than arrows, use the one that points in the same
direction as the corresponding turn).

 

A
IIIP

:. “o “3.97
Ii ...)... -11.,5fluh»

APPENDIX 2

142

 

 

 

 

 

 

 

 

 

 

>Jmsam «uses he
sees»
mxmu o¢¢2oua
[I0 004 Nd“ W
accmumum ...I N n+1
woouo mom
eccezq mo .
use «so accmumuz ago
- «04 no ecu x~.~ ow¢ “Mwwwwwm
Aw. mwzmw mooHo and. «no
unooom «0 ice naooou no ado
o.n« on m :mu :10 v we wooHo
O) H) o . ‘s—HE
«ohmnmmm " . «of as 1 «so to
v: new. umau zepmumum
1mm rzo v at
Eyelash . ....
mmonzN so
:10 owe me AP
In... mocmnmwm LI III .w.
moo—o xm.~ Na
mopmwmwm W vOOoza No mmwmao use. «no mormnmmu
sou scu acemnmum
and. «u .
hr mw2m~ moons c xm N am
unooow no ecu u:ooo« qu ado
«.m no m mmuu 10w :10 a we wooHo
o: S . Assn
“Ohmumwz ..n . dm.# 4... ’ aCO 40
:10 ans he a: acumumuz
so» :10 a to
RENT.“ .sz
canola No
.invlflnifiltun'flvhlfiii. -uruhl.u.

 

 

1J13

 

 

 

fi.’

 

    

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

         

 

 

 

    

 

 

  

flOhmHmUC

 

J)\(
¥0A $41

 

 

 

 

 

 

flOhMHWMC
OOfi 0d“

 

 

.7:
”mm

0021

+ comm 03-7

 

 

c m
L :gFflch-UO .NM
>Jasam muses we
ages .
stun occzaua AMP
. 11:411..
”v c - - e
Z . . . s
m m + .
.... ... . 3... m
J .3
cwg zaoow w... w w m m m
known mww [all uoflflwuoozul osmonmn m m u m m i +
m a
voovza on x» «no
acumumuo
nHuHMUBMHHVIIIII one one own an. - cormumum
cobmnmu- me An. xua one
an on» «no .r
19... IIHII
oomnzu no
a:
«05me o e .
trams. «as
NHHwoos m hm m
n... ....n m
usoooa wee w .3
zoom «4 m
cohooozn Demons: m m m
...w... s
HHMMEHMNHHMUV I
voovai

  

 

   

IIOd.O 0d¢

 

 

ObmHm a
CC C N
Movmumwmdfl
Dd -
l h”
XOd VAC

kl OHM 040 .00

KOPWHMNK

~q+.bmbil.

 

 

 

 

APPENDIX 3

144

 

 

 

 

 

 

 

         

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

  

 

 

 

 

 

   

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

>U. 80132 1‘4
o\u disses he .suxu oxn ze zo muco: 432200
.2: . shaman mmzun. mmuxooc oz» xuxoa us»
...... ......
in; 0» 103” E 3260.. macs o\n umw—t 20:02:... mowouzzou
Emma AUHNu I...
\ BN5
Os new
. 2 0c
mNmQ IOhquZOO uQCkw—Mb H “0» 1h mmNm MVNmJVh
can on «a «.... Q
I use do «a me n as
a.» «as no no me "
Q no... no to es m
a... ... «cm to on me u
. . ”a. mm mm ...n .
Ink nu NH .
n. ...as so .3 an we . Q Ne
"... one 0
..~ dmk m3 KHA-
I“. Nmt _ Q
Is mm“ cumwn m _ a ..nc
a 3.. B mm _ Q3.
Du». OWL d. éflm
33.x; ems J
.... ...wu ...» m
NOE m» U ONC
no...
1m cu... l. e> sue
mus m> 3 Too:
was or «no «3
sum . At. one
o:
coo II N \
seen» no umau cc unwmzn o ewe
. mmzum ...th
woe mnntu «0
mac
E
copmummm
V: vs +u> wmm cue
>w+ 00 «no a:
Iw> nmo I 3:
«oemumuc vao I g nae
$30 moo
h x9. nu me one ea." «a mo u> Q «as
l poo capmnmmm
H : “.6 .
known «0: ago o N
am”; Wm Hui and an
_ u mopmumuc
FT tau «0 moo
_m clue genomic... xu.~ we a. mu «m. to .54 «o I
3 Too own .3
m? 00

 

 

14S

 

 

 

 

 

 

 

GIUK OKCZKUQ

 

 

IddNFIUH

l
O

”M DNNWNNNN 'hhfiflddd
Idi£flsflaflfll¢ {NLUCMHIC

 

.WJQCQ ZOGQHC >0 0 bUXUOm 0h Ouhowzzoo d buxoom

PMCM 3m

 

 

 

 

 

 

 

LC).

 

 

 

 

XOd .v

IObMHmum
204 d!

 

 

 

 

 

 

 

HI :9
O

 

V""
C

 

 

1AAAAA

 

 

 

 

 

0 >

X00 OJUIIOZ¢I OKCOOPDO OCCOO O\H #4

 

 

 

APPENDIX 4

 

 

 

 

>Jmuam “NICE “thltOO UDCJM

¢IUK O¢C2¢UO

 

146

COPMHMUK
£10 DOA N1

 

r—ih'

 

 

h

1'

DOONN MU
had.

“OFMHWUK
1J\/\/V
:16 0d d“

OUJ kDOONN d0

owm MO

ado ECU

NU

EGO 1‘0

T—«L—I»

 

(...

‘
l

 

J

 

 

 

 

.hZGP EDNN ho

P-ié—ih-

”COMO

tug

NOOCZA do

 

 

mad. #0

 

 

 

 

304

I

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

1Jl7

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

#0.! L "Gain
a « ah c
LOW-.32. MCEJUD IN.—
o o» c uJozc Juuzx
asuw
arms ocezzum
pqu oh zoos >m.« x
prone oh zoos >w.m v
would Juuzx
ozu c .mr ozu «Aw. ozo c .mr
boa
1¢0 EGO 0130... ”CHI v.4 hm.
E=.«o §=.«uU“
_ cobmnmmo
III Tat/\(
:10 oo« oo usmco ouosuuxm
- ozo c >m+ 00>
."Iafiu"
ole o -
new I!
\0 WE. unload MOH
nuflummmHHMHmHMHV - oao twaidnll - H
"w ....
mohmnmmo
moo In:
E coo a xoq vs
- HMO. mum 5.30 a."
t «a .1
I. x4: c
uoOwo m sue) m are q
ecu eooezw «o
..SN.~. to H\ uzwpucum udum ecumnmue
xm.u «c
I >m.~
«ephemuc
:04 no cobmnmmc
xm.a «c
>m+ 00>
>m+ 00>

 

 

148

 

LUJISZ

 

 

CMthIOU U>¢Jm uth

 

CINE OGGZCWO

 

 

 

KO»MHWN&

             

  

 

(r—‘AN—o—“I

 

 

 

 

 

 

 

           

 

   

 

 

 

 

 

 

 

 

 

 

 

 

ZIO DOA hm tepmumum ado
x0." #1 H“
has. v0
211 «Ohmumwz
633 3 g
1a ”1
11a zepmumw¢
WOOMZN No
Xe mm
«men
1 «1mm can“ Jaemtiu
“ klNN NU uaNN «O ¢OhMHmm¢
Wm“ «x, w. H . _E xoq «a when :m
mum“ Nx NIIOOO v
won. ...3 no humus one on
humus >m+ oo>
OhZH
ubZH P
Dr II
up
«an g 5% ..m
N.dl IIHIIIAHHHMMNE EOhm )UZUUKNIN mm
M.«t
on I _IIANEE
..s... m _
A.
I.e¥i..l.§..i.as? 3 E «2%me

>m+ 00>

 

 

 

 

i ‘l I: l .llill'Il-II' It";

 

 

manta—O IO hMHNZCCh

 

 

 

 

 

      

 

 

 

 

 

 

 

 

 

 

ages
arm... ozczcum
21... is
onmmxm mo .13. mo
genumuo ado
.. :10 own 31 mobmnmuo
hMlm >¢JU¢ VOOVZd NO mOhmHmUK alt /_— é
.3 £3 ~20 «we movvzw so I
xo... as
HUJ ECU
o >
m _ _ «caesium .30 90
2.62.. so Ema W
o >
uzumuupm
9 Edamlaoa \ ummu>u¢Io¢¢3¢ou \ mouusm op 5:050 u>~oo resuo
4 U
1
“VI enoomnu muuuom occmnmzomp
h h. god 3.
Ilmilu II CObWHMU“
sou
has. «u
0 one... zo~ozHJ¢co xn.» no
will newtzmao.
.. g... “33”
row «a
.30 . sou I—M>e~+ «3.5me
scummmmsaom x05. wowpuaozn oza_ cued... wwwoz v... me . cepmumuc
mmomzu no 5 he oopmumue cormwmuo 2mm «a
......a.
oznpunrm Ju>w4 >. ...3. e0

>GNL

 

 

 

LIST OF REFERENCES

LIST OF REFERENCES

Asimov, Isaac. 1950. I, Robot. Doubleday and Company,
Inc. Garden City, New York.

Ballard, D. H., C. M. Brown. 1982. Computer Vision.
Prentice-Hall, Inc.

Becker, W. J., W. D. Shoup, and J. V. Sinden. 1983.
Field measurement and analysis of operator speed-load
stress. ASAE Paper 83-1628.

Brandon, J. R., S. W. Searcy, and R. J. Babowicz. 1989.
Distributed Control For Vision Based Tractor Guidance.
ASAE Paper 39-7517.

Dickmanns, E. D., A. Zapp. 1987. Autonomous High Speed
Road Vehicle Guidance by Computer Vision.
International Federation of Automatic Control (IFAC)
World Congress. July 1987.

Doss, Howard J. 1988. A Look At Agriculture’s Safety
Record. Safety News, Cooperative Extension Bulletin.
Michigan State University, Cooperative Extension
Service.

Dunlay, R. T., D. G. Morgenthaler. 1986. Robot road-
following using laser-based range imagery. Tans. SME
Second World Conference on Robotics Research,
Scottsdale, AZ, August 1986.

Dunlay, R. T., D. G. Morgenthaler. 1986. Obstacle
detection and avoidance from range data. Proc. SPIE
Mobile Robots Conference, Cambridge, MA, October 1986.

Fehr, Bernard W., An Electronic Guidance System For Row
Crop Planters: A Feasibility Study. Masters Thesis
1986. Southern Illinois University.

Fehr, Bernard W. and J. B. Gerrish. 1989. Vision-Guided
Off-Road Vehicle. ASAE Paper 89-7516.

150

151

Gerrish, J. B. and T. C. Surbrook. 1983. Application of
mobile robotics in agriculture. Proceedings of the
First International Conference on Robotics and
Intelligent Machines in Agriculture. ASAE Publication
4-84.

Gerrish, J. B., G. C. Stockman, L. Mann, and G. Hu. 1986.
Path-finding by image processing in agricultural field
operations. SAE Trans. 95(5):540-554.

Goody, R. W. 1982. The Intelligent Microcomputer.
Science Research Associates, Inc.

Gross, T. A. O. 1978. Controlling With Ultrasonics.
Machine Design, March 9, 1978.

Kaminaka, M. 8., G. E. Rehkugler, and W. W. Gunkel. 1981.
Visual monitoring in a simulated agricultural
machinery operation. Human Factors, 23(2) : 165-173.

Kantowitz, B. H., R. D. Sorkin. 1983. Human Factors,

Understanding People-System Relationships. John Wiley
and Sons, Inc.

Larson, W. B., D. A. Tyler, and G. A. Nielsen. 1988.
Field Navigation Using The Global Positioning System
(GPS). ASAE Paper No. 88-1604.

Levine, M. D. 1985. Vision In Man And Machine. McGraw-
Hill, Inc.

Lillesand, T. M., R. W. Kiefer. 1979. Remote Sensing and
Image Interpretation. John Wiley and Sons, Inc.

McMahon, C. B. 1982. Development Of A Microprocessor-
Based Steering Control System For An Apple Harvester
Utilizing An Ultrasonic Sensing System. A Ph.D.
Dissertation. Agricultural Engineering Department,
Michigan State University.

Murphy, Brian R., An Automatic Guidance Controller Using
Networked Microcomputers. Masters Thesis 1988.
Michigan State University.

Palmer, R. J. and S. K. Matheson. 1988. Impact of
Navigation on Farming. ASAE Paper 88 - 1602.

Patterson, R. J., B. W. Fehr, and L. P. Sheets. 1985.
Electronic Guidance System For A Planter. ASAE Paper
85-1587.

152

Polaroid Corp. 1982. Ultrasonic Ranging System.
Cambridge, Mass. Ultrasonic Ranging Marketing
Division.

Reid, John F., The Development of Computer Vision
Algorithms for Agriculture Vehicle Guidance. PH.D.
Dissertation 1987. Texas A and M University.

Reid, J. F. and S. W. Searcy. 1988. An Algorithm for
Separating Guidance Information from Row Crop Images.
Transactions of the ASAE Number 0001-2351/88/3106-
1624.

Reid, J. F. and S. W. Searcy. 1987. Vision-based guidance
on an agricultural tractor. IEEE Paper No. 0272-
1708/87/0400-0039.

Searcy, S. W., J. F. Reid. 1989. Machines See Red... and
So Much More. Agricultural Engineering
November/December 1989.

Smith, G. L. 1987. Farmer's Perception of Human Factors.
National Institute for Farm Safety, Inc. Paper 87 - 5.

Surbrook, T. C., J. B. Gerrish, R. E. Squires, and M. A.
Shanblatt. 1982. An automatic steering controller
for a robotic lawn tractor. ASAE Paper No. 82-3039.

Touretzky, D. S., D. A. Pomerleau. 1989. What’s Hidden in
the Hidden Layers? Byte Magazine. August 1989.

TRI-R Innovations. 1988. (Advertisement.) Gibson City,
Illinois.

Turk, M. A., D. G. Morgenthaler, K. D. Gremban, M. Marra.
1987. Video Road—Following For The Autonomous Land
Vehicle. IEEE Paper NO. CHZ413-3/87/OOOO/0273.

Wallace, R. S. 1987. Robot Road Following by Adaptive
Color Classification and Shape Tracking.
IEEE Paper NO. CH2413-3/87/OOOO/0258.
(Copyright 1987 IEEE). International Conference on
Robotics and Automation, Volume 1, 1987.
(Used by permission).

Walter, S. A. 1987. The Sonar Ring: Obstacle Detection
for a Mobile Robot. IEEE Paper No. CH2413-
3/87/0000/1574.

Williams, T. 1988. HSI Conversion Brings True Color
Processing to Life. Computer Design. January 15,
1988.

153

Whittaker, W. L., G. Turkiyyah, M. Hebert. 1987. An
Architecture and Two Cases in Range-Based Modeling and
Planning. IEEE Paper No. CH2413-3/87/0000/1991.
(Copyright 1987 IEEE). International Conference on
Robotics and Automation, Volume 3, 1987.

(Used by permission).

Wolpert, H. D. 1987. Autoranging/Autofocus: A Survery of
Systems, Part 1, Part 2, Part 3. Photonics Spectra
Journal. June, August, and September (respectively).

Young, R. E. 1976. Automatic Guidance Of Farm Vehicles:
A Monograph. Agricultural Engineering Department
Series No. 1. Agricultural Experiment Station, Auburn
University.

"IIIIIIIIIIIIIIIII