$9,.

. Mum:—

r;

..u

fin“; .
u.» .«7.
J

M.
a...

.. ;,
......
3%.

.09

3h.

!
i

4.. .43.:
5.3%.
..., ..."uur ...

£

v, 3 o
'1. ..
:5...

‘I .

.| t v

. ‘ ...... V .. .
3“ ibufim

. hi... 2} a
(.....xlflr . I..- . .

It

v‘v‘

......Z -T , ...

J ..-.-. ..x ...-..i.
.‘x O 3a.

. a...

Awmwwflmhfi.»

 

i

' 2
. L' ,1
$43?

' ti:

1;."
$7? : -.
«:{5‘ 9‘ ‘
«1
1‘.“ ~

I

§
9"

O
,

.‘J
p ‘
411‘
’l
{.513
' r 4“?)
. 4"
m
‘ V’L‘rhagfl
‘1 (ry‘gfi‘; '

V.
In
.50

lllllllllllllllll“Hill“llllllllllllllllllllH||lllllllll||l
3 12 820

93 01834 1

LanAnv 1

Michigan State
University J

 

 

 

This is to certify that the

thesis entitled

REAL-TIME ROBOT CONTROL OVER THE
INTERNET WITH FORCE REFLECTION

presented by
Imad Elhajj

has been accepted towards fulfillment
of the requirements for

, MamLL degree in _£len.tnical Eng

dzx'

Major professor

Date 71/ l JT/ 9?

0-7639 MS U is an Affirmative Action/Equal Opportunity Institution

PLACE IN RETURN BOX to remove this checkout from your record.
TO AVOID FINES return on or before date due.
MAY BE RECALLED with earlier due date if requested.

 

DATE DUE DATE DUE DATE DUE

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

ma comma.“

REAL-TIME ROBOT CONTROL OVER THE INTERNET WITH FORCE
REFLECTION

BY

IMAD HANNA ELHAJ J

A THESIS

Submitted to

Michigan State University

in partial fulfillment of the requirements

for the degree of

MASTER OF SCIENCE

DEPARTMENT OF ELECTRICAL AND COMPUTER ENGINEERING

1999

ABSTRACT

Real-Time Robot Control over the Internet with Force Reflection
By

Imad Hanna Elhajj

The use of the Internet is not limited anymore to the transmission of data. In the past
few years many successful attempts have been made to use the Internet as a command
transmission media; where control can be sent to remote systems and feedback can
be obtained via the Internet. But with this media comes several limitations; delay,
lost packets and disconnection. All of these limitations may cause instability in the
system especially if the system loop is closed via the Internet.

All the previous work done that addresses these problems assumed several con-
ditions; for example, time delay is constant or has an upper bound, control is not
in real-time. A new real-time control approach is presented that deals with these
limitations and difficulties without any assumptions made regarding the time delay.
The approach is based on Event Based Control, which was implemented on a mobile
robot over the Internet. The commands sent to the robot are velocity and the feed-
back is force based on the environment. It will be shown that this approach results in
a stable system. In addition, a new force feedback generation method is used, which
would give the Operator a specific perception about the physical environment that

the robot is in.

To my great and loving Family ......

iii

ACKNOWLEDGEMENTS

I would like to acknowledge all the help from Dr. Ning Xi. Without him, this would
not have been possible. And I would like to thank Mr. Keith Jones for all the help

with the document formatting.

iv

1

4

TABLE OF CONTENTS

INTRODUCTION 1
1.1 Applications and Motivations ...................... 1
1.2 Limitations and Difficulties ........................ 3
1.3 Overview of Existing Methods and Approaches ............. 4
1.3.1 Robotics Approach ........................ 4
1.3.2 Communication Approach .................... 5
1.4 Limitations of Existing Solutions .................... 6
1.5 Previous Internet Teleoperation and Their Limitations ........ 6
1.6 Outline ................................... 8
THE INTERNET: REAL-TIME CONTROL MEDIA 9
2.1 Characteristics of Internet Communication ............... 9
2.2 Effects of Time Delay on Real-Time Control .............. 13
NON-TIME REFERENCED PLANNING AND CONTROL FOR INTERNET-
BASED REAL-TIME TELE—OPERATION 14
3.1 Non-Time Based Real-Time Control ................... 14
3.2 Modeling of The System ......................... 15
3.3 Comparison with Other Models and Approaches ............ 21
3.4 Stability Analysis ............................. 23
IMPLEMENTATION AND EXPERIMENTATION 27
4.1 System Implementation .......................... 27
4.1.1 Hardware ............................. 27
4.1.2 Software .............................. 28
4.1.3 Obstacle Avoidance and Virtual Force Generation ....... 32
4.2 Experimental Results ........................... 36

4.2.1 Tele-Operation Over the Same LAN ............... 36

4.2.2 Tele-operation from Hong Kong ................. 46

4.2.3 Experimental Conclusions .................... 47

5 CONCLUSION AND FUTURE WORK 55
5.1 Conclusion ................................. 55
5.2 thure Work ................................ 56
BIBLIOGRAPHY 59

vi

LIST OF TABLES

3.1 The Explanation of the various variables in the Tele-Operation systems. 17

4.1 Specifications of the system hardware. ................. 29

vii

2.1

2.2

2.3

2.4

2.5

3.1

3.2
3.3
3.4

3.5

4.1
4.2
4.3
4.4
4.5

4.6

LIST OF FIGURES

Inter-arrival times of groups of packets on the August 1989 Bellcore
ethernet trace [37] [38]. .......................... 10
Round trip delay in ms for packets transmitted between the robot and
MSU station. ............................... 11
Round trip delay in ms for packets transmitted between the robot and
Hong Kong station ............................. 11
Round trip delay in ms for packets transmitted between the robot and
Hong Kong station ............................. 12
The frequency of control and feedback signals between the robot and
Hong Kong station ............................. 12

The comparison between traditional time-based and event-based plan-

ning and control. ............................. 15
Block diagram of a traditional tele-operation system. ......... 17
Detailed block diagram of our tele—operation system. ......... 17

Network representation of teleOperator, the different terms are as ex-

plained in table3.1 ............................. 22
Transformer ................................. 22
Hardware structure of system ....................... 28
Motion server, Client, and Camera server general flow chart. ..... 30
Flow chart for deciding velocity based on sensors. ........... 30
The XR4000. ............................... 32

A t0p view of the robot that gives the distribution and numbering of
the different sensors. ........................... 33
The fraction of velocity that is deducted based on the distance d in

meters to the closest obstacle. ...................... 35

viii

4.7 The two safety regions around the robot according to the direction Of
motion. .................................. 35
4.8 The behavior Of the system under normal Operation with relatively far
obstacles. ................................. 37
4.9 The behavior Of the system under normal Operation ........... 40
4.10 Comparison between the desired velocities and the sum Of the actual
velocities and the force felt. ....................... 41
4.11 Comparison between tracking error and force felt for normal Operation. 41
4.12 The behavior Of the system under 2 seconds Of round trip delay. . . . 42
4.13 Comparison between the desired velocities and the sum Of the actual
velocities and the force felt. ....................... 43
4.14 Comparison between tracking error and force felt for 2 seconds delay
Operation. ................................. 43
4.15 The behavior Of the system under random round trip delay, ranging
from 1 tO 4 seconds. ........................... 44
4.16 Comparison between the desired velocities and the sum Of the actual
velocities and the force felt. ....................... 45

4.17 Comparison between tracking error and force felt for random delay

Operation. ................................. 45
4.18 The behavior of the system during the control from Hong Kong. . . . 48
4.19 The behavior Of the system during the control from Hong Kong. . . . 49

4.20 Comparison between the desired velocities and the sum Of the actual
velocities and the force felt. ....................... 50

4.21 Comparison between tracking error and force felt for the two Hong
Kong experiments. ............................ 51

4.22 The behavior Of the system during Operation from Hong Kong. . . . . 52

ix

4.23 Comparison between the desired velocities and the sum Of the actual
velocities and the force felt. ....................... 53
4.24 Comparison between tracking error and force felt for teleOperation from

Hong Kong. ................................ 53

CHAPTER 1
INTRODUCTION

In the past decade the Internet have experienced orders Of magnitude growth. This
growth was not only characterized by the number Of networks and terminals connected
to it, but also by the several application that it was used for. It is not only a data
transmission media, but also a place tO buy, sell, learn, explore and “control”.

Control over the Internet can be characterized by several prOperties and takes
different forms. Regardless Of the system used all real-time controlled systems over
the Internet share many similar difficulties and advantages. A major characteristic
and limitation of tele-Operation is delay. Data Packets transmitted between 2 points
will be delayed. This delay, although not desired, does not constitute a major problem
for data being passed. But when we start considering the network (Internet) as an
action super-highway, via which we send control commands, time delay becomes an
important factor in the stability and efficiency Of the system.

We present here a new approach for tele—Operation with force feedback that over-
comes the problems associated with delay and several other tele-Operation problems.

In what follows We will give the motivation behind using the Internet for tele-
Operation, then we will discuss the difficulties and problems associated with this
media. After that we will cover some Of the existing methods that deal with these

problems and point out their limitations.

1.1 Applications and Motivations

The applicability of a certain technology is a main factor in its advance. When
it comes tO tele-Operation, several applications motivated the advance Of this field.

Among the major ones we have:

1. Tele-medicine: The attractive thing about this application is the ability to treat

patients by experts that are not in the same city or even the same country. The
ultimate Objective is to have systems with very high reliability to be able to
trust them with human life. Several experiments and improvements have been

done with these systems [1][2].

2. Tele-manufacturing: The ability to use SOphisticated and expensive manufac-

turing facilities by several users around the world is very cost effective [5].

3. Exploration (sea and space): This application got very popular during the
NASA Mars Pathfinder Mission [3]. There is no need to mention the vari-
ous advantages for this application and the importance Of tele—Operation for
space and deep sea exploration [6] [8], especially in the safety and cost reduction

aspects.

4. Hazard material manipulation: One Of the first noticed application Of tele-
operation is handling hazard material; such as, radioactive or explosive Ob-

jects [4].

5. Tele—Operated games: Several tele-Operated games can be found on the Internet;
for example, in Germany there is an interactive railroad that can be Operated

remotely.

It was shown in [20]-[21]that the Operators performance can be improved in these
systems by having force feedback. Force feedback is simply sending back from the
slave side force or velocity information that can be felt by the Operator, in which
case the Operator will have a better understanding of the environment. When this is
done, the Operator is said to be kinethetically coupled tO the environment, and the
teleoperator is said to be controlled bilaterally.

All Of these tele-Operation applications, including force feedback, can be done

using any kind Of communication media. But what interests us is the use Of the

Internet as the link between the Operator and the robot. The use Of the Internet is
motivated by many reasons; in comparison tO a dedicated link, the Internet is much
cheaper, more publicly available and does not require any Special hardware. However,
with all these advantages comes several disadvantages which will be discussed in the

next section.

1.2 Limitations and Difficulties

As any communication link, the Internet introduces delay in the system. This delay
can be due tO several reasons; prOpagation, congestion, or being low on resources. For
a communication link between the ground station and a space telerObot in low earth
orbit delay is expected to be as long as 2-8 sec [7]-[11]. Same amount Of delay can
be expected for deep sea Operations. As for the Internet, the delay is unpredictable,
in the sense that no upper bound can be Specified and it could have a very complex
stochastic model. Several attempts have been made to model delay over the Internet,
and several complex models have been derived [37]. In [37] a wavelet-based model1
was derived. This delay was shown to be self-similar [38], in a statistical sense.

In addition, packets sent over the Internet can be lost and can arrive out Of order,
that is why we used T CP / IP protocol for communication, which insures the arrival
Of all packets in order. More severely, Internet connections can be lost, that is why
several design precautions had to be taken. These design issues will be discussed in
more details in subsequent chapters.

All these can render the system unstable. Especially time delay; where it was
shown in [7]-[17] [20]-[24] that delays on the order Of a tenth Of a second can desta-
bilize the teleoperator. And when force feedback is being used the system is sta—
bilized only when the bandwidth was severely reduced. In addition to instability,

de-synchronization can occur between the master and slave.

 

1Wavelet analysis is most used for statistically self-similar processes.

Several approaches have been taken to solve the above mentioned problems, and

those will be the tOpic Of sections 1.3 and 1.4.

1.3 Overview of Existing Methods and Approaches

What follows is a summary Of the different methods and techniques used to overcome
the problems faced in tele-Operation in general. The different approaches will be
presented here and then section 1.4 will cover their weaknesses. The approaches are
divided into 2 groups. One Of them concentrates on the robotics side by introducing
techniques and force/ image feedback with prediction. The other concentrates on the
communication side by using new communication protocols or modifying the existing

ones. What was noted is no approach combined both sides in the solution.

1.3.1 Robotics Approach

Most Of the approaches in this category relate to control schemes and force/image

feedback with prediction. Some Of them are even a combination Of several approaches.

Control

The problem is handled by changes in the controller modeling/design and control
algorithms. Several design techniques are used, some Of which apply tO the Specific
system used and others that are more general.

In [18][19] an Observer for linear feedback control with delays is designed. [7] [8] use
shared compliant control; where the control task is Shared by both the human Operator
and the autonomous compliant control Of the remote robot system. As for [9], time
and position de-synchronization was used and in [13] Virtual Internal Model (VIM)
was the approach. [15] used design method based on the Hog-Optimal control and
wsynthesis framework. The work Of Anderson and Spong in [20] and [21] was a good

starting point for our research, they introduced a controller that was derived from
modeling the network as a loss-less transmission line. Another method was based on

stochastic control theory and a separation prOperty [17].

Sensory Feedback

This includes force/ torque feedback and image feedback. Either Of these fed back
elements can be either real or virtual. So some Of these techniques actually use
prediction.

The most common feedback in tele-Operation is video, where cameras capture the
task Space and then these images are sent back to the Operator [7] [8][10][11] [12]. The
other sensory feedback is usually force or torque, where the feeling Of the robot is sent
back tO the Operator [20]-[24]. Although these two techniques usually use real values,
sometimes some kind Of prediction is used in parallel. For example, the position
of the manipulator can be predicted using a simulation program and it can then be
displayed with the real image SO that the Operator can compare [11][12] [16]. Predicted
force can also be used where the simulated value can be sent to the master or just

displayed on the monitor [16].

1 . 3. 2 Communication Approach

The approaches in this category try mainly to change the underlying communication
protocol or at least modify it. That is why these methods are difficult to implement
and test, since in order to do that we have to modify the routing algorithm and
communication protocol used by all the different components in our communication
system. The thing to note is that non tried to develop a method that does not
need major system modification; for example, an application level solution, where
the routine that is handling the delay problem is running at the application level.

One Of the proposals was to design a rate-based flow control, where the rate at

which the user is allowed to transmit is determined by the switches on the VC2
between the source and destination [34] [35]. Another proposal was used dynamic

congestion control and error recovery algorithm over a heterogeneous Internet [36].

1.4 Limitations Of Existing Solutions

Most Of the methods used, share some disadvantages that will be discussed here.
By far the most common limiting assumption, is taking delay as having an upper
bound. By upper bound we mean taking a value for the delay beyond which the
system would either become unstable or has to stOp Operating. For example, they
might assume delay in their analysis and implementation not to be more than 3
sec, and if it happens that the delay actually exceeds 3 sec the system either becomes
unstable or has to stOp Operating. Another limitation is not analyzing or doing testing
with random time delays. Both Of these assumptions are quit important when the
Internet in involved, since no upper bound can be Obtained and the delays are proven
to be “random” [37] [38]. Moreover; some Of the prOposed solutions require a lot of
computing power which might in itself be a delaying factor. In addition; in some cases
the remote system is very complex and in others the approach causes more network
traffic, as in video feedback. As for the predictive methods, they require a lot Of
computation and they dO not work well with uncertainties in the environment. As
for the communication proposed fixes, they are difficult to implement and test Since

they require major changes in the existing systems.

1.5 Previous Internet Tele—Operation and Their Limitations

In this section we introduce the previous work done Specifically in Internet tele-

Operation, and then we discuss their limitations. Several Internet based robots have

 

“Virtual Circuit

been deveIOped and studied, we present here some examples that cover most Of the
categories. Before we go on, we would like to clarify a misconception when Internet
robots are considered. Many researchers consider Internet robots to be the ones
which are web based, meaning the ones that are controlled via a web page. Many
discussions have taken place lately", and the general trend was to consider Internet
robots as being the ones controlled over the Internet in any fashion and not only via
a web page.

Tarn and Brady [39] implemented a semi-autonomous telerobot, where the tele-
operator supplies a trajectory for the robot tO execute. Then the Operator only
intervenes in case Of unexpected circumstances. Pai [40] made an expensive exper-
imental facility, the ACME, available via the Internet, where a user can conduct
customized measurements and built accurate models from anywhere. Stein [41] had
introduced a Puma manipulator that can be controlled to paint over the Internet.
Simmons [42] has an autonomous mobile robot (Xavier) that can be commanded to
go to different locations and dO different tasks. Other web robots have been intro-
duced, and currently the NASA Space Telerobotics Program web Site" lists more than
20 of them. These robots perform different tasks, ranging from manipulating Objects
and navigation tO manipulating camera view.

All these systems have several limitations and differ from ours in several aspects.
First and most important is that these systems dO not have real-time feedback. Mean-
ing that the only feedback the Operator has is visual, where either video or sensory
information is sent back tO be displayed. This is why these systems can be considered
to be Open lOOp control systems. Where as in our system, the lOOp is closed once we
include force feedback, and it is considered now a closed lOOp control system. More-
over, the above examples are either autonomous or tele—programmed, which means

the Operator gives a high level command for the robot and then the robot executes

 

3IEEE International Conference on Robotics and Automation, Internet Robotics Workshop.
‘http: / / ranier.oact.hq.nasa. gov / teleroboticspage / realrobots.html

at a different time. Then the Operator can either wait or check later on and see the
status Of the Operation. As for our case, we have real-time control and feedback,
where the commands are executed once received and feedback sent directly. In none
Of the cases that we examined did we find a kinesthetically coupled Operator with the
environment, which would increase the sense Of telepresence. Another limitation is
assumptions concerning time delay, no work was found dealing with real-time control

with force feedback in the presence Of random time delay.

1.6 Outline

We outline here the remaining parts Of the thesis:

1. Chapter 2 examines the Internet characteristics when viewed as a real-time
control link. We study the Internet behavior and what it causes in a feedback
system. We look at the “randomness” Of time delay and the instability that it

causes.

2. Chapter 3 gives an introduction to the event-based control, which uses an event
as a reference and not time. Then the modeling and stability analysis Of our
system is presented, where the general structure Of the system and the dynamics
equations Of each part is given. In addition, the stability Of the system is proven

using a non-time based control approach.

3. Chapter 4 provides the details Of the implementation and experimentation. In
this chapter we give the details Of the hardware and software used in the ex-
periments, then a detailed discussion about the different results Obtained is

presented.

4. Chapter 5 is the conclusion and a brief overview of future work and improve-

ments.

CHAPTER 2
THE INTERNET: REAL-TIME CONTROL MEDIA

The use Of the Internet as a communication link for Tale-robotics is not surprising re-
gardless Of its very weaknesses. The Internet can connect any two machines anywhere
in the world or in space, that is why it is a natural media for real-time tele-Operation.
But once we start talking about real-time control there are several conditions tO in-
sure the stability and robustness Of the system. The issue here is that the Internet
does not satisfy all of these conditions. That is why in the next section we will look
at the prOperties Of the Internet from a real-time control perspective and examine

their effect on the system.

2.1 Characteristics of Internet Communication

First we will discuss the services that the Internet provide for tele-Operation. Consid-
ering the communication protocol TCP/IP the Internet can provide a reliable com-
munication link. By reliable we mean a link that does not loss packets nor rearrange
them. Many protocols do not insure this, but when TCP/IP is used that means
we are given the packets in order and no packets are lost (as long as we are not
disconnected).

What the Internet does not provide is a permanent link, since the connection
might be lost at any time. And no matter how good your local area network is, once
your on the Internet you are subject to all kinds Of risks. You might be disconnected
at any time and you might face indefinite random amount Of delay; there is no upper
bound on delay over the Internet. SO the inter-arrival times Of packets can not be
estimated, for an example Of this inter-arrival time Fig.2.1 gives a graph Of inter-
arrival times Of groups Of packets on the August 1989 Bellcore ethernet trace [37] [38].

From this figure we can have an idea about the randomness Of the delay.

 

N
.4
-1
q
4
q
.1
.r
.r

.5
O
I
1

...-s
a:
I
l

...A
b
I
l

..s
N
I
l

 

A

.°
a:

 

 

 

 

 

InterarrivalflmetoHOOpackete

 

 

 

.0
a:

 

 

9
:-

 

9
re

 

o l l 1 l l 1 L l l
0 1000 2000 3000 4000 5000 6000 7000 9000 10000
PacketGrOtpNumber

Figure 2.1: Inter-arrival times of groups Of packets on the August 1989 Bellcore
ethernet trace [37] [38].

AS for the Specific network that we experimented on; we had two main communi-
cation setups. Either a station in the Robotics and Automation lab at Michigan State
University was used to control the robot, or a machine in the Robot Control lab, at
the department Of mechanical and automation engineering at the Chinese university
Of Hong Kong‘, was used. Fig.2.2 and Fig.2.3 give a plot Of the round trip time
delay between the robot and the teleoperating stations at MSU and in Hong Kong
respectively. As for Fig.2.5, it gives the frequency Of Operation; that is, the number
Of times per second that the robot receives a command and the Operator feels force
feedback during actual real-time control. From these plots we can see that the delay
is random and can not be easily predicted; therefore, any assumption made about the
time delay will be quit limiting.

Fig.2.3 and Fig.2.4 give plots Of the round trip delay in ms during different days.

 

1distance between East Lansing, where the robot was, and Hong Kong is about 7824 miles

10

 

70 r I r I I r r

8

 

Rmrnpnmoinm
8
r
l

8
T
l

 

 

 

l l l l l
5 10 15 20 25 30 35 4O
Packet number

.
p

 

10
0

Figure 2.2: Round trip delay in ms for packets transmitted between the robot and
MSU station.

 

i

I

nomruprmtnm

 

 

 

 

 

27° " .1
26° P W“ J
l 1 l 1 1
0 20 40 60 80 100 120

Padre! mm

 

 

 

Figure 2.3: Round trip delay in ms for packets transmitted between the robot and
Hong Kong station.

11

 

 

 

 

 

I l l l l l l l I

seek ]

MOL -4
{mi <
5
%300 <
p.
8230

230 ]

I l
240
22: A I l A J 1 J L l
o 20 40 so 100 120 140 180 no zoo
Plum Mum

Figure 2.4: Round trip delay in ms for packets transmitted between the robot and
Hong Kong station.

n.)

 

UI
.J

N
O
r

       

U‘
l

 

Frequency of feedback per second
8
A

 

 

00
Time smce connection In seconds

Figure 2.5: The frequency Of control and feedback signals between the robot and
Hong Kong station.

12

Horn these figures we can note that these delays change according to several factors;
especially which time Of day the experiment is being done, what day and what time Of
the year. Meaning that these delay can not be predicted, which makes the real time
control over the Internet more difficult. This existence and randomness Of delay in
Internet communication has major effect on the real time control, and more SO when

force feedback is involved.

2.2 Effects of Time Delay on Real-Time Control

The main effects Of these communication characteristics are, instability and desyn-
chronization. In 1966, the first work dealing with force feedback with time delay was
done (Ferrell 1966). It was Shown that delays on the order Of a tenth Of a second
were shown to destabilize the teleoperator. Keeping in mind that the delays we are
getting and might expect could range from 100ms tO a few seconds, we could expect
that teleoperation systems via the Internet to be unstable unless a fix is found. AS for
de-synchronization, it is the most intuitive effect Of delay, Since the Operator would
have no idea about the current robot status. With visual feedback, by the time the
Operator sees an image it would be an old image Of the robot. And taking decisions
based on that would be risky and would cause the system to get to a completely differ-
ent state. SO while the Operator thinks that the robot is in a certain location it would
have already moved tO a new one, causing by that the two tO be de—synchronized in
time and space. But these issues have no influence on the performance Of our system.
The reason for this immunity to delay is the use Of non-time based control, which will
be introduced in the next chapter.

The stability Of our system will be proven in chapter3. In addition, chapter3 will
introduce non-time based control, and will cover the detailed modeling Of the different

components Of our system, including the robot and the Operator.

13

CHAPTER 3
NON-TIME REFERENCED PLANNING AND CONTROL
FOR INTERNET-BASED REAL-TIME
TELE—OPERATION

In this chapter we introduce non-time based planning and control, and our system
with its different components. We start in section3.1 by introducing the event based
approach for control and planning, then section3.2 gives a general structure Of the
system with the detailed modeling Of all the components. Then in section3.3 we will
compare our model to other models used, specifically the one used in[20]. In section3.4
we analyze the system and discuss the event-based reference that was used, and we

prove the stability Of our approach.

3.1 Non-Time Based Real-Time Control

The delay that exists in communication links have several effects on the stability and
synchronization Of the system. But this delay is resulting from the use Of time as
our reference variable; therefore, if we were able tO use in our system a non-time
based reference it would become immune tO delay. This suitable action or motion
non-time reference variable, is called event. The use Of event-based planning and
control provides the teleoperation system the capability tO c0pe with uncertainties
and delays in real time, without the need to replan or resynchronize [25]-[27].

In traditional control systems the dynamics of the system is modeled by differential
equations in which the free variable is the time variable t. And usually the trajectory
is a function Of time, but if we do not constraint the trajectory by time we would
allow the system to be at any point at any time. SO the general idea is to model the
system and the trajectory with parametric equations. The parameter is called motion

reference or action reference, and usually denoted by s [25]-[27]. The planning and

14

am
y‘tt) ctt) y(t)

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Planner Controller _. Robot p
(a) traditional control
d (t)
Ydf 8) 0 (I) y (t)
Planner Cont rolled, Robot: ,

 

 

 

 

 

 

 

 

 

 

 

 

 

Non-Time Beeed
Action Reference ‘

 

 

 

 

 

(b) non-time based control

Figure 3.1: The comparison between traditional time-based and event-based planning
and control.

control Of the traditional time-based and the event-based schemes is shown in Fig.3.1.
The Action Reference block in Fig.3.1 acts like a clock for the system. It is a map
from the output or state Of the robotic system y to a scalar variable 3, and the robot
plan as well as the Operator commands can be described as functions of this variable.
This reference is usually taken to be a physical quantity; distance, position. In our
case we will use a different approach, we will use an imaginary reference because in
real time teleoperation we dO not have a predefined path or trajectory to follow.
This new approach, which was first introduced in [43], has been used in many
studies and several robotic applications[26], and have been proven tO be very efficient
in dealing with uncertainties and delay. But this is the first time that this approach

will used for real-time control with force feedback.

3.2 Modeling of The System

We include here the details Of the modeling Of the system. We will include in Fig.3.2

a general block diagram Of a traditional teleoperation system, and then in Fig.3.3 we

15

present a detailed schematic diagram Of our system. The modeling Of the blocks in
Fig.3.2 and Fig.3.3 will be discussed in details and each term is explained in Table3.l.
As Fig.3.2 Shows, the general teleoperation system consists Of a Human Operator,
master‘, communication network, Slavez, and the environment. The same applies to
our system as shown in Fig.3.3 but with some modifications with the connections.
Now we will model each block in our system giving the details Of each variable
and the equations that relate them. The thing to note is that all the models and
dynamic equations are in terms Of our event 3, and that we are controlling a mobile

robot with 3 degrees Of freedom, which makes our variables vectors with 3 terms.

1. Human Operator: This was the hardest part to model, but eventually we as-
sumed a spring-like behavior which was shown in [29] and used in several places
in the literature [28]. SO inspite Of all the internal complexity the wrist has a
spring-like effect. In addition, the human can compensate for certain machine
instabilities making the coupled human-machine system stable [30]—[33]. SO once
the Operator feels a force he will generate a new joystick position according to

the following:

Xm(s) = 5%? (3.1)

where K, is a constant and X7,, and F}, are:

 

 

 

 

- Xm(3) - P Fin“) 1
Xm(s) = x...(s) Ms) = Fins) <32)
. Xm9(3) .1 . FM(3) ..

As table3.1 indicates Fh(s) iS the applied force, which means the force that the

1Usually the master is a joystick.
2Usually the slave is a robot.

 

16

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Vnu Vm Vsd Vs
Human _’ Master "TE Commun- —’ Slave "" Environ-
Operator ‘— ‘_ ication ‘— ‘J ment
Fh Fmd Fe

Figure 3.2: Block diagram Of a traditional tele-Operation system.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Vin“) Environ-
. ment ‘—
Xm‘” Vm‘” vms) v.(s)
Human Master Commun-‘ Slave V (s)
Operator ication ‘
Master Commun-
' F ication t (s
h m
Tm(sl

 

Figure 3.3: Detailed block diagram Of our teleoperation system.

 

Block Traditional Variables Our Variables

 

Fh: applied force
Xm: position Of Joystick

Human Operator Fh: applied force

 

Master Vm: velocity desired Vm: velocity desired

 

Fmd: desired force
V“: desired velocity

Communication Link

 

 

Slave V,: velocity Va: Actual velocity Of robot
F,: force rm: desired force
Environment Fe: contact force V2": virtual contact effect

V,: velocity set for the robot

 

 

 

 

 

Table 3.1: The Explanation Of the various variables in the Tele-Operation systems.

17

Operator feels (Newton’s third law). Thus, the z and y components are due tO
the force fed back and due to the additional force required tO get the joystick
to the new location. Since we do not feed back force in the 0 direction, this
component is just a result Of getting the joystick tO the next location. As we
see in eq.3.1 Xm(s) is related tO Fh(s — 1), SO Xm(s) at event 3 is generated by
the previous force at event 3 — 1. This results in having an event-based system
where each event is triggered by the previous one. Several other issues related
to human behavior where taken into account in the design and these will be

discussed in chapter4.

. Master (Joystick):The dynamics Of the joystick are described in the following

equation:

Mmem(s + 1) = Fh(s) + Tm(3) (3.3)

where Mm is the mass Of the joystick handle, me is velocity Of joystick move—
ment, F}, is as described before and rm is the feedback from the robot and which
would be the force played by the joystick. The result from this dynamics is the
joystick getting to a new position Xm(s + 1), from this position the desired

velocity Vm is derived according to:

v...(s) = Kme(s) (3.4)

where Km is a scaling constant, X m(s) as before and Vm (s) is the desired velocity

18

Of the robot. The different vectors are:

 

 

 

 

- me(3) 1 - me(3) 1
VTR(S) = me(3) Tm(8) : Tmy(3) (3'5)
- Vmg(8) .. . 0 _]

From 3.5 we see that rm9(s) = 0, Since we do not feedback force in the rotational

direction.

3. Communication Block (Internet): Resulting from event based control, the com-
munication link is Simply a delay element that plays no role in the modeling Of
the system. We assume that the Internet is simply supplying the link and in
case this connection is lost, the system will simply stop awaiting the connection.
Since the advance Of time does not affect our system and only the advance Of
the event 3 will, when the connection is lost the system will still be stable and
would resume action after the connection is back. This makes the system very

robust Since no initialization or synchronization is required.

4. Environment: In our system we interact with the environment in a new way.
We dO not have contact to generate force, we actually use the different sensors
around the robot to detect Objects. Based on our distance to the Objects in our
way, we generate a virtual force. This is done by having a function Of distance
f (d)3 that would give us a velocity value V,-,,(s) to subtract from the desired

velocity Vm(s).

V.(S) = Vm(8) - vi (8) (3-6)

where V, (s) is the velocity set to the robot, Vin(s) is the effect of the environment

 

3Details of this function are included in section4.1.2.

19

and Vm(s) as before. SO now the desired velocity that the robot gets is less

than the actual one, this slowing down will be used to generate the virtual force

feedback. Keep in mind that we do not generate force in the rotational direction

that is why V,-,,9(s) = O and V,9(s) = Vm0(3), as seen in eq.3.7.

 

 

Vin(3) =

 

v...(s) ‘
lenv(3)
O

 

. Slave (Mobile robot): Once the robot receives V,(s) it will he commanded tO

move with that velocity but would actually have V},(s) as its velocity. Then the

robot will calculate the velocity tracking error with respect to original desired

velocity Vm(s) and send that back tO the master to be played as the virtual

force due to the environment. SO rm(s) and the dynamics Of the robot are:

rm(s)
M,Va(s)
73(3)

14,.(3)

1(r(l4n(3)"lvh(3))

Fe(s) + T,(s)

—'yVa(s) + KVe"(s) — afFe(s)

V3 (3) - Va(3)

14(8)

.-

Ms) 1
Var/(3)

 

 

.. Vm0(3) .J

(3.8)
(3.9)
(3.10)

(3.11)

(3.12)

K,, '7, K and a are constants, M, is mass of robot, Fe is the actual environment

2CD

forces if any and usually assumed very small. 7, and V8,, are robot internal
controller terms. Eq.3.8 Shows us that what the Operator is feeling is actually
the velocity tracking error, which could be a result Of getting close to an Object,
this implies that from the direction and magnitude Of the force we can know
where and how far are the Obstacles to the robot. The important point here is
that the Operator will still feel force in case Of actual force being applied to the
robot during contact. In the actual contact case Va(s) would decrease by that

increasing rm(s). Moreover; we take Vag(s) = Vmo(3) since we have rmo = O.

3.3 Comparison with Other Models and Approaches

It is very difficult to compare our model to the other approaches used in teleoperation
in general. Because the approach we use is quit different and many Of the presented
systems are not even modeled. Moreover, our system has more complex parts that
deal with Obstacle avoidance, which will be discussed in more details later. There are
some common methods used to model and analyze teleOperation systems, the most
- pOpular ones are networks, wave variables and dynamic equations. The model and
analysis we examine here is based on network modeling and was used by Anderson
in [20] and [21].

Anderson bases his analysis on the fact that for the system tO be stable, it Should
be passive. Using the analogy between mechanical and electrical systems he repre-
sented a teleoperator as a network, as Shown in Fig.3.4. In this model, the master,
communication block, and slave are represented by two-port, as for the Operator and
environment they are represented by one-port. An n-port is characterized by the
relationship between effort F (force, voltage), and flow 12 (velocity, current). This

relationship for one-port is specified by its impedance Z (3) according to:

F(s) = Z(s)v(s) (3.13)

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

v", v... Va: V,
—e —> —> —>
f\ A f\ M
J V II V J I V I I V I
Human + Master + Commun- + Slave + Environ-
Opcrator F” Fm“ ication F‘ F‘ ment
I A I I__O J J n 1 I A J
V U \f

 

 

Figure 3.4: Network representation Of teleoperator, the different terms are as ex-
plained in table3.1.

 

 

v1 v2
—> —>
I I
+ +
n

 

 

 

II

Figure 3.5: Transformer.

where F(s), 12(3) are the Laplace transforms Of F(t) and v(t). As for the two—port,

the relationship is:

171(3) h11(3)h21(3) ”1(3) =H(3) ”1(3) (3.14)

—oz(s) h12(s)h22(s) F2(s) F2 (3)

where H (s) is the hybrid matrix and F1, F2, 111 and 222 are as defined in Fig.3.5.

For the system tO be stable, its different components have to be passive. That is,
they can dissipate energy but can not increase the total energy Of the system they
are part Of. This is why Anderson modeled all the components Of the system using
passive circuit elements. And then to achieve a stable system under time delay, he
choose a control law that would make the characteristics Of the communication block
identical tO a two-port lossless transmission line, which is a passive element. The

derived control law for the communication circuit is:

22

Fmd“) = F.1(t- T) " ”ad“ — T) + Um“) (3'15)

2),. = vm(t — T) — F,(t) + Irma — T) (3.16)

where T is delay, and the other terms are explained in table3.1. Comparing this
approach and model tO ours, we find several differences. The stability analysis in
this case relies on the Specific model used, whereas in our case the stability analysis
is done independent of the model. Moreover, the model here assumes same amount
Of delay in both directions Of the communication; in our model we do not make any
assumptions regarding the delay. In addition, the human model here is assumed to

be passive but in our case it is not.

3.4 Stability Analysis

Several methods have been used tO analyze the stability Of teleoperation systems,
scattering theory [20] [21], wave variables [22]. The main concept used in their analysis
is based on proving that the system is passive. A passive system can not create energy,
and thus from a control point Of view a passive system can not go unstable [20]. Our
approach is slightly different, and the first difference is our use Of an event based
control. Concerning the stability Of the system under event-based referencing, the

following theorem was proven in [25]-[27]:

Theorem 1 If the original robot dynamic system ( without remote human/autonomous
controller) is asymptotically stable with time t as its action reference; and the new
non-time action reference, 3:“ (y) is a (monotone increasing) nondecreasing func-
tion of time t, then the system is (asymptotically) stable with respect to the new action

reference 3.

23

The only assumption we need to make is that the robot is a stable system, which
means that the original robot dynamic system (without remote human Operator)
is asymptotically stable with t as its action reference. This would allow us tO use
theoreml and prove the (asymptotical) stability Of our system with respect tO the
new action reference 3, Simply by proving that the new non-time action reference
is (monotone increasing) nondecreasing function Of time t. The advantage Of this
approach is that stability is proven independent of the human model or the statistics
Of time—delay. Meaning, the system would be stable for random time delay and
regardless Of the Operator.

The major difficulty Of selecting s was the many uncertainties in the system. The
main uncertainties were, the path (trajectory) and the environment. Usually the
event 3 is taken to be a function Of the physical output; for example, the distance
tO the origin, the angle, the absolute position. But, in tele-Operation, usually non Of
these parameters is defined. In our case the selection Of the event will be clear after
we explain the flow Of commands and feedback on our system. And this is the first
time this Specific approach have been used for event-based teleoperation.

This is how the control and feedback commands are communicated; the Operator
places the joystick in a certain position that corresponds to velocity vector, the be—
havior Of the joystick is similar to the gas pedal in the car. This vector is sent to
the sensing unit on the robot. The Sensing unit scans the environment and based
on the position Of Obstacles the velocity is reduced and sent tO the robot motors to
be executed. The motors will execute the command, after that the actual velocity is
calculated. Then the actual velocity is subtracted from the original velocity required
by the Operator, and this difference is sent back to the joystick motor to be played as
force. The important thing in this Operation is that non Of these step can be executed
out Of order, each event is triggered but the end Of the previous one. And what should

be clear here is that during all this time, although the Operator could move the joy-

24

stick but his commands will not be sampled by the joystick until feedback is received
and played. This might seem to cause instability or de—synchronization, but it does
not. On the contrary, every force the Operator feels, he can be sure that it is the
most recent one and that is this the current status of the robot, so he will be always
synchronized with the robot. In addition, he can be sure that his new command is
the next one to be executed and no other commands are pending. So for example,
the Operator sends a series of commands {V1, Vg, - -- ,Vn}, V1 will be sent to the robot
and feedback will be awaited. Until feedback is received, no other command will be
sent, SO {172, - - - ,Vn} will actually be discarded by the system. When feedback is
sensed, the operator would assume that its the result of {V1, 172,-- - , Va} but actually
its only the result Of V1. Inspite of this all the system components are synchronized
and stable, because no commands and no feedback is flowing in the system. Now the
Operator based on his feeling can take a decision and send a new set of velocities, and
the whole sequence repeats. SO the control and feedback Speed varies depending on
the time-delay and the speed of the different machines used but will have no effect
on the stability of the system. Based on this control algorithm a natural selection of
the event variable 3 was some kind Of a counter. In our case, it is taken to be the
number of forces felt by the Operator since the connection was established. This way
the Operator would know when the reference increments and the robot would know
the exact 3 Since it corresponds to the number of commands executed till now.

The Operator might not know the exact value of s but this is not required since
knowing when it is incrementing would be enough to inform the operator that the
system is advancing to the next step. Its obvious that s is nondecreasing function
of time t, since at any time the number of forces felt or commands executed can not
decrease. We can run a new command or feel a new force but we can not unfeel or

undo an execution.

25

This approach constitutes a general method to designing and implementing event
based teleoperation systems. And based on this design the system stability, synchro-
nization and robustness is guaranteed, since the proof given above is not based on a
particular model or system. So our frame work applies to any teleoperation system,

regardless of the robot used.

26

CHAPTER 4
IMPLEMENTATION AND EXPERIMENTATION

The implementation of the project was mainly done in house. We have not seen in
the literature a similar implementation, with the Specific hardware used. In the next
section we present the details of the system structure and the implementation details

and difficulties. After that we present the experimental results of our work.

4.1 System Implementation

We can divide the implementation into two parts; hardware and software. In hard-
ware, we will detail the different systems used and their Specification. In addition, we
will describe their connection and communication. As for software, the main struc-
ture and the different modules will be examined. Then we will explain how obstacle

avoidance was achieved and how the virtual force was generated.

4.1.1 Hardware

The diversity of the hardware used caused some problems in the integration of all the
parts. As we see in Fig.4.1 and table4.1, the system consists of different Operating
systems and different configurations. SO the problem of interconnection had to be

studied carefully. The system consists of:

1. Joystick: Programmable force feedback joystick, which is used to obtain com-
mands and play force. Joystick is connected to the local PC with WIN98 Oper-

ating system.

2. Local PC: Used to control the joystick and interface to it, and to communicate

over the Internet with the mobile robot.

27

Flt Xm Vm vm vm

fl mm W?

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

I
Human Joystick Local Internet Access E
Operator PC Point 1 7
I I I I E I E [ 1 Processing Unit
1,, z... m Tm in v. v...
V.
Sensory
Unit
Motors
Mobile Robot

 

 

 

Figure 4.1: Hardware structure of system.

3. Access point: Makes up the link between the Internet and the mobile robot via

wireless communication.

4. Mobile Robot: A mobile robot that is equipped with several sensors (Infrared,
Ultra sonic and Bumper), which are used to detect and avoid Obstacles. The
robot will execute the velocity commands depending on the surrounding en-
vironment and then calculates the velocity error and sends that back to the

joystick.

5. Camera: The robot has a camera mounted on tOp to allow visual feedback to
the operator also through the Internet.

4.1.2 Software

The deveIOpment Of the software took a considerable amount of time because of the
complexity Of the systems and their diversity. The main issue was the ability to have
two different Operating systems, Linux and WIN98, communicating together. Here

came the role of the communication protocol TCP/IP, which can be understood by

28

 

 

 

Device Specification

Joystick Microsoft SideWinder Force Feedback Pro.
3 degrees of freedom

Local PC PentiumII 300MHz, 128MB, WIN98

Interfaces to the joystick through the game port
and to the Internet through an ethernet card.

 

Access Point

Proxim RangeLanZ radio access point
Connects the robot to the Internet

 

Mobile Robot

Called ARON E

Nomadic XR4000, 2 PCS on board
Pentium 233MHz, Linux
Interfaces to the Internet through
wireless ethernet card
Equipped with 3 kinds of sensors, Infrared,
Ultra sonic and Bumper

 

Camera

 

 

Sony EVI—D30 camera
Connected to the robot through a frame grabber

 

Table 4.1: Specifications of the system hardware.

both Operating systems. So the Internet was not only acting as a communication link
but also as a translator. The software developed can be divided into three main parts;
motion server, camera server and client. Motion and camera servers run on the robot

and where developed under C++; where as the client, runs on the local machine and

was written in Microsoft visual C++.

1. Motion Server: In Internet communication usually we have 2 kinds of processes,
servers and clients. Servers job is to wait for a client to connect to it and request
a service. In our case the service is moving the robot and getting back feedback.
AS seen in Fig.4.2, this is mainly what the motion server does. Except that the
server does not execute the request blindly, it first checks the sensors and based

on that a decision is made according to the flow chart in Fig.4.3 that will be

explained in more details in the next section.

After the velocity to be set is decided, it is sent to the motors for execution.
Then the server checks the actual velocity of the robot and subtracts that from

the original desired one Vm and sends it back to the client as force feedback. TO

29

 

MOtion Server

 

   

Client Camera Server
Connect to Robot C .
Servers onnectron
J

 

 

 

 

 

 

Accept
connection

 

 

 

 

 

 

Translate position
of joystick to
velocity commands

 

i

 

 

Check sensors
for Obstacles

 

 

Send velocity
commands to
Motion server

 

 

 

 

I

Force feedback \I
received?

 

Reduce speed
according to
Obstacles

 

Y

 

 

f

 

No

Y.

 

 

 

Measure actual
velocity and
feedback error

 

 

 

Play constant
force

 

 

 

 

 

 

 

 

     

Button 1
Pressed?

 

request
received?

 

  

 

Accept

 

Translate position
of joystick to
camera commands

connection

 

 

 

 

i

 

 

Send camera
commands to
Camera server

 

 

 
      

Acknowledgment
received?

Yes

 

 

 

 

No

 

Send commands
to camera

 

 

I

Send
acknowledgment
to client

 

 

Figure 4.2: Motion server, Client, and Camera server general flow chart.

   

Stop motion
Set Flag

 

 

Infrared Close
than dc?

 

 

Y
Stop motion

 

 

No

 

 

Check
Ultra-sonic ~>

 

 

Set velocity to
(Vm'f(d)vm)

 

 

 

Figure 4.3: Flow chart for deciding velocity based on sensors.

30

 

over come the problem of disconnection, the server would execute the velocities
for 250ms; after which, if no new command is received, it would time out and

stOp moving waiting for the next command.

2. Camera Server: This server is much simpler than the motion server, its job is to
Simply interface to the camera. When the client sends camera commands (pan,
tilt and zoom), the camera server sends these commands to the camera through
the serial port and sends back to the client a confirmation that it has received
the request. During all of this, even when the client is not requesting camera
motion, a process would be running in the background sampling the camera
as fast as possible and displaying these images on a web page1 to be used as
visual feedback for the Operator. The image is updated on the web page at a

frequency Of 1 frame per second.

3. Client: This is the program that runs on the local machine and communicates
with both servers and the joystick. The client sends commands to either of the
servers based on one of the buttons, whether it is pressed or not. If the button
is not pressed the joystick position will be translated into velocity commands
and sent to the motion server. On the other hand, if the button is pressed, the
position of the joystick is translated into pan and tilt commands for the camera
and 2 other buttons specify whether zoom in or out is requested. In addition,
based on which state the client is in, it will either send the force command back
to the joystick once feedback is received or just wait for acknowledgment. The
communication with the joystick is done with DirectX technology, and with
the servers its done over the Internet. The client has a few other convenient
Options, for example force effect can be turned off, and disconnection can be
done with one button. Another client implementation trick was to delay the

sampling of the next command, so when feedback is received and force played,

 

la.rone.egr.msu.edu

31

 

Figure 4.4: The XR4000.

the client would wait 250ms and then sample the position of the joystick. This
way the Operator is given time to feel the force and plan the next step based
on that. Moreover; the force played is actually the average Of the last 10 forces
received, this way the high frequency changes in the force will be filtered giving

the Operator a smoother feeling of force with no abrupt changes.

4.1.3 Obstacle Avoidance and Virtual Force Generation

Obstacle avoidance was a very important part Of the project that affects the behavior
of the whole system, that is why we will discuss it in details here. Because of delay
we had to make the robot more intelligent, since the Operator might be seeing an old
image whereas the robot is very close to Objects. This might cause a collision, and to
avoid this we had to keep the final decision of the velocity set to the robot Since the
sensory unit are the ones with the most up to date environment information. We will
first give a detailed explanation about the sensors available and there distribution on
the body Of the robot, then we will explain their use for obstacle avoidance and force
generation.

AS we see from Fig.4.4 the XR4000 is a barrel shaped mobile robot. It is equipped

32

Set 0=top set on front door (sensom0-7)

Set 1=bottom set on front door (sensor#O-7)
Set 2=top set on left door (scnsor#0-7)

Set 3=bottom set on left door (sensor#0-7)
Set 4=top set on right door (sensorlt0-7)

Set 5=bottom set on right door (sensor#0-7)

Set 2.3

 

Figure 4.5: A top view Of the robot that gives the distribution and numbering of the
different sensors.

with 3 kinds of sensors:

1. Tactile Sensors (Bumper): They provide information about the physical contact
with the environment. It is hoped that our Obstacle avoidance would work, but
since nothing is perfect these sensors will be used to detect any actual contact.
The Nomad XR4000 has 48 bi-level tactile sensors that surround its top and
bottom perimeters. Additionally, it has 4 door bumpers on each door that sense
contact between the tOp and the bottom perimeters. The sensors are divided
into 6 sets, 2 per each Of the 3 doors on ARONE. Fig.4.5 gives the distribution

and numbering of these sensors[44].

2. Infrared Proximity Sensors: Give a range information to nearby objects (typi-
cally less than 30 to 50 cm away). The range is determined by emitting infrared
energy using high-current LEDS and sensing the amount of returned energy
with infrared photodiodes. The thing to note is that the returned energy is a
function of the object’s reflectivity, that is why these sensors are not very good
for the measurement of distances. The robot has 48 of these sensors distributes

just like the tactile sensors, in 6 sets and have the same numbering shown in

Fig.4.5[44].
3. Sonar Proximity Sensors: These provide range information to Objects that are

33

relatively far away (between 15 and 700 cm away). The distance is calculated
by multiplying the speed Of sound by the time of flight of a short ultrasonic
pulse traveling to and from a nearby object. The robot has 48 of these sensors
distributes just like the others, in 6 sets and have the same numbering shown

in Fig.4.5[44].

As for obstacle avoidance and force generation, it is accomplished according to
the flow chart in Fig.4.3. First all the bumper sensors are checked, which gives 360
degrees coverage around the robot. If any is hit, the motion is stOpped and a flag
is set and sent back to the joystick where a sudden jerking force is played tO tell
that a bump has occurred. If non is hit, the infrared sensors, 90 from both Sides of
the desired motion, are checked. If any, detects an object closer than dc, which is
a predefined programmable critical distance, the motion is stOpped. If it is non of
these cases, the ultra-sonic sensors are checked. Again we check 90 from both sides
Of the desired velocity. Based on the one that gives the closest distance d, we set the
velocity to V,, which is given in eq.3.6, where Vin = f (d)Vm. So the velocity set would
be a fraction of the desired one based on the predefined programmable function f (d)
f (d) can be any function of distance depending on how conservative or risky you
want to be. In our case f(d) is plotted in Fig.4.6. So the robot will Slow down linearly
when an object is detected between 0.5m and 1m, then it would stOp if an Object is
detected closer than 0.5m. As seen in Fig.4.7, we end up with two regions around the
robot, one that would cause the robot to Slow down and another to completely stOp.
Now the force is generated by measuring the actual velocity of the robot and taking
the difference between that and the desired and sending it back to the joystick. To
avoid the problem of disconnection and long delay, the robot would move for 250ms
only. After these 250ms, in case no new command is received, the robot would stop

and wait.

34

F(d)

0.5 l d

Figure 4.6: The fraction of velocity that is deducted based on the distance din meters
to the closest obstacle.

 

Figure 4.7: The two safety regions around the robot according to the direction of
motion.

35

4.2 Experimental Results

Several experiments where done during the duration of the project, we will include
the most important and relevant to our discussion. Two kinds of experiments where
done, one is where we control the robot over the same LAN, and another is where

the robot was controlled from Hong Kong.

4.2.1 Tele-operation Over the Same LAN

There was 3 scenarios for the control over the same network; first case is normal
Operation, second case is where we induced 2 seconds of round trip delay and third is
when we had simulated random time delay. The results of these three cases are plotted
in Fig.4.9, Fig.4.12 and Fig.4.15 consecutively, and will be discussed in more details.
But first let us examine the performance of the system in a friendly environment
with relatively far Obstacle, and let us see how close the robot actually tracks the
desired velocity and if it is stable. In this experiment, the Operator tries to keep away
from obstacles (about 1m away), but with no other constrains. Meaning there is no
predefined path and the Operator is free to move around randomly. Fig.4.8 shows a
plot of the operation of the robot with most of the obstacles being more than 1m
away. The figure Shows the desired a: and y velocities, then under them we see the
actual r and y velocities, and the last row gives the distance detected at each event.
All of these plots are versus event and not time, Since that is our reference. We can
see that we have a very close tracking of the desired velocities, except when the robot
gets to Obstacles which are closer than a meter, which is what we expect. SO when so
obstacles are found the system is showing a close tracking performance and is stable.

When the environment becomes more hostile, that is having several close objects,
the behavior Of the system changes. Since now we have obstacle avoidance taking

place. In all the following experiments, the Operator is not constrained, complete

36

 

 

o8

 

   
 

Desired X Velocity
I
8

 

 

 

 

 

 

200 400 600 800 1000

 

M v v

 

 

Actual X Velocity

 

 

 

 

 

 

200 400 600 800 1000

 

 

I V V

i

i

g

   

 

Distance to Obstacle
Distance to Obstacle

 

 

 

 

L

o I i L L 1' L I
O 200 400 600 800 1000 O 200 400 600 800 1000

O

 

Figure 4.8: The behavior of the system under normal Operation with relatively far
Obstacles.

37

freedom is given to move around in any direction or fashion and is allowed to get close
to Obstacle as desired. To understand the system Operation under these conditions
let us consider Fig.4.9. In the first row we have a plot of time versus 3, the event. Its
clear that s is nondecreasing function of time, which was very crucial in the proof of
the system stability. The other plot in the first row is the desired rotational velocity.
The second row is the desired velocities in :r and y directions, which we call Vm. The
third row displays the actual velocities in the both directions, which we call Va. Next
we give the force that is played by the joystick in both directions, we call this rm. The
last row displays the same plot, which is the closest distance detected to obstacles in
the environment. To get an idea about the performance and behavior of the system
let us consider all these plots together. Looking at the actual velocity we see that it
is changing according to the distance detected, for example if the distance gets closer
the actual velocity decreases, when this happens we see that the force increase, which
is what we expect. Thus a close object causes the robot tracking error to increase
by that telling the operator that an obstacle exists. Whenever the robot gets to the
critical distance of 0.5m the actual velocity becomes zero, and this applies to both 2:
and y directions. At the time when the robot gets further away from obstacles the
actual velocity starts tracking the desired one again, by that reducing the tracking
error and therefore the force. So looking at the sum Of the actual velocity and the
force plots, they should make up the desired velocity. This is what Fig.4.10 is Showing,
the first row is the desired velocities and the second row is the sum of the force and
the actual velocities. It is clear that the plots are Similar although not identical. The
difference is due to the fact that the force we play is a filtered version of the tracking
error as seen in Fig.4.11. In this figure we Show the tracking error and the actual
force felt, which is clearly a low pass filtered version of the error. As we mentioned
before, this is done so that the force felt is more smooth and no abrupt changes are

felt. That is why the plots in Fig.4.10 are not identical, since whenever the tracking

38

error changes suddenly the force would change slowly, and that is why the differences
in these plots are at the points where the actual velocity is directly reduced to zero,
by that causing a big sudden increase in the tracking error.

As for the second case, where we induce in the system an additional round trip
delay of 2 seconds, the results are shown in Fig.4.12. The plots are the same as
the ones shown in Fig.4.9, which were explained before. These plots Show that the
system have a similar behavior to the case where no additional delay exists. The
actual velocity tracks the desired one until an Obstacle becomes close Then the actual
velocity starts deviating from the desired one, by that increasing the tracking error.
This increase would increase the force felt. The opposite happens when the robot is
getting away from an obstacle, the tracking becomes closer and the error, thus the
force, decreases. Again adding the force and actual velocity would give a result similar
to the desired velocity as seen in Fig.4.13. The cause of the differences is again the
filtering that was mentioned before and seen in Fig.4.14.

The third case is when we simulate a random time delay, that can range from 1
to 4 seconds. The results of one of these experiments is shown in Fig.4.15. The same
thing that we saw for the previous two cases is displayed here. The tracking error,
that is the force, is a function of the distance detected. The closer the distance the
more force is felt and the slower the robot moves, until it comes to a complete stOp.
Then as the distance increases, the force reduces and the robot follows the commands
closer. A plot of the sum of the force and the actual velocity is Shown in Fig.4.l6.
Here too the errors are due to the filtering illustrated in Fig.4.l7.

Considering all these cases, we can conclude that the robot is stable as proven
before. And more importantly, is that this stability is unaffected by the time delay,
which can be constant or random and can take any value. Not only the robot is
stable, but also has a very close tracking performance, where the actual velocities

track the desired ones very closely as long as there are no Obstacles. Once obstacles

39

 

 

 

 

 

 

 

    

  
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

. ............ ....... m m .................... m m m
. ...... . ....... ...... m m ..................... m m m
................... ..m. .m. m .m. .m.
m ... a. ... m m. ... a m m. .... m .
e_8_o> seesaw Samoa e_8_o> > 3:30 e_8_o> > .52 20330 2 8535
m J. .I m m w .i m

300

300

.
. .
. .
.
.
. .
............. .....uaem
. s
. .
.

 

 

200

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

w w W W o w W
mmmommommmmommmmmommmmo
6:89. m. 95... b_8_o> x noemoo §3_o> x Rag 090a x 0.0830 2 3:530

Figure 4.9: The behavior of the system under normal Operation.
40

 

 

 

 

 

 

 

 

 

 

300 . . .
E. 200 t .................................. 4 g.
3 . .. 8
3 1 00 ................................ 3
> >
X 0 ...... >.
E 3
'6 -100 .................................... '6
8 : T : 8
‘2“) ......... ........ ......... ........
—aoo - . -
0 1 00 200 300 400

F'

 

 

_-'. km wran- w—eI-f

ifm‘
.
\.

 

 

 

Actual Y Velocity + Y Force

 

 

Actual X Velocity + X Force

 

 

   

O 1 00 200 300 400

Figure 4.10: Comparison between the desired velocities and the sum of the actual
velocities and the force felt.

X Component. Normal Operation Y Component. Normal Operation

 

 

 

Tracking Error
3
0
Tracking Error
0

 

 

 

 

 

 

 

 

 

   

 

 

 

 

 

 

o .200 4
-100 i i ; —400 i i 4
o 100 200 300 400 o 100 200 300 400
300 f _' : 400 f f f
g 200 g 200
E '73.
100 2 0
§ §
63 0 L8 -200 »-
—1oo - L '- —400 1 i L
o 100 200 300 400 o 100 200 300 400

Figure 4.11: Comparison between tracking error and force felt for normal Operation.

41

 

 

 

 

 

 

 

 

 

 

   

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 
  

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

fl. . . a . w a J. a.
. ........ 1 .......... .m. . ......... . .......... .m. . ............ A ....... .m. . ................... .m.
..................... m .w .Rou
m I m H H . . m
.t o h 0 w o i. 0 A0 h o m . 00
.3 3m. 32 3 7a.:
23% .aceaam 3....th 2.8.; > 8:30 58.; > .33 83a > 20330 2 85220
t . m. m m
. .................... m m .m.
T ................... w w ...w
W o o W o l. o H 0
was... :03 :03 :03 m m m ..

Figure 4.12: The behavior of the system under 2 seconds of round trip delay.
42

 

 

    

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

300 a, - 300 - a
z~ 200 E 200
:g 100 .§ 100
> >
x >- o
8 8
i 3“”

-200
-300

§ 300 a - § 400

200
E E zoo’
+ 100 +
z- .39
3;, o g o
> >
x400 >-
2 . . 3 ;

-aoo - - —4oo - L

so 100 150 0 so 100 150

Figure 4.13: Comparison between the desired velocities and the sum of the actual
velocities and the force felt.

 

X Component. ZSec Delay Y Component. 230a Delay

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

400 400
g 200 g
LU u]
D O
c O .c
i i
r- -200 +-
-400 L L
o 50 100 150
400 ._ t 300 _. s
' ' 200 ' i
.3
3 100
§ 0
O
“- -100
—400 L L —200 L L
0 so 100 150 0 so 100 150

Figure 4.14: Comparison between tracking error and force felt for 2 seconds delay
Operation.

43

 

 

 

 

,
150
150
150
150
150

 

 

50

  

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 
 

amass

 

 

 

 

 

w 0 w 0 0 00
m a ... a. _ a. a. a
_
b_oo_o> .mcozflom poemoo b_oo_o> > 3:80 b_oo_o> > 332 cocoa > 0.0330 2 3.67.5
a a . a. . ._ a. - ....

  
 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

.m o M . o . . 0 II N 0
mam .. macs.“ :03 “was.“ a. m. a ..
”Ur—000m m. DEC. b_8_0> x 85300 >u_00_0> X 332 0050mm x ESuMDo Ou chflo

 

 

 

 

Figure 4.15: The behavior of the system under random round trip delay, ranging from
44

l to 4 seconds.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

   

 

 

 

 

 

 

300 fi T
j? j? 200
g g 100
>» >»
>< >- 0
It I?
'6 -100 '3 -100
8 8
-200 -200
-3“) 4 400 I L
O 50 100 150 0 50 100 150
g IMX) . . g 4mo v .
o o
32 200 t
+ +
z~ .2:
B 0 3
3 3
x >-
3 -200 a
‘3 3
2 . E . .
-400 ‘ ‘ -400 ‘ ‘
0 50 100 150 0 50 100 150

Figure 4.16: Comparison between the desired velocities and the sum of the actual
velocities and the force felt.

X Component, Random Delay Y Component, Random Delay

 

 

 

   

 

 

 

 

 

 

400 400
E 2
DJ LIJ
I) O)
F .c
l- l—
—400 L *4 -400 1 ‘
0 50 100 150 0 50 100 150
400 -

 

 

........................

 

 

 

 

 

 

 

 

0 50 100 150

Figure 4.17: Comparison between tracking error and force felt for random delay
Operation.

45

are detected, the robot uses its obstacle avoidance algorithm. This algorithm allows
it to avoid collision and more interestingly, generate force feedback that would notify
the Operator of the existence Of these obstacles. An important note here is that,
since we did not have a predefined path or trajectory and the Operator had complete
movement freedom, all the things discussed here would apply regardless of the path.
This is desirable since in tele-Operation we rarely have a pre-known complete idea

about the trajectory.

4.2.2 T ale-operation from Hong Kong

As for the experiments done with the Robot Control Lab in Hong Kong, we experienced
a control frequency2 between 1.4 and 1.5 events/ sec. Some of the results of these
experiments are shown in Fig.4.18 and Fig.4.19. These plots are similar to the ones
in Fig.4.9, Fig.4.12 and Fig.4.15. In all the following experiments, the Operator is not
constrained, complete freedom is given to move around in any direction or fashion
and is allowed to get close to Obstacle as desired.

The first plot was time since the beginning Of the connection versus our event 3,
it is clear that in all the cases 3 is a nondecreasing function of time. Then we plotted
the 3 desired velocities in the x, y and 0 directions. Under those, we gave the actual
velocities and forces in the :c and y directions. Last we gave the distances to the closest
Obstacles detected. Observing those figure, we see that the forces increase as we get
closer to objects and decreases as we get further. Meanwhile, the Opposite happens to
the actual velocity, where it becomes smaller and smaller than the desired one as we
approach an object, and starts approaching the desired velocity as we go away. For
all of these cases if we add up actual velocity with force then we should Obtain the
desired velocity. This is what is shown in Fig.4.20. There we see the desired velocity

and under it the sum of the force with the actual velocity. As is expected, the two

 

2Control frequency is the number of times per second that the control commands are sent and
the feedback received, this gives us a measure of events occurring per second

46

are not identical although close. Since we have to keep in mind that the force being
played is actually a low pass filtered version Of the velocity tracking errors, the only
effect this has is smoothing the forces out as seen in Fig.4.21, which gives the tracking
error and the force played which simply a filtered signal.

Actually the experiments done with Hong Kong were repeated several times to
figure out if the system is affected by the time of day or the network load. To study
this we present Fig.4.22. In this figure, which shows the same data as in Fig.4.18, we
observe the same tracking and Obstacle avoidance behavior as all the previous cases.
As for Fig.4.23 we see the similarity between the desired velocity and the sum of the
force with the actual velocity. Fig.4.24 simply shows the tracking error and the actual
force played just to illustrate the filtering process.

From all the experiments in this section we can conclude that the system’s behavior
is consistent and similar for all the testing that was done regardless of the time or

the network load. In addition, the system is stable under control over the Internet.

4.2.3 Experimental Conclusions

Horn all the experiments done and discussed previously we can conclude several things
regarding our approach. The most important is that the system is consistent and
behaves as expected. It is clear that the system works similarly for all cases. Meaning,
whether you have constant time delay, random delay, control over the Internet or
even no delay, the system is having the same performance. This is a very important
advantage since the system is immune to time delay and its variations. In addition,
it is Obvious that the system is stable and that the actual velocities track the desired
ones as long as there are no obstacles. Moreover, the system is executing the Obstacle
avoidance similarly for all the cases. Again this stability and Obstacle avoidance apply
in either scenario. Moreover, all of these properties apply irrespective of the path or

trajectory taken since in all the experiments the Operator had complete freedom of

47

 

 

 

 

 

150

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

. w o m . o . o . ml 0 W o
“:4 2.3 if...“ a. W. m .
b_oo_o> icozmuom “.9300 b_8_o> > “.2300 Eoo_o> > 332 8.0”. > 0.0830 2 8535

 

 

 

 

 

 

 

 

150

 

 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

  

.................... m m .
....... ...............m w .. ..w
p O p h 0 h n 0
o o 0 O
m m w s .o_. w. m m W
00:08» m. 05:. b.00.0> X U938 >u_nuo_o> X 333

0.03.0.8 2 0053.0

Figure 4.18: The behavior of the system during the control from Hong Kong.
48

 

...................

 

....................

 

 

 

 

1

000
500..H...

0
_5oo.....u

000

 

.....................

.....................

 

 

 

1

3503 m_ OEC.

 

  

 

 

 

>goo::/>,Do:mmfi_

this.“

 

 

 

 

 

 

m

ammo

$0...

>goozzzvauogm05_

 

uuuuuuuuuuuuuuuuuuuu

   

 

 

 

$03

.
2.8.; > 3224

 

....................

....................

 

 

 

 

§8_o> x 332

 

 

 

 

 

 

 

 

 

 

 

 

 

...........

....................

 

 

 

.
m 4| 1

28.80 2 mosaic

 

 

 

 

 

 

150 200

100

 

Figure 4.19: The behavior of the system during the control from Hong Kong.

49

 

 

 

 

  
  

 

 

 

 

 

 

 

 

 

 

 

 

 

w . w o H W o . m . W
mm mmmm swam mmmmomm
b_oo_m> > “5ng0 090n— > + b_oo_o> > 323‘ >u_oo_o> > 3:80

 

 

 

 

200

 

150

 

 

 

 

 

 

 

 

 

 

 

 

mmowm mass“ 0 s s m mosaic

b_00_0> x 8:80 850nm x + b_8_0> x _M3u0< >300—0) x 8.:WOD 0050K x ... >u-nvO_Q> x _N3u0<

 

 

 

 

Figure 4.20: Comparison between the desired velocities and the sum of the actual
50

velocities and the force felt.

X Component, Normal Operation From Hong Kong Y Component, Normal Operation From Hong Kong
- - Y 400 , -

‘7

 

 

8

§

Tracking Error
I
8
Tracking Error
0

 

 

 

 

 

 

 

O

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

    

 

 

 

 

 

 

 

 

.40 —200
—60 ‘ i —400
100 150 200
10 200
3 0 100
s-.. . ,
§ :
£3.20 -100 ;
430 -200 ~ ;
0 50 100 150 200
400 r 300 T
b a. 2m .....................................
g 2°° g
L” I.“ 1m. .................. T
O) O)
.5 0 .5
a g .
...-200 l._1oo ...................................
—400 _200 A
o 50 100 150 200 0 so 100 150 200
- 150 -
B _ . 3100
§ § 5,, .............................
§ §
.2 ,2 o ....... _ .......
-50 L L L __50 L L L
0 50 100 150 200 0 50 100 150 200

Figure 4.21: Comparison between tracking error and force felt for the two Hong Kong
experiments.

51

 

 

 

 

 

 

 

 

 

 

 

 

 

.. w o
0

m m a. m

_
b_8_o> .mcozsom 8:30
.w.
s .................... .m.
................... . w

..m

x 0

mm:

mucooom m_ 05:.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

  

$8.; > 8:89

use“. > + b_8_o> > 322

150
150

 

 

.....................

 

 

 

 

 

 

 

 

 

 

...:m

b_8_o> x 2580 8.5m x + a_oo_o> x 322

2

 

 

wwm

 

 

 

 

 

 

was.

M...

20880 2 8:220

 

 

 

 

 

0

somuMnO 2 35320

52

100

Figure 4.22: The behavior of the system during Operation from Hong Kong.

 

 

200 t i 1 00

O

 

-1 00

lo
0
O

 

Desired X Velocity
Desired Y Velocity

 

 

 

 

 

s

 

50 100 150

 

 

 

—L
o
o O

..........................

,t, .I.
O O
O O

..........................

 

 

 

 

 

Actual X Velocity + X Force

 

 

Actual Y Velocity + Y Force

 

 

l:
O
O

i

 

50 1 00 150 50 100 1 50

0

Figure 4.23: Comparison between the desired velocities and the sum Of the actual
velocities and the force felt.

 

 

 

 

 

 

 

 

 

300 . 200 .
x >-
E 200 .E 100
g 100 E
LU UJ 0
O! 03
.E 0 .5
x x
E 100 E ‘1°°
l— ‘ l-
-200 —200 ‘ L
0 O 50 100 150

 

 

 

 

Force Applied in X
Force Applied in Y

 

 

 

 

 

 

 

0 50 100 150 0 50 100 150

Figure 4.24: Comparison between tracking error and force felt for teleoperation from
Hong Kong.

53

movement. A few notes about the experiments; first, we note that in some cases
the actual velocities do not track the desired ones even if the Obstacles are far, and
therefore a force was generated. This tracking error can be due to several things; such
as, a slope, a load or simply robot internal controller tracking errors. Although this
is very rare but we have to note that a force can be felt even if there are no Obstacle,
and this is to our advantage since we want to feel the behavior of the robot and not
only the Obstacles. We want to know if the robot is going as fast as we want it, even
if there are no Objects in the way. Another thing, is that these experiments are the

only ones we know of where a mobile robot is controlled over the Internet with force

feedback in real-time.

54

CHAPTER 5
CONCLUSION AND FUTURE WORK

In this chapter we conclude our work, giving a summary of our results. Then we
discuss additional improvements that can be made On our approach, and what future

work will be done.

5.1 Conclusion

TeleOperation is a very attractive and quickly growing research field. But this tech-
nology faces a lot of difficulties. The most important one, which will always exist, is
time delay. This delay is caused by several things, most Obviously signal propagation
and hardware speed. At first thought this might be seen as a trivial issue not affecting
the system performance. But it was shown that this delay would render the system
unstable, especially when force feedback, which improves the Operator’s interaction
with the environment, is used.

In this study we presented a new event-based control method, which can be used
in any mobile robot, that solves the problem of delay. All the previous studies have
certain limitations on the delay; for example, constant or having an upper bound. In
our case nO assumptions are made regarding the delay. This approach is immune to
any kind of time delay, no upper bound is assumed and the delay can be random or
fixed; therefore, it is ideal for Internet applications. The approach was demonstrated
over the Internet, where no previous work was done having real-time control with
force feedback. It was found that this method results in a stable and synchronized
system. The whole method is based on having an event 3 as our reference and not
time. Our implementation, does not only work over the Internet but also over any
communication network, and can also be used in tele—Operation with no feedback.

In addition, we came up with a new force generation method, where the force felt

55

is actually a function of the distance to Obstacles in the environment of the robot.
This means the Operator would be able to deduce the position of Objects in the
robot’s vicinity simply by moving around and feeling the force. This way we get force
feedback with Obstacle avoidance built in.

We demonstrated experimentally and theoretically that the system obtained is
asymptotically stable. Not only that but also that the system was safe due to the
built in obstacle avoidance. In addition, the system is reliable and robust, since a
disconnection does not affect the system. And the Operation can be resumed once a
connection is reestablished. Another advantage to our system and approach is that
it decreases the task execution time by giving the Operator a better feeling about the

surrounding environment.

5.2 Future Work

Concerning the future work and the improvements on the system, there are many.
First we would like to work on synchronizing the visual feedback with the force
feedback so that all the components of the system would be at the same event during
any time. Not much work have been done in that aspect, where several feedback
forms are used in the system and all being synchronized. As an extension of the
work presented here; is using the event, not only to control, but also to synchronize
different parts Of the system. In addition we would like to improve the speed of
visual feedback from 1 frame per second to about 4 frames per second. Second, the
system should be able to monitor the delay in real time and change some of the
control and feedback parameters based on that so we have a faster response, and
reduce the task execution time. Another interest, is to include both tele-Operation
and local control; meaning, the ability of the robot to track a certain predefined path
but when unexpected events happen the Operator would intervene to alter the path.

Then the Operator can release the robot to continue with the previous path tracking.

56

In addition, auto-tracking of certain specified Object in real time is of interest. As
for the filtering that is done from the error to the force felt, we are in the process
Of develOping a smarter filter which would give a faster feeling of the environment.
Maybe this filter could monitor the delay and based on that change the weight given
to the different terms. Another extension would be using event based control for

manipulators control over the Internet with force reflection.

57

BIBLIOGRAPHY

58

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

BIBLIOGRAPHY

D. Kwon, K. Y. Woo, H. S Cho, “Haptic Control of the Master Hand Con-
troller for a Microsurgical Telerobot System”, IEEE Int. Conf. on Robotics and
Automation, Vol. 3, pp. 1722-1727, May 1999.

M. Tanimoto, F. Arai, T. Fhkuda, and M. Negoro, “Force Display Method to
Improve Safety in Teleoperation System for Intravascular Neurosurgery”, IEEE

Int. Conf. on Robotics and Automation, Vol. 3, pp. 1728-1733, May 1999.

P. G. Backes, K. S. Tso, and G. K. Tharp, “Mars Pathfinder Mission Internet-
Based Operations Using WITS” , Workshop on Internet Robotics IEEE Int. Conf.
on Robotics and Automation, pp. 284-291, May 1999.

S. E. Everett, R. V. Dubey, “Model-Based Variable Position Mapping for Teler-
obotic Assistance in a Cylindrical Environment”, IEEE Int. Conf. on Robotics
and Automation, Vol. 3, pp. 2197-2202, May 1999.

R. Luo, W. Z. Lee, J. H. Chou, and H. T. Leong, “Tele-Control of Rapid Pro-
totyping Machine Via Internet for Automated Tele-Manufacturing”, IEEE Int.
Conf. on Robotics and Automation, Vol. 3, pp. 2203—2208, May 1999.

L. Hsu, R. Costa, F. Lizarralde, J. Soares, “Passive Arm Based Dynamic Posi-
tioning System for Remotely Operated Underwater Vehicles”, IEEE Int. Conf.
on Robotics and Automation, Vol. 1, pp. 407-412, May 1999.

W. Kim, B. Hannaford, and A. Bejczy, “Force-Reflection and Shared Compli-
ant Control in Operating Telemanipulators with Time Delay”, IEEE Trans. on

Robotics and Automation, Vol. 8, April 1992.

G. Hirzinger, B. Brunner, J. Dietrich, and J. Heindl, “Sensor-Based Space
Robotics-ROTEX and Its Telerobotic Features”, IEEE Trans. on Robotics and
Automation, Vol. 9, No. 5, October 1993.

59

[9]

[10]

[11]

[12]

[13]

[14]

[15]

[15]

L. Conway, R. Volz, M. Walker, “Teleautonomous Systems: Projecting and Co-
ordinating Intelligent Action at a Distance”, IEEE Trans. on Robotics and Au-
tomation, Vol. 6, NO. 2, April 1990.

G. Hirzinger, K. Landzettel, Ch. Fagerer, “Telerobotics with Large Time Delays-
The ROTEX Experience”, IEEE/RSJ Int. Conf. on Intelligent Robots and Sys-
tems, Munich, Germany, pp. 571-578, September 1994.

A. Bejczy, W. Kim, S. Venema, “The Phantom Robot: Predictive Displays For
Teleoperation with Time Delay”, IEEE Int. Conf. on Robotics and Automation,
pp. 546—551, Cincinnati, May 1990.

M. mitsuishi, T. Hori, T. Nagao, “Predictive, Augmented and Transformed In-
formation Display for Time Delay Compensation in Tele-handling/ Machining”,
IEEE Int. Conf. on Robotics and Automation, pp 45-52, 1995.

,M. Otsuka, N. Matsumoto, T. Idogaki, K Kosuge, T. Itoh “Bilateral Tele-
manipulator System With Communication Time Delay Based on Force-Sum-
Driven Virtual Internal Models”, IEEE Int. Conf. on Robotics and Automation,
pp. 344-350, 1995.

D. Lawrence, “Stability and Transparency in Bilateral Teleoperation”, IEEE
Trans. on Robotics and Automation, Vol. 9, No. 5, October 1993.

G. Leung, B. Francis, J. Apkarian, “Bilateral Controller for Teleoperators With
Time Delay via p—Synthesis”, IEEE Trans. on Robotics and Automation, Vol 11,
NO. 1, February 1995.

T. Sheridan, “Space Teleoperation Through Time Delay: Review and Prognosis”,
IEEE 'fians. on Robotics and Automation, Vol 9, NO. 5, October 1993.

60

[17] J. Nilsson, B. Bernhardsson, “Stochastic Analysis and Control of Real-Time
Systems with Random Time Delays”, Automatica, Vol. 34, pp. 57-64, 1998.

[18] J. Klamka, “Observer for Linear Feedback Control of Systems With Distributed
delays in Controls and Output”, Systems and Control Letters, Vol. 1, 1982.

[19] K. Watanbe, M. Ito, “An Observer for Linear Feedback Control Laws of mul-
tivariable Systems with Multiple Delays in Control and Output”, Systems and
Control Letters, Vol. 1, No. 1, July 1981.

[20] R. Anderson, M. Spong, “Asymptotic Stability for Force Reflecting Teleoperators
with Time Delay”, The Int. Journal Of Robotics Research, Vol. 11, April 1992.

[21] R. Anderson, M. Spong, “Bilateral Control Of Teleoperators with Time Delay”,
IEEE Trans. on Automatic Control, Vol. 34, NO. 5, May 1989.

[22] G. Niemeyer, J. Slotine, “Stable Adaptive Teleoperation”, IEEE Journal of

Oceanic Engineering, Vol 16, NO. 1, January 1991.

[23] C. Lawn, B. Hannaford, “Performance testing of Passive Communication and
Control in Teleoperation with Time Delay”, Proc. IEEE Int. Conf. on Robotics
and Automation, Vol. 3, pp. 776-781, Atlanta, May 1993.

[24] B. Hannaford, W. Kim, “Force Reflection, Shared Control, And Time Delay in
Telemanipulation”, Proc. IEEE Int. Conf. on Systems, Man, and Cybernetics,
Cambridge, November 1989.

[25] T. J. Tarn, N. Xi, A. Bejczy, “Path-Based Approach to Integrated Planning and
Control for Robotic Systems”, Automatica, Vol. 32, NO. 12, pp. 1675-1687, 1996.

[26] J. Tan, N. Xi, T. J. Tarn, “Non-Time Based Tracking Controller for Mobile
Robots”, IEEE Canadian Conf. on Electrical and Computer Engineering, Ed-
monton Canada, 1999.

61

[27]

[28]

[29]

[30]

[31]

[32]

[33]

[34]

[35]

N. Xi, T. J. Tarn, “Action Synchronization and Control of Internet Based Teler-
obotic Systems”, IEEE Int. Conf. on Robotics and Automation, Vol.1, pp. 219-
224, Detroit, May 1999.

Y. Zheng, “Human-Robot Coordination for Moving Large Objects”, Workshop
Note, 1997 ICRA.

N. Hogan, “Multivariable Mechanics of The Neuromuscular System”, IEEE Eight

Annual Conference of the Engineering in Medicine and Biology Society, 1986.

T. Milner, “Human Operator Adaptation to Machine Instability”, Advances in
Robotics, Mechatronics, and Haptic Interfaces, DSC-Vol. 49, ASME 1993.

E. Todosiev, R. Ross, L. Summers, “Human Performance in Single and Two-Axis
Tracking Systems”, IEEE Trans. on Human Factors in Electronics, Vol. HFE—8,
No. 2, June 1967.

G. Bekey, H. Meissinger, R. Rose, “Mathematical Models of Human Operators
in Simple Two-Axis Manual Control Systems”, IEEE Trans. on Human Factors

in Electronics, September 1965.

H. Tan, M. Srinivasan, B. Eberman, B. Cheng, “Human Factors For the Design
of Force-Reflecting Haptic Interfaces”, Dynamic Systems and Control, DSC-Vol.
55-1, 1994.

E. Altman, T. Basar, R. Srikant, “Multi-User Rate-Based Flow Control with Ac-
tion Delays: A Team-Theoretic Approach”, Proceedings of the 36‘“ Conference

on Decision and Control, San Diego, December 1997.

E. Altman, T. Basar, “Multi-User Rate-Based Flow Control: Distributed Game-
theoretic Algorithms” , Proceedings of the 36‘“ Conference on Decision and Con-

trol, San Diego, December 1997.

62

[36]

[37]

[38]

[39]

[40]

[41]

[42]

[43]

[44]

U. Madhow, “Dynamic Congestion Control and Error Recovery Over a Hetero-
geneous Internet”, Proceedings Of the 36‘“ Conference on Decision and Control,

San Diego, December 1997.

R. Riedi, M. Course, V. Ribeiro, R. Baraniuk, “A Multifractal Wavelet Model
with Application to T CP Network Traffic”, IEEE Trans. on Information Theory
(Special Issue on Multiscale Signal Analysis and Modeling), pp. 992-1018, April
1999.

W. Leland, M. Taqqu, W. Willinger, D. Wilson, “On the Self-Similar Nature Of
Ethernet Traffic (Extended Version)”, IEEE/ACM Trans. on Networking, Vol.
2, NO. 1, February 1994.

K. Brady, T. J. Tarn, “Internet-Based Remote Teleoperation”, Proceedings of
the 1998 IEEE International Conference on Robotics and Automation, Leuven,
Belgium, May 1998.

D. Pai, “ACME, A Telerobotic Measurement Facility for Reality-Based Mod-
elling on the Internet”, IROS Workshop on Robots on the Web, Canada, 1998.

M. Stein, “Painting on the World Wide Web: The PumaPaint Project”, IROS
WorkshOp on Robots on the Web, Canada, 1998.

R. Simmons, “Xavier: An Autonomous Mobile Robot on the Web”, IROS Work-
shop on Robots on the Web, Canada, 1998.

N. Xi, “Event-Based Planning and Control for Robotic Systems”, Doctoral Dis-

sertation, Washington University, December 1993.

Nomad XRDEV Software Manual, Release 1.0, Nomadic Technologies, Inc., 1999.

63

"llllllllllllllllllllll