"Robots have been transforming our daily lives by moving from controlled industrial lines to unstructured and dynamic environments such as home, offices, or outdoors working closely with human co-workers. Accordingly, there is an emerging and urgent need for human users to communicate with robots through natural language (NL) due to its convenience and expressibility, especially for the technically untrained people. Nevertheless, two fundamental problems remain unsolved for robots to working in such environments. On one hand, how to control robot behaviors in dynamic environments due to presence of people is still a daunting task. On the other hand, robot skills are usually preprogrammed while an application scenario may require a robot to perform new tasks. How to program a new skill to robots using NL on the fly also requires tremendous efforts. This dissertation tries to tackle these two problems in the framework of supervisory control. On the control aspect, it will be shown ideas drawn from dynamic discrete event systems can be used to model environmental dynamics and guarantee safety and stability of robot behaviors. Specifically, the procedures to build robot behavioral model and the criteria for model property checking will be presented. As there are enormous utterances in language with different abstraction level, a hierarchical framework is proposed to handle tasks lying in different logic depth. Behavior consistency and stability under hierarchy are discussed. On the programming aspect, a novel online programming via NL approach that formulate the problem in state space is presented. This method can be implemented on the fly without terminating the robot implementation. The advantage of such a method is that there is no need to laboriously labeling data for skill training, which is required by traditional offline training methods. In addition, integrated with the developed control framework, the newly programmed skills can also be applied to dynamic environments. In addition to the developed robot control approach that translates language instructions into symbolic representations to guide robot behaviors, a novel approach to transform NL instructions into scene representation is presented for robot behaviors guidance, such as robotic drawing, painting, etc. Instead of using a local object library or direct text-to-pixel mappings, the proposed approach utilizes knowledge retrieved from Internet image search engines, which helps to generate diverse and creative scenes. The proposed approach allows interactive tuning of the synthesized scene via NL. This helps to generate more complex and semantically meaningful scenes, and to correct training errors or bias. The success of robot behavior control and programming relies on correct estimation of task implementation status, which is comprised of robotic status and environmental status. Besides vision information to estimate environmental status, tactile information is heavily used to estimate robotic status. In this dissertation, correlation based approaches have been developed to detect slippage occurrence and slipping velocity, which provide grasp status to the high symbolic level and are used to control grasp force at lower continuous level. The proposed approaches can be used with different sensor signal type and are not limited to customized designs. The proposed NL based robot control and programming approaches in this dissertation can be applied to other robotic applications, and help to pave the way for flexible and safe human-robot collaboration."--Pages ii-iii.
Read
