Search results

Title: Natural language based control and programming of robotic behaviors
Creator: Cheng, Yu (Graduate of Michigan State University)
Date: 2019
Collection: Electronic Theses & Dissertations
Description: "Robots have been transforming our daily lives by moving from controlled industrial lines to unstructured and dynamic environments such as home, offices, or outdoors working closely with human co-workers. Accordingly, there is an emerging and urgent need for human users to communicate with robots through natural language (NL) due to its convenience and expressibility, especially for the technically untrained people. Nevertheless, two fundamental problems remain unsolved for robots to working...
Show more"Robots have been transforming our daily lives by moving from controlled industrial lines to unstructured and dynamic environments such as home, offices, or outdoors working closely with human co-workers. Accordingly, there is an emerging and urgent need for human users to communicate with robots through natural language (NL) due to its convenience and expressibility, especially for the technically untrained people. Nevertheless, two fundamental problems remain unsolved for robots to working in such environments. On one hand, how to control robot behaviors in dynamic environments due to presence of people is still a daunting task. On the other hand, robot skills are usually preprogrammed while an application scenario may require a robot to perform new tasks. How to program a new skill to robots using NL on the fly also requires tremendous efforts. This dissertation tries to tackle these two problems in the framework of supervisory control. On the control aspect, it will be shown ideas drawn from dynamic discrete event systems can be used to model environmental dynamics and guarantee safety and stability of robot behaviors. Specifically, the procedures to build robot behavioral model and the criteria for model property checking will be presented. As there are enormous utterances in language with different abstraction level, a hierarchical framework is proposed to handle tasks lying in different logic depth. Behavior consistency and stability under hierarchy are discussed. On the programming aspect, a novel online programming via NL approach that formulate the problem in state space is presented. This method can be implemented on the fly without terminating the robot implementation. The advantage of such a method is that there is no need to laboriously labeling data for skill training, which is required by traditional offline training methods. In addition, integrated with the developed control framework, the newly programmed skills can also be applied to dynamic environments. In addition to the developed robot control approach that translates language instructions into symbolic representations to guide robot behaviors, a novel approach to transform NL instructions into scene representation is presented for robot behaviors guidance, such as robotic drawing, painting, etc. Instead of using a local object library or direct text-to-pixel mappings, the proposed approach utilizes knowledge retrieved from Internet image search engines, which helps to generate diverse and creative scenes. The proposed approach allows interactive tuning of the synthesized scene via NL. This helps to generate more complex and semantically meaningful scenes, and to correct training errors or bias. The success of robot behavior control and programming relies on correct estimation of task implementation status, which is comprised of robotic status and environmental status. Besides vision information to estimate environmental status, tactile information is heavily used to estimate robotic status. In this dissertation, correlation based approaches have been developed to detect slippage occurrence and slipping velocity, which provide grasp status to the high symbolic level and are used to control grasp force at lower continuous level. The proposed approaches can be used with different sensor signal type and are not limited to customized designs. The proposed NL based robot control and programming approaches in this dissertation can be applied to other robotic applications, and help to pave the way for flexible and safe human-robot collaboration."--Pages ii-iii.
Show less

Title: Modeling physical causality of action verbs for grounded language understanding
Creator: Gao, Qiaozi
Date: 2019
Collection: Electronic Theses & Dissertations
Description: Building systems that can understand and communicate through human natural language is one of the ultimate goals in AI. Decades of natural language processing research has been mainly focused on learning from large amounts of language corpora. However, human communication relies on a significant amount of unverbalized information, which is often referred as commonsense knowledge. This type of knowledge allows us to understand each other's intention, to connect language with concepts in the...
Show moreBuilding systems that can understand and communicate through human natural language is one of the ultimate goals in AI. Decades of natural language processing research has been mainly focused on learning from large amounts of language corpora. However, human communication relies on a significant amount of unverbalized information, which is often referred as commonsense knowledge. This type of knowledge allows us to understand each other's intention, to connect language with concepts in the world, and to make inference based on what we hear or read. Commonsense knowledge is generally shared among cognitive capable individuals, thus it is rarely stated in human language. This makes it very difficult for artificial agents to acquire commonsense knowledge from language corpora. To address this problem, this dissertation investigates the acquisition of commonsense knowledge, especially knowledge related to basic actions upon the physical world and how that influences language processing and grounding.Linguistics studies have shown that action verbs often denote some change of state (CoS) as the result of an action. For example, the result of "slice a pizza" is that the state of the object (pizza) changes from one big piece to several smaller pieces. However, the causality of action verbs and its potential connection with the physical world has not been systematically explored. Artificial agents often do not have this kind of basic commonsense causality knowledge, which makes it difficult for these agents to work with humans and to reason, learn, and perform actions.To address this problem, this dissertation models dimensions of physical causality associated with common action verbs. Based on such modeling, several approaches are developed to incorporate causality knowledge to language grounding, visual causality reasoning, and commonsense story comprehension.
Show less

Title: Formalization and verification of property specification patterns
Creator: Bryndin, Dmitriy
Date: 2010
Collection: Electronic Theses & Dissertations
Description: Finite-state verification (FSV) techniques are intended for proving properties of software systems. Although significant progress has been made in the last decade automating FSV techniques, the adoption of these techniques by software developers is low. The Specification Pattern System (SPS) is intended to assist users in creating such specifications. It identifies common specification patterns and indicates how to translate the patterns into a variety of different specification languages....
Show moreFinite-state verification (FSV) techniques are intended for proving properties of software systems. Although significant progress has been made in the last decade automating FSV techniques, the adoption of these techniques by software developers is low. The Specification Pattern System (SPS) is intended to assist users in creating such specifications. It identifies common specification patterns and indicates how to translate the patterns into a variety of different specification languages. However, the patterns in the SPS are defined informally and their translations are not verified. This work discusses the informal nature of these definitions, proposes a formalization for them and provides formal proofs for the translation of patterns to Linear Temporal Logic.
Show less

Title: Semantic role labeling of implicit arguments for nominal predicates
Creator: Gerber, Matthew Steven
Date: 2011
Collection: Electronic Theses & Dissertations
Description: Natural language is routinely used to express the occurrence of an event and existence of entities that participate in the event. The entities involved are not haphazardly related to the event; rather, they play specific roles in the event and relate to each other in systematic ways with respect to the event. This basic semantic scaffolding permits construction of the rich event descriptions encountered in spoken and written language. Semantic role labeling (SRL) is a method of automatically...
Show moreNatural language is routinely used to express the occurrence of an event and existence of entities that participate in the event. The entities involved are not haphazardly related to the event; rather, they play specific roles in the event and relate to each other in systematic ways with respect to the event. This basic semantic scaffolding permits construction of the rich event descriptions encountered in spoken and written language. Semantic role labeling (SRL) is a method of automatically identifying events, their participants, and the existing relations within textual expressions of language. Traditionally, SRL research has focused on the analysis of verbs due to their strong connection with event descriptions. In contrast, this dissertation focuses on emerging topics in noun-based (or nominal) SRL.One key difference between verbal and nominal SRL is that nominal event descriptions often lack participating entities in the words that immediately surround the predicate (i.e., the word denoting an event). Participants (or arguments) found at longer distances in the text are referred to as implicit. Implicit arguments are relatively uncommon for verbal predicates, which typically require their arguments to appear in the immediate vicinity. In contrast, implicit arguments are quite common for nominal predicates. Previous research has not systematically investigated implicit argumentation, whether for verbal or nominal predicates. This dissertation shows that implicit argumentation presents a significant challenge to nominal SRL systems: after introducing implicit argumentation into the evaluation, the state-of-the-art nominal SRL system presented in this dissertation suffers a performance degradation of more than 8%.Motivated by these observations, this dissertation focuses specifically on implicit argumentation in nominal SRL. Experiments in this dissertation show that the aforementioned performance degradation can be reduced by a discriminative classifier capable of filtering out nominals whose arguments are implicit. The approach improves performance substantially for many frequent predicates - an encouraging result, but one that leaves much to be desired. In particular, the filter-based nominal SRL system makes no attempt to identify implicit arguments, despite the fact that they exist in nearly all textual discourses.As a first step toward the goal of identifying implicit arguments, this dissertation presents a manually annotated corpus in which nominal predicates have been linked to implicit arguments within the containing documents. This corpus has a number of unique properties that distinguish it from preexisting resources, of which few address implicit arguments directly. Analysis of this corpus shows that implicit arguments are frequent and often occur within a few sentences of the nominal predicate.Using the implicit argument corpus, this dissertation develops and evaluates a novel model capable of recovering implicit arguments. The model relies on a variety of information sources that have not been used in prior SRL research. The relative importance of these information sources is assessed and particularly troubling error types are discussed. This model is an important step forward because it unifies work on traditional verbal and nominal SRL systems. The model extracts semantic structures that cannot be recovered by applying the systems independently.Building on the implicit argument model, this dissertation then develops a preliminary joint model of implicit arguments. The joint model is motivated by the fact that semantic arguments do not exist independently of each other. The presence of a particular argument can promote or inhibit the presence of another. Argument dependency is modeled by using the TextRunner information extraction system to gather general purpose knowledge from millions of Internet webpages. Results for the joint model are mixed; however, a number of interesting insights are drawn from the study.
Show less

Title: Mental models and home virtual assistants (HVAs : a single turn-taking is not enough to have a human-like conversation
Creator: Cho, Janghee
Date: 2018
Collection: Electronic Theses & Dissertations
Description: As virtual assistants become more intelligent, these applications are dispersed in our everyday life. However, virtual assistants are still used in very routine tasks (Wingfield, 2018). People are expected to speak to virtual assistants in a conversational manner; nevertheless, there has been little research looking at people's mental models for what kinds of interactions they think the devices are capable of. I conducted a qualitative study to describe people’s strategies to seek a wide...
Show moreAs virtual assistants become more intelligent, these applications are dispersed in our everyday life. However, virtual assistants are still used in very routine tasks (Wingfield, 2018). People are expected to speak to virtual assistants in a conversational manner; nevertheless, there has been little research looking at people's mental models for what kinds of interactions they think the devices are capable of. I conducted a qualitative study to describe people’s strategies to seek a wide range of information when interacting with Google Home, which is a type of Home Virtual Assistant (HVA), and how their understandings of HVAs may change their interaction. I found that people believed applying human-to-human communication skills may facilitate a conversation with Google Home (i.e., common-sense models). Also, people expected Google Home performs like Google search or an artificial intelligence system that is able to become more intelligent if people provide more information (i.e., machine-oriented models). I present implications for the design of HVAs to encourage inexperienced people to be involved in the interaction with HVAs based on these results.
Show less

Title: On interleaving syntax and semantics in parsing
Creator: Ra, Dongyul
Date: 1989
Collection: Electronic Theses & Dissertations

Title: Discourse representation theory and pronouns
Creator: Lee, Daehee
Date: 1996
Collection: Electronic Theses & Dissertations

Title: Natural language inference : from textual entailment to conversation entailment
Creator: Zhang, Chen
Date: 2010
Collection: Electronic Theses & Dissertations

Title: Grounded language processing for action understanding and justification
Creator: Yang, Shaohua (Graduate of Michigan State University)
Date: 2019
Collection: Electronic Theses & Dissertations
Description: Recent years have witnessed an increasing interest on cognitive robots entering into our life. In order to reason, collaborate and communicate with human in the shared physical world, the agents need to understand the meaning of human language, especially the actions, and connect them to the physical world. Furthermore, to make the communication more transparent and trustworthy, the agents should have human-like action justification ability to explain their decision-making behaviors. The goal...
Show moreRecent years have witnessed an increasing interest on cognitive robots entering into our life. In order to reason, collaborate and communicate with human in the shared physical world, the agents need to understand the meaning of human language, especially the actions, and connect them to the physical world. Furthermore, to make the communication more transparent and trustworthy, the agents should have human-like action justification ability to explain their decision-making behaviors. The goal of this dissertation is to develop approaches that learns to understand actions in the perceived world through language communication. Towards this goal, we study three related problems. Semantic role labeling captures semantic roles (or participants) such as agent, patient and theme associated with verbs from text. While it provides important intermediate semantic representations for many traditional NLP tasks, it does not capture grounded semantics with which an artificial agent can reason, learn, and perform the actions. We utilize semantic role labeling to connect the visual semantics with linguistic semantics. On one hand, this structured semantic representation can help extend the traditional visual scene understanding instead of simply object recognition and relation detection, which is important for achieving human robot collaboration tasks. On the other hand, due to the shared common ground, not every language instruction is fully specified explicitly. We proposed to not only ground explicit semantic roles, but also implicit roles which is hidden during the communication. Our empirical results have shown that by incorporate the semantic information, we achieve better grounding performance, and also a better semantic representation of the visual world. Another challenge for an agent is to explain to human why it recognizes what's going on as a certain action. With the recent advance of deep learning, A lot of works have shown to be very effective on action recognition. But most of them function like black-box models and have no interpretations of the decisions which are given. To enable collaboration and communication between humans and agents, we developed a generative conditional variational autoencoder (CVAE) approach which allows the agent to learn to acquire commonsense evidence for action justification. Our empirical results have shown that, compared to a typical attention-based model, CVAE has a significantly higher explanation ability in terms of identifying correct commonsense evidence to justify perceived actions. The experiment on communication grounding further shows that the commonsense evidence identified by CVAE can be communicated to humans to achieve a significantly higher common ground between humans and agents. The third problem combines the action grounding with action justification in the context of visual commonsense reasoning. Humans have tremendous visual commonsense knowledge to answer the question and justify the rationale, but the agent does not. On one hand, this process requires the agent to jointly ground both the answers and rationales to the images. On the other hand, it also requires the agent to learn the relation between the answer and the rationale. We propose a deep factorized model to have a better understanding of the relations between the image, question, answer and rationale. Our empirical results have shown that the proposed model outperforms strong baselines in the overall performance. By explicitly modeling factors of language grounding and commonsense reasoning, the proposed model provides a better understanding of effects of these factors on grounded action justification.
Show less

Title: WWN : language acquisition and generalization using association
Creator: Miyan, Kajal
Date: 2011
Collection: Electronic Theses & Dissertations
Description: Based on some recent advances in understanding and modeling cortical process- ing for space and time, we propose a developmental, general-purpose model for language acquisition using multiple motor areas. The thesis presents two main ideas: a) early language acquisition is a grounded and incremental process, i.e., the network learns as it performs in the real world b) language is a complex perceptual, cognitive and motor skill that can be acquired through associative learning and skill...
Show moreBased on some recent advances in understanding and modeling cortical process- ing for space and time, we propose a developmental, general-purpose model for language acquisition using multiple motor areas. The thesis presents two main ideas: a) early language acquisition is a grounded and incremental process, i.e., the network learns as it performs in the real world b) language is a complex perceptual, cognitive and motor skill that can be acquired through associative learning and skill transfer principles. The network architecture is informed by the existing neuroanatomic studies and the associative learning literature in psychol- ogy. Through the ventral pathway, the "what" motor learns, abstracts and feeds back (as recurrent top-down context) information that is related to the meaning of the text. Via the dorsal pathway, the "where/how" motor learns, abstracts and feeds back (as top-down context) information that relates to the spatial informa- tion of text, e.g., where is the text on a page. This is a major departure from the traditional symbolic and connectionist approaches to natural language processing (NLP) -- the nature of the motor areas, i.e., actions or abstract meanings, play the role of "state hubs" in language acquisition and understanding. The "hubs" correspond to multiple concepts that form the state of the current context. As any human communicable concept can be either verbally stated (what) or demonstrated through actions (how), this model seems to be the first general purpose develop- mental model for general language acquisition, although the size of our experiments is still limited. Furthermore, unlike traditional NLP approaches, syntax is a special case of actions. The major novelty in our language acquisition is the ability to gen- eralize, going beyond a probability framework, by simulating the primary, secondary and higher order associations observed in animal learning through the generalization of internal distributed representations. A basic architecture that enables such a gen- eralization is the overall distributed representation: not only a retina image but also an array of muscles is considered high-dimensional images. An emergent internal dis- tributed representation is critical for going beyond experience to enable three types of generalization: member-to-class, subclass-to-superclass, member-to-member, and relation-specification. In our cortex inspired model, syntax and semantics are not treated differently, but as emergent behaviors that arise from grounded real-time experience.
Show less

MSU Libraries

Digital Repository

Enabled Filters

MSU Libraries

Digital Repository

You are here

Search results