Using Semantic Structure of the Data and Knowledge in Question Answering Systems

Understanding and reasoning over natural language is one of the most crucial and long-standing challenges in Artificial Intelligence (AI). Question answering (QA) is the task of automatically answering questions posed by humans in a natural language form. It is an important criterion to evaluate the language understanding and reasoning capabilities of AI systems. Though machine learning systems on Question Answering (QA) have shown tremendous success in language understanding, they still suffer from a lack of interpretability and generalizability, in particular, when complex reasoning is required to answer the questions. In this dissertation, we aim to build novel QA architectures that answer complex questions using the explicit relational structure of the raw data, that is, text and image, and exploiting external knowledge. We investigate a variety of problems, including answering natural language questions when the answer can be found in multiple modalities, including 1) Textual documents (Document-level QA), 2) Images (Cross-Modality QA), 3) Knowledge graphs (Commonsense QA) and, 4) Combination of text and knowledge graphs. First, for Document-level QA, we develop a new technique, Semantic Role Labeling Graph Reasoning Network (SRLGRN), via which the explicit semantic structure of multiple textual documents is used. In particular, based on semantic role labeling, we form a multi-relational graph that jointly learns to find cross-paragraph reasoning paths and answers multi-hop reasoning questions. Second, for the type of QA that requires causal reasoning over textual documents, we propose a new technique, Relational Gating Network (RGN), that jointly learns to extract the entities and their relations to help highlight the important entity chains and find how those affect each other. Third, for the type of questions that require complex reasoning over language and vision modalities (Cross-Modality QA), we propose a new technique, Cross-Modality Relevance (CMR). This technique considers the relevance between textual tokens and visual objects by aligning the two modalities. Fourth, for answering questions based on given Knowledge Graphs (KG), we propose a new technique, Dynamic Relevance Graph Network (DRGN). This technique is based on a graph neural network and re-scales the importance of the neighbor nodes in the graph dynamically by training a relevance matrix. The new neighborhoods trained by relevance help fill in the knowledge gaps in the KG for more effective knowledge-based reasoning. Fifth, for answering questions using a combination of textual documents and an external knowledge graph, we propose a new technique, Multi-hop Reasoning Network over Relevant Commonsense Subgraphs (MRRG). MRRG technique extracts the most relevant KG subgraph for each question and document and uses that subgraph combined with the textual content and question representations for answering complex questions. We improve the performance, interpretability, and generalizability of various challenging QA benchmarks based on different modalities. Our ideas have proven to be effective in multi-hop reasoning, causal reasoning, cross-modality reasoning, and knowledge-based reasoning for question answering.