Reasoning over procedural text, which encompasses texts such as recipes, manuals, and 'how to' tutorials, presents formidable challenges due to the dynamic nature of the world it describes. Reasoning over procedural text has many challenges. These challenges are embodied in tasks such as 1) tracking entities and their status changes~(entity tracking) and 2) summarization of the process(procedural summarization).This thesis aims to enhance the representation and reasoning over textual procedures by harnessing semantic structures in the input text and imposing constraints on the models' output. It delves into using semantic structures derived from the text, including relationships between actions and objects, semantic parsing of instructions, and the sequential structure of actions. Additionally, the thesis investigates the integration of structural and semantic constraints within neural models, resulting in coherent and consistent outputs that align with external knowledge. The thesis contributes significantly to three main areas: Entity tracking, Procedural Abstraction, and the Integration of constraints in deep learning.In the entity tracking task, we have made four primary contributions: 1) Developed a novel architecture for encoding event flow in pretrained language models. 2) Enabled seamless transfer learning from diverse corpora through task reformulation. 3) Enhanced language models by incorporating knowledge from semantic parsers and leveraging ontological abstraction of actions. 4) Created a new evaluation scheme considering fine-grained semantics in tracking entities.Regarding procedural summarization, the thesis proposes a model for an explicit latent space for the procedure that is indirectly supervised to ensure the summary's action order corresponds to the order of events in the multi-modal instructions.sIn the realm of integrating domain knowledge with deep neural networks, the thesis makes two significant contributions, 1) it contributes to the development of a generic framework that facilitates the incorporation of first-order logical constraints in neural models, and 2) it creates a new benchmark for evaluating constraint integration methods across five categories of tasks. This benchmark introduces novel evaluation criteria and offers valuable insights into the effectiveness of constraint integration methods across various tasks.
Read
