A computational framework for quality assessment and curation of models at scale

University Of Pittsburgh At Pittsburgh, Pittsburgh PA

Investigators

Abstract

Project Summary Model development is a common step of a research pipeline in many domains, including biology. However, most often there is no systematic approach to annotate models during their development stage, which can complicate their evaluation and reuse in the following stages of the research pipeline. Computational and systems biologists create models of various biological systems, a procedure that, even when partially automated, often takes months, or even years, to achieve a trusted model. The verification step, assessing whether model’s components are correct by finding support for all its elements and interactions, and the validation step, evaluation of model behavior against experimental observations and data, usually occur iteratively with model expansion before the model can be used to make predictions or explanations. A standardized procedure for model verification and validation including well-defined requirements for model annotations at these stages is still lacking. Once published, models are usually added to one of the existing model repositories and databases. Most of the steps described above are done manually. On the other hand, recent advances in natural language processing (NLP) and large language models (LLMs) have enabled processing thousands of papers and the rapid extraction of entity, event, and causal relationship information. Several tools have been developed to collect this information and ground it to standard database identifiers. These text mining resources can be utilized when automatically verifying and validating the existing or newly built models. Our goal is to standardize and fully automate the iterative process of assembly, documentation, verification, and validation of models of cell signaling. This will maximize the model accuracy and reuse potential and their long-term value, while minimizing the risk of models becoming obsolete. The objective of this research project is to develop an architecture that will allow researchers to automatically assess the quality, annotate and expand their models, utilizing available literature and model databases. These objectives will be accomplished by carrying out three Specific Aims: 1. development of a framework architecture to automatically verify, validate, document, and increase the completeness of models of cellular signaling, 2. development and implementation of methods to automatically identify and resolve contradictions in available information, or suggest conflict resolution, and 3. development of translators and interfaces to ensure compatibility of our methods with model formats used by existing knowledge bases and databases, and to streamline communication between our architecture and these online sources. At the conclusion of this project, we will have designed, implemented, tested, and made publicly available this framework that will enable model curation at scale with automated verification, validation, documentation, and improvement of both new models during their development stage, as well as existing models. Outcomes of this work will contribute to the substantial increase in the quality and reuse of models and aid computational and systems biology researchers in assembling or selecting comprehensively annotated and trusted models.