Building Reinforcement Learning and Normative Models in the Cloud

Mclean Hospital, Belmont MA

Investigators

Linked publications & trials

Abstract

The parent proposal aims to address a critical need in the field of precision psychiatry by identifying a complex behavior, such as Reinforcement Learning based Decision Making (RLDM), which is impaired across various psychiatric disorders and adopting a computational framework to explain heterogeneity at an individual level. By building normative models of RLDM constructs and charting heterogeneity at the individual level, the proposal aims to advance precision medicine. The main goals of this proposal are to (1) parse RLDM sub- processes into mathematically-defined parameters in a large sample using a diverse set of tasks; (2) assess test-retest reliability of these parameters; and finally (3) build normative models of the parameters and chart the heterogeneity at the level of the individual. We will reach these goals by collecting behavioral data from a diverse set of tasks in a large community sample (n=1000) and 500 of these participants will complete the tasks a second time within two weeks to enable us to assess test-retest reliability of the computationally-derived RLDM parameters. The framework that was proposed in the parent R21 involved deploying 6 RLDM tasks online and collecting data using one of the cloud/cloud-like services such as AWS, Pavlovia or testmybrain. We were then planning on downloading all the behavioral data and running our RLDM and normative models in our local compute cluster, due to limited funds available in the parent R21 to use cloud computing. In this proposal, we aim to conduct the entirety of our project on the cloud with the funds provided by this supplement. The entire parent project could benefit tremendously from having access to the cloud resources – from online tasks deployment, data collection and automated large scale computationally intensive data analyses. Running RLDM models and creating normative charts are computationally intensive and require significant resources. Our plan was to collect data from six tasks, run three to five RLDM models on each task, estimate RLDM parameters and develop normative models of the eight most stable RLDM parameters in 1000 participants. However, with the use of affordable cloud computing through this supplement, we will be able to not only vastly reduce computational time (which would be very slow on our computing cluster that is a shared resource across the Hospital), but this will also give us an opportunity to explore complex RLDM models and test novel estimation techniques. Additionally, by reducing the burden on local compute clusters and costs (budgeted in our parent grant), we might be able to increase our originally proposed sample size, thereby enhancing the robustness of the normative models with data from a larger sample size. With the entire project on the cloud, there will be seamless integration from data collection to data analyses and statistical interpretation, which will improve the overall efficiency of the project.