Analyzing the SSA Disability Evaluation Process

Clinical Center

Investigators

Linked publications & trials

Abstract

Analytics (Objective 1) The SSA adjudication process is dynamic, involving a complex sequence of decisions by several offices within SSA as well as the decisions and resources of the claimants themselves. NIH undertook an overarching project, the Adjudication 1 project, to comprehensively model this process. Project goals are to: 1. Develop analytical tools to analyze various aspects of the adjudication process in terms of accuracy, consistency, and timeliness; 2. Develop tools to predict how the system responds to external shocks; 3. Develop methods to analyze data taking into account the multi-stage application process in which data are collected; 4. Develop tools to assist with the disability determination process; 5. Quantify the extent to which SSA can adjust the system to respond to changes; 6. Derive useful statistics to monitor and adjust program performance, based on important outcomes measures (accuracy, timeliness, consistency). We approach these goals by working on small empirical projects that, over time, will build a collection of tools. We have made noteworthy progress in several areas as described below. Case Status Change Model and Queuing Theory: The objective of this subproject of Adjudication 1 is to develop methods to analyze system timeliness, measure processing times, and derive optimal flow characteristics. Queues are used to analyze SSA system operating characteristics and optimize performance, and the case status change model is used to analyze time spent in the system. Our work in the area of system timeliness took on two separate but complementary directions. To study system delays, we built a queuing model for the adjudication system that allows the user to obtain system performance statistics. Using the model, analysis of a very complex system is now possible with a rather low complexity requirement. However, this model cannot capture the fact that some parts of the system may have to wait for other parts to finish processing before it can proceed with delegated work. Addressing the separation between wait times and processing time is our second direction of inquiry. For this purpose, we developed a batch processing model. We introduced a new queue that allows multiple tasks to be switched back and forth within a batch to make distribution estimates possible. To date, we have scripted code to obtain the needed transition probabilities, determined the rate at which jobs enter the system, completed the queue code, and estimated the distributions of processing/waiting times by using three techniques. Additional data are required to further develop the models to yield better estimates when detecting batch exits tasks, classifying the types of cases, observing linear combinations of tasks, and implementing non-parametric methods. Case review nominator: The SSA evaluates adjudicated claims for benefits administered by SSA. These reviews often include the evaluation of legal text for adherence to relevant policies, regulations and laws. As a result of the wide varieties of language that might be used, reviews focused on particular issues are typically limited to samples that included entire populations without any ability to screen out unlikely cases. With assistance from SSA's staff, NIH developed an automated document classification algorithm to identify cases with specific issues. The algorithm used a set of labeled legal documents to build its predictive model. We implemented the automated document classification algorithm in Python, using Python NLTK and Scikit-learn for text processing, normalization, and feature extraction, as well as Python Scikit-Learn or Weka for feature selection and classification. Python was used with regular expressions to distinguish between documents containing text associated with the issue of interest and those that likely did not. This process was implemented successfully in June 2014. Data Mining Feasibility Study: This project placed a high-performance server inside the SSA firewall for the purpose of determining the software and hardware configuration that will allow us to extract information from decision files and medical data. With this server, we will be pursuing three areas of inquiry: methods of information retrieval (IR) for measuring relevance of documents, methods in natural language processing (NLP) for extracting medical records, and partial NLP methods for analyzing decision files. To date, we wrote the scraper program for document extraction and we began optical character recognition (OCR) of decision files, residual functional capacity (RFC) forms, and medical records using Omnipage software. Listings nominator: Under current SSA rules, the presence of a condition that meets criteria in the Listings of Impairments (or that is of equal severity) is considered sufficient to establish medical considerations for allowances. We are currently building a tool for SSA to quickly find relevant listings from the Listings of Impairments, given a description of the case using natural language processing and using a search engine to return the relevant Listing(s). To date, we pulled the Listings from SSA website and summarized them into 14 body system documents and 149 sub-function documents. For each document, keywords (queries) are generated to compare against claimant medical records. We are devising information retrieval methods to automatically generate queries and calculate the odds of relevance as a score that measures the distance between any given pair of query and medical record. The sub-function documents will be ranked by their relevance scores, ultimately yielding a list of the most potentially relevant Listings to be considered by the SSA examiner. CAT development (Objective 2) The NIH/RMD awarded a contract to the Boston University Health and Disability Research Institute (BU-HDRI) to develop a comprehensive set of tools to characterize the full continuum of individual capabilities (i.e. human function) relevant to work. This method uses Computer Adaptive Testing (CAT) coupled with Item Response Theory (IRT). In order to understand distinct factors influencing work, individual capabilities as well as workplace demands and critical features of the workplace environment must be captured. The work with Boston University encompasses CAT development to capture the person side of this interaction, in other words, the assessment of individual capabilities. The development of CAT tools (now referred to as the Functional Assessment Battery or FAB) is a sequential process; one step must be completed before advancing to the next step. The initial functional domains selected for FAB development were physical function (PF) and behavioral health (BH). These domains were compared to legacy instruments intended to reflect the same underlying domains of function in order to evaluate content validity. A reliability study was planned, launched, and is underway to determine consistency of functional reporting and associated scores. In FY14, a user simulation study was conducted to examine the utility of the FAB with the intent of identifying and resolving potential FAB administration issues in the context of field office operations. Boston University has embarked on a number of additional post development studies to enhance the functionality, utility, and comprehensiveness of the PF and BH instruments. Content experts were assembled to inform development of the remaining FAB domains, daily activities (which encompasses self-care, domestic life and the transportation sub-domain) and learning and applying knowledge (which includes cognition, as well as general tasks and demands and communication). Item banks were developed for these domains. Cognitive tests were completed on these item banks.