Contributing to the Mathematical Rigor of Approximate Dynamic Programming

South Dakota School Of Mines And Technology, Rapid City SD

Investigators

Abstract

Adaptive/approximate dynamic programming (ADP) is a methodology for control of dynamical systems that is motivated by the ways in which humans learn to control mechanisms or to operate machinery. ADP has shown great promise in practice. The methodology provides suitable sets of commands that maximize the performance of the system while minimizing energy consumption under different challenging circumstances, including the presence of nonlinearities, constraints, and/or modeling uncertainties. Aerospace vehicles, autonomous robots, power generators and grids, chemical processes, bioengineering systems, and economic processes are sample applications in which ADP has provided numerous benefits for society through its superior performance compared with alternative methods. However, the utilization of the scheme for sensitive systems requires ironclad guarantees of suitable performance, the lack of which is typically a shortcoming of ADP. This project is aimed at contributing to alleviating this weakness by providing performance guarantees for delicate and sensitive systems, where undesirable performance can have catastrophic consequences. Besides the research objectives, the PI will pursue educational objectives including promoting undergraduate research and involving underrepresented minorities in science and engineering. The project addresses optimal control problems with discrete-time dynamics, continuous state spaces, continuous or discrete action spaces, undiscounted finite horizon or infinite horizon cost functions, and possibly unknown dynamics. Analysis of convergence of the learning iterations, stability of the results, and continuity of the functions subject to approximation (which guarantees the possibility of their uniform approximation), comprise the initial stage the project. Analyzing the ramifications of the presence of approximation errors on the optimality and stability of the resulting controllers is another task of the project along with investigating the implications of ceasing learning after a finite number of iterations. Moreover, firm estimates of the domain of attraction for several variations of ADP will be sought in order to guarantee that the states of the system starting in the domain in which the solution is learned will never exit it; hence, ensuring the solution will remain valid and reliable. The main idea of the project is utilizing the uniform approximation capability of parametric function approximators to investigate their consequences on the learning iterations and the reliability of the result. Various learning schemes, including value iteration, policy iteration, multi-step look-ahead policy iteration, and optimistic policy iteration, will be investigated for both model-based and model-free settings.