Deep Reinforcement Learning Via Nonlinear Model Predictive Control for Thermal Process with Variable Longtime Delay

Abstract

The main concern in thermal process control revolves around uncertainties and disturbances, arising from external processes, unmodeled dynamics, or simplified characteristics, to name a few. For instance, a primary source of uncertainties involves disturbances and long-time delays, which typically lead to loose robust control performance. This paper develops a robust control technique based on Reinforcement-Learning (RL) strategies via Deep Deterministic Policy Gradient (DDPG), integrating Nonlinear Model Predictive Control (NMPC). The NMPC works as a policy generator and the DDPG strategy is devoted to evaluating the learning process. While NMPC was able to approach tracking performance, the combined scheme with DDPG allowed further robust performance in terms of adaptation to changing thermal process conditions such as external disturbances and variations to internal model parameters. Indeed, combining strategies (NMPC-based DDPG) rendered unnecessary offline design of a terminal cost and constraints typically required in traditional robustified NMPC strategies. The RL agent was trained, tested, and validated in a simulation environment using a thermal process with longtime delay. Results demonstrated that the proposed NMPC-based DDPG technique achieved nearly similar tracking performance compared to traditional NMPC strategies, even maintaining control objectives. However, the proposed control strategy exhibited enhanced adaptivity regarding NMPC under the presence of disturbances and model parameter variations. The latter findings are expected to have an impact on the energy resources of real thermal processes in the industry.