Multi-Agent Differential Graphical Games with Saturating Actuators and Integral Reinforcement Learning Solutions

In this project a class of games known as differential graphical games is studied, where the information flow between agents is prescribed by a communication graph structure. The differential graphical game formulation explicitly captures the structure of the communication graph. Therefore, its analysis clearly reveals the interplay of individual node dynamics and the graph topology within a multi-player game. Thus these graphical games are special type of the standard games discussed normally in the literature. Cooperative control ideas (mainly pinning control) will be used to formulate the error protocols and the error dynamics in terms of the local information available to each agent. Herein, the constrained optimal control problem is studied through the frame work of the Hamilton-Jacobi-Bellman (HJB) equation. Necessary conditions for optimality are derived using the Hamiltonian mechanics. The solution for the Hamilton-Jacobi-Bellman (HJB) equation is challenging due to its nonlinearity. The saturated control requirements make the value function and the control law nonlinear. In the nonlinear cases, we cant find explicit solutions for the Hamilton-Jacobi-Bellman (HJB) equations. Online adaptive Integral Reinforcement Learning Techniques (Policy Iteration) will be used to solve a set of coupled Hamilton-Jacobi-Bellman (HJB) equations developed for the differential graphical games using non-quadratic performance indices to face the saturation issue. In this project, Nash solutions for the differential graphical games are found in terms of the local information available to each agent. Moreover, Nash solutions are related to the Hamilton Jacobi Bellman (HJB) equations or equivalently the Integral Reinforcement Leaning Bellman optimality equations. The graphical game Integral Reinforcement Learning Bellman equations will be developed for the differential graphical games. The relation between these equations and the graphical game coupled Hamilton-Jacobi-Bellman equations will be investigated. Novel multi-agent policy iteration algorithm will be developed to learn the Nash solution for the differential graphical games with constrained inputs in real-time without knowing the complete dynamic models of the agents. In this project, policy iteration convergence proofs will be investigated in terms of the graph interconnectivity properties. It is worth to note that policy iteration convergence proofs for the dynamic games depend on lose conditions about the games. Actor-critic neural network techniques will be used to implement the adaptive multi-agent Integral Reinforcement Learning algorithm. These structures will be used to solve all the Hamilton-Jacobi-Bellman equations simultaneously for the differential graphical game. The implementation of policy iteration algorithm using actor-critic neural networks will be introduced in terms of gradient descent techniques. This project lays the mathematical foundation to solve the differential graphical games using Integral Reinforcement learning techniques with constrained control inputs.