Integral Reinforcement-Learning-Based Optimal Containment Control for Partially Unknown Nonlinear Multiagent Systems

Abstract

This paper focuses on the optimal containment control problem for the nonlinear multiagent systems with partially unknown dynamics via an integral reinforcement learning algorithm. By employing integral reinforcement learning, the requirement of the drift dynamics is relaxed. The integral reinforcement learning method is proved to be equivalent to the model-based policy iteration, which guarantees the convergence of the proposed control algorithm. For each follower, the Hamilton–Jacobi–Bellman equation is solved by a single critic neural network with a modified updating law which guarantees the weight error dynamic to be asymptotically stable. Through using input–output data, the approximate optimal containment control protocol of each follower is obtained by applying the critic neural network. The closed-loop containment error system is guaranteed to be stable under the proposed optimal containment control scheme. Simulation results demonstrate the effectiveness of the presented control scheme.

1. Introduction

Distributed coordination control of multiagent systems (MASs) has drawn expansive interest due to its potential application on agricultural irrigation [1], disaster rescue [2], microgrid scheduling [3], marine survey [4] and wireless communication [5]. The distributed coordination control aims to guarantee that all agents which exchange local information by communicating with their neighbors reach an agreement on some variables of interest [6]. Over the last decade, containment control has received increasing attention because of its remarkable performance in addressing the secure control issues, such as hazardous material treatment [7] and fire rescue [8]. The goal of containment control is to drive the followers to enter and keep within the convex hull spanned by multiple leaders. Numerous interesting and significant results of containment control have been presented. Reference [9] developed a fuzzy-observer-based backstepping control to achieve the containment of MASs. An adaptive funnel containment control was proposed in [10], where the containment errors converged to an adjustable funnel boundary. In practical applications, containment control has been developed for autonomous surface vehicles [4], unmanned aerial vehicles [11] and spacecrafts [12]. Notice that most of the aforementioned works have ignored the control performance with a minimum of energy consumption.

It is well-known that the Riccati equation or the Hamilton–Jacobi–Bellman equation (HJBE) are solved to acquire the optimal control for linear or nonlinear systems [13], respectively. In other words, the Riccati equation is a particular case of the HJBE. As a classical optimization algorithm, dynamic programming (DP) [14] is regarded as an effective way to obtain the optimal solution of the HJBE. However, as the dimension of state variables increases, the computation of the DP approach expands as a geometric series, which arouses the dilemma of the “curse of dimensionality”. With the success of AlphaGo, reinforcement learning (RL) has stimulated increasing enthusiasm from scholars to tackle the “curse of dimensionality” problem [15]. As is synonymous with RL, adaptive DP (ADP) [16] forward-in-time-solves the optimal control problem with the aid of neural network (NN)-based approximators. Moreover, ADP has been increasingly exploited for the optimal coordination control of MASs. Reference [17] established a cooperative policy iteration (PI) algorithm to solve the differential graphical games of linear MASs. In the nonlinear case, Reference [18] investigated the consensus problem via model-based PI with a generalized fuzzy hyperbolic critic structure. An event-triggered ADP-based optimal coordination control was proposed for the communication load and the commutation consumption was reduced [19]. To tackle the optimal containment control (OCC) problem, a finite-time fault-tolerant control was proposed via model-based PI [20]. In the presence of state constraints, Reference [21] presented a proper barrier function to transform the state constraint problem into an unconstrained case, thereafter the event-triggered OCC protocols were obtained. In Reference [22], distributed RL was applied to handle an OCC problem with collision avoidance of nonholonomic mobile robots. When the accurate model of the plant is not obtained, system identification is always employed. It should be pointed out that system identification is intractable for responding to dynamic changes of systems in time, which brings inevitable identification errors.

Recently, the integral RL (IRL) method was adopted to relax the accurate model requirement of the plant by constructing the integral Bellman equation [23,24]. An actor–critic architecture was adopted to execute the IRL algorithm, in which an actor NN learned the optimal control strategy and a critic NN was devoted to approximating the optimal value function. In the presence of heterogeneous linear MASs (HLMASs), the IRL method was developed to handle the robust OCC problem [25]. An adaptive output-feedback method was developed for the containment control for HLMASs via the IRL algorithm [26]. In Reference [27], the off-policy IRL-based OCC scheme was presented for unknown HLMASs with active leaders. However, the OCC problem of the nonlinear MASs with partially unknown dynamics has rarely been investigated via the IRL method. Moreover, the actor–critic architecture requires constructing the actor NN, which makes the control structure more complex. It is crucial to develop an IRL-based OCC scheme by implementing a simplified control structure. In addition, most of the aforementioned OCC approaches ensure the weight estimation error of the critic NN is uniformly ultimately bounded (UUB) only, which may degrade the control performance. All the above concerns motivated our research.

Inspired by the aforementioned works, we developed an IRL-based OCC scheme with asymptotically stable critic structure for partially unknown nonlinear MASs. The main contributions are reflected as follows.

(1)

Different from existing control schemes [9,20], an IRL method is introduced to construct the integral Bellman equation without the system identification. Furthermore, IRL proves to be equivalent to model-based PI, which guarantees the convergence of the developed control algorithm.

(2)

The IRL-based OCC scheme is implemented by a critic-only architecture for nonlinear MASs with unknown drift dynamics, rather than by an actor–critic architecture for linear MASs [25,26,27]. Thus, the proposed scheme simplifies the control structure.

(3)

In contrast to the existing OCC schemes [20,21,22] which guarantee the weight errors to be UUB, a modified weight-updating law is presented to tune the critic NN weights, whose weight error dynamic is asymptotically stable.

This paper is organized as follows. In Section 2, graph theory and its application to the containment of MASs are outlined. In Section 3, the IRL-based OCC scheme and its convergence proof are presented for nonlinear MASs. Then, the stability of the closed-loop containment error systems is analyzed in detail. In Section 4, two simulation examples demonstrate the effectiveness of the proposed scheme. In Section 5, concluding remarks are drawn.

2. Preliminaries and Problem Description

2.1. Graph Theory

For a network with N agents, the information interactions among agents are reflected by a weighted graph $\mathcal{G}=(V,\epsilon ,\mathbb{A})$ with the nonempty finite set of nodes $V=\{{\upsilon }_1,\dots ,{\upsilon }_N\}$, the edge set $\epsilon \subseteq V\times V$ and the nonnegative weighted adjacency matrix $\mathbb{A}=\left[a_{ip}\right]$. If node ${\upsilon }_i$ links to node ${\upsilon }_p$, the edge $({\upsilon }_i,{\upsilon }_p)\in \epsilon$ is available with $a_{ip}>0$; otherwise, $a_{ip}=0$. For a node ${\upsilon }_i$, the node ${\upsilon }_p$ is named as a neighbor of ${\upsilon }_i$ when $({\upsilon }_i,{\upsilon }_p)\in \epsilon$. In this way, $N_i=\{{\upsilon }_p\in V:({\upsilon }_p,{\upsilon }_i)\in \epsilon \}$ represents the set of all neighbors of ${\upsilon }_i$. Denote the Laplacian matrix as $L=D-\mathbb{A}=\left[l_{ip}\right]$, where $D=\mathrm{diag}\{d_{11},d_{22},\dots ,d_{NN}\}$, $d_{ii}={\sum }_{p=1}^{N_i}{a}_{ip}$ and $l_{ip}$ satisfies

$$l_{ip}=\left\{\begin{array}{cc}\hfill \sum _{q=1,q\ne i}^{N_i}{a}_{iq},& i=p,\hfill \\ \hfill -a_{ip},& i\ne p.\hfill \end{array}\right.$$

It implies that each row sum of L equals to zero. A sequence of edges described by $({\upsilon }_1,{\upsilon }_2),({\upsilon }_3,{\upsilon }_4),\dots$ with ${\upsilon }_i\in V$ is defined as a directed path. For arbitrary $({\upsilon }_i,{\upsilon }_p)\in V$, a directed graph is strongly connected, if there is a directed path from ${\upsilon }_i$ to ${\upsilon }_p$, while the directed graph is said to contain a spanning tree if there exists a directed path from a root node to every other nodes with respect to $\mathcal{G}$. This paper focuses on a strongly connected digraph with a spanning tree.

2.2. Problem Description

Consider the leader–follower nonlinear MASs in the form of the graph $\mathcal{G}$ with M leaders and N followers, where the node dynamic of the ith follower is modeled by

$${\dot{x}}_i=f\left(x_i\left(t\right)\right)+g_i\left(x_i\left(t\right)\right){\mu }_i\left(t\right),$$

(1)

where $x_i\in {\mathbb{R}}^n$ is the state vector for the ith follower, ${\mu }_i\in {\mathbb{R}}^m$ is the control input vector, $i=1,2,\dots ,N$, and the nonlinear functions $f\left(x_i\right)\in {\mathbb{R}}^n$ and $g_i\left(x_i\right)\in {\mathbb{R}}^{n\times m}$ represent the unknown drift dynamic and the control input matrix, respectively. Denote the global state vector as $x={[x_1^{\mathsf{T}},x_2^{\mathsf{T}},\dots ,x_N^{\mathsf{T}}]}^{\mathsf{T}}\in {\mathbb{R}}^{N\times n}$.

Assumption 1.

$f\left(x_i\right)$ and $g_i\left(x_i\right)$ are Lipschitz continuous on the compact set ${\mathsf{\Omega }}_i$ with $f\left(0\right)=0$ and the system (1) is controllable.

Define the node dynamic of the jth leader as

$${\dot{r}}_j=h_j\left(r_j\left(t\right)\right),$$

(2)

where $r_j\in {\mathbb{R}}^n$ stands for the state vector of the jth leader, $j=1,2,\dots ,M$ and $h_j\left(r_j\right)\in {\mathbb{R}}^n$ satisfies Lipschitz continuity.

Definition 1

(Convex hull [8]). A set $\mathcal{C}\subseteq {\mathbb{R}}^{M\times n}$ is convex if for any $y_1,y_2\in \mathcal{C}$ and $\forall \rho \in (0,1)$, $\left((1-\rho )y_1+\rho y_2\right)\in \mathcal{C}$. A convex hull of a finite set $Y=\{y_1,y_2,\dots ,y_M\}$ is the minimal convex set, i.e., $Co\left(Y\right)=\left\{{\sum }_{j=1}^M{\rho }_j{y}_j|y_j\in Y,{\rho }_j\in \mathbb{R},{\rho }_j\ge 0,{\sum }_{j=1}^M{\rho }_j=1\right\}$.

The containment control aims to find a set of distributed control protocols $\mu =\{{\mu }_1,{\mu }_2,\dots ,{\mu }_N\}$ such that all followers stay in the convex hull formed by the leaders, i.e., $x_i\left(t\right)\to Co\left(Y\right)$ with $Y=\{r_1,r_2,\dots ,r_M\}$. For the ith follower, the local neighborhood containment error $e_i$ is formulated as

$$\begin{array}{cc}\hfill e_i=& \sum _{p\in N_i}{a}_{ip}(x_i-x_p)+\sum _{j=1}^M{b}_{ij}(x_i-r_j)\hfill \\ \hfill =& d_{ii}{x}_i-\sum _{p\in N_i}{a}_{ip}{x}_p+\sum _{j=1}^M{b}_{ij}(x_i-r_j),\hfill \end{array}$$

(3)

where $e_i\in {\mathbb{R}}^n$, $b_{ij}\ge 0$ represents the pinning gain. Define $B_j=\mathrm{diag}[b_{1j},\dots ,b_{ij},\dots ,b_{Nj}]\in {\mathbb{R}}^{N\times N}$. In fact, the connection between the ith follower and the jth leader is available if and only if $b_{ij}>0$. Denote the communication graph as ${\mathcal{G}}_x=(\mathcal{G},x)$. The global containment error vector of ${\mathcal{G}}_x$ is

$$e=(G\otimes I_n)x+\left(\left(B(I_M\otimes 1_N)\right)\otimes I_n\right)\overline{r},$$

where $e={[e_1^{\mathsf{T}},e_2^{\mathsf{T}},\dots ,e_N^{\mathsf{T}}]}^{\mathsf{T}}\in {\mathbb{R}}^{N\times n}$, $\overline{r}={[r_1^{\mathsf{T}},r_2^{\mathsf{T}},\dots ,r_M^{\mathsf{T}}]}^{\mathsf{T}}\in {\mathbb{R}}^{M\times n}$, $G=L+B(1_M\otimes I_N)$, $I_n$ represents the n-dimension identity matrix, $1_M$ stands for the M-dimensional column vector whose every element equals to 1 and $B=[B_1,B_2,\dots ,B_M]\in {\mathbb{R}}^{N\times NM}$. Considering (1), (2) and (3), for the ith follower, the local neighborhood containment error dynamic is formulated as

$$\begin{array}{c}\hfill {\dot{e}}_i=F_i+c_i{g}_i\left(x_i\right){\mu }_i+\sum _{p\in N_i}{a}_{ip}{g}_p\left(x_p\right){\mu }_p,\end{array}$$

(4)

where $c_i=\left(d_{ii}+{\sum }_{j=1}^M{b}_{ij}\right)$ and $F_i=c_{i}f\left(x_i\right)-{\sum }_{p\in N_i}{a}_{ip}f\left(x_p\right)-{\sum }_{j=1}^M{b}_{ij}{h}_j\left(r_j\right)$. For the ith follower, the local neighborhood containment error is dominated not only by local states and local control inputs, but also by the information from its neighbors and the leaders. In order to implement the synchronization of the partially unknown nonlinear MASs (i.e., $e_i\to 0$), an IRL-based OCC scheme is designed in the next subsection.

3. IRL-Based OCC Scheme

3.1. Optimal Containment Control

For the local neighborhood containment error dynamic (4), define the cost function as

$$J_i\left(e_i\left(0\right)\right)={\int }_0^{\infty }{P}_i\left(e_i\left(\xi \right),{\mu }_i\left(\xi \right),{\mu }_{-i}\left(\xi \right)\right)\mathrm{d}\xi ,$$

(5)

where $P_i(e_i,{\mu }_i,{\mu }_{-i})=e_i^{\mathsf{T}}{Q}_i{e}_i+{\sum }_{p\in \{N_i,i\}}{\mu }_p^{\mathsf{T}}{R}_{ip}{\mu }_p$ is a utility function, ${\mu }_{-i}=\left\{{\mu }_p\right|p\in N_i\}$ represents a set of the local control protocols from the neighbors of node ${\upsilon }_i$, and $Q_i\in {\mathbb{R}}^{n\times n}$ and $R_{ip}\in {\mathbb{R}}^{m\times m}$ are the positive definite matrices.

Definition 2

(Admissible control policies [17]). The feedback control policies ${\mu }_i\left(e_i\right)(i\in \mathcal{I})$ are defined to be admissible with respect to (5) on a compact set ${\mathsf{\Omega }}_i$, denoted by ${\mu }_i\left(e_i\right)\in \mathcal{A}\left({\mathsf{\Omega }}_i\right)$, if ${\mu }_i\left(e_i\right)$ is continuous on ${\mathsf{\Omega }}_i$ with ${\mu }_i\left(0\right)=0$, ${\mu }_i\left(e_i\right)$ stabilizes (4) on ${\mathsf{\Omega }}_i$ and $J_i\left(e_i\left(0\right)\right)$ is finite $\forall e_i\left(0\right)\in {\mathsf{\Omega }}_i$.

Definition 3

(Nash equilibrium [17]). An N-tuple admissible control policy ${\mu }^{*}\left(e\right)=\{{\mu }_1^{*}\left(e_1\right),$${\mu }_2^{*}\left(e_2\right),\dots ,{\mu }_N^{*}\left(e_N\right)\}$ is said to constitute a Nash equilibrium solution in graph ${\mathcal{G}}_x$, if the following N inequalities are satisfied

$$\begin{array}{c}\hfill J_i(e_i,{\mu }_i^{*},{\mu }_{-i}^{*})\le J_i(e_i,{\mu }_i,{\mu }_{-i}^{*}),i=1,2,\dots ,N,\end{array}$$

where ${\mu }_{-i}^{*}=\{{\mu }_1^{*},\dots ,{\mu }_{i-1}^{*},{\mu }_{i+1}^{*},\dots ,{\mu }_N^{*}\}$.

This paper aims to find an N-tuple optimal admissible control policy ${\mu }^{*}\left(e\right)$ to minimize the cost function (5) for each follower such that the Nash equilibrium solution in ${\mathcal{G}}_x$ (i.e., the OCC protocols) is obtained.

For arbitrary ${\mu }_i\left(e_i\right)\in \mathcal{A}\left({\mathsf{\Omega }}_i\right)$ of the ith follower, define the value function

$$C_i\left(e_i\left(t\right)\right)={\int }_t^{\infty }{P}_i\left(e_i\left(\xi \right),{\mu }_i\left(\xi \right),{\mu }_{-i}\left(\xi \right)\right)\mathrm{d}\xi .$$

(6)

When (6) is finite, then the Bellman equation is

$$\begin{array}{c}\hfill 0=e_i^{\mathsf{T}}{Q}_i{e}_i+\sum _{p\in \{N_i,i\}}{\mu }_p^{\mathsf{T}}{R}_{ip}{\mu }_p+\nabla C_i^{\mathsf{T}}\left(e_i\right)\left(F_i+c_i{g}_i\left(x_i\right){\mu }_i+\sum _{p\in N_i}{a}_{ip}{g}_p\left(x_p\right){\mu }_p\right),\end{array}$$

(7)

where $V_i\left(0\right)=0$ and $\nabla C_i\left(e_i\right)=\partial C_i\left(e_i\right)/\partial e_i$. For the ith follower, the local Hamiltonian is

$$\begin{array}{cc}\hfill H_i(e_i,{\mu }_i,{\mu }_{-i},C_i\left(e_i\right))=& e_i^{\mathsf{T}}{Q}_i{e}_i+\sum _{p\in \{N_i,i\}}{\mu }_p^{\mathsf{T}}{R}_{ip}{\mu }_p\hfill \\ \hfill & +\nabla C_i^{\mathsf{T}}\left(e_i\right)\left(F_i+c_i{g}_i\left(x_i\right){\mu }_i+\sum _{p\in N_i}{a}_{ip}{g}_p\left(x_p\right){\mu }_p\right).\hfill \end{array}$$

Define the optimal value function as

$$C_i^{*}\left(e_i\right)=\underset{{\mu }_i\in \mathcal{A}\left({\mathsf{\Omega }}_i\right)}{min}{C}_i\left(e_i\right).$$

(8)

According to [13], the optimal value function $C_i^{*}\left(e_i\right)$ satisfies the HJBE as follows

$$0=\underset{{\mu }_i\in \mathcal{A}\left({\mathsf{\Omega }}_i\right)}{min}{H}_i(e_i,{\mu }_i,{\mu }_{-i},C_i^{*}\left(e_i\right)).$$

(9)

The local OCC protocol is

$$\begin{array}{cc}\hfill {\mu }_i^{*}\left(e_i\right)=& arg\underset{{\mu }_i\in \mathcal{A}\left({\mathsf{\Omega }}_i\right)}{min}{H}_i(e_i,{\mu }_i,{\mu }_{-i},C_i^{*}\left(e_i\right))\hfill \\ \hfill =& -\frac{1}{2}{c}_i{R}_{ii}^{-1}{g}_i^{\mathsf{T}}\left(x_i\right)\nabla C_i^{*}\left(e_i\right).\hfill \end{array}$$

(10)

It should be mentioned that the analytical solution of the HJBE is intractable to obtain since $C_i^{*}\left(e_i\right)$ is unknown. According to [15], the solution of the HJBE is successively approximated through a sequence of iterations with policy evaluation

$$\begin{array}{cc}\hfill 0=& e_i^{\mathsf{T}}{Q}_i{e}_i+\sum _{p\in \{N_i,i\}}{\mu }_p^{(k-1)\mathsf{T}}{R}_{ip}{\mu }_p^{(k-1)}\hfill \\ \hfill & +\nabla C_i^{\left(k\right)\mathsf{T}}\left(e_i\right)\left(F_i+c_i{g}_i\left(x_i\right){\mu }_i^{(k-1)}+\sum _{p\in N_i}{a}_{ip}{g}_p\left(x_p\right){\mu }_p^{(k-1)}\right),\hfill \end{array}$$

(11)

and policy improvement

$$\begin{array}{c}\hfill {\mu }_i^{\left(k\right)}=-\frac{1}{2}{c}_i{R}_{ii}^{-1}{g}_i^{\mathsf{T}}\left(x_i\right)\nabla C_i^{\left(k\right)}\left(e_i\right),\end{array}$$

(12)

where $\left(k\right)$ represents the kth iteration index with $k\in {\mathbb{N}}^{+}$.

From (11), we can see that the policy evaluation requires the accurate mathematical model of (1). However, the accurate mathematical model is always difficult to obtain in practice. To break this bottleneck, the IRL method is developed to relax the requirement of the accurate model in the policy evaluation.

3.2. Integral Reinforcement Learning

For $t_{\tau }>0$, (6) can be rewritten as

$$\begin{array}{c}\hfill C_i\left(e_i\left(t\right)\right)={\int }_t^{t+t_{\tau }}\left(e_i^{\mathsf{T}}\left(\xi \right)Q_i{e}_i\left(\xi \right)+\sum _{p\in \{N_i,i\}}{\mu }_p^{\mathsf{T}}\left(\xi \right)R_{ip}{\mu }_p\left(\xi \right)\right)\mathrm{d}\xi +C_i\left(e_i(t+t_{\tau })\right).\end{array}$$

(13)

Based on the integral Bellman Equation (13), $V_i^{*}\left(e_i\right)$ and ${\mu }_i^{*}$ satisfy

$$\begin{array}{cc}\hfill 0=& {\int }_t^{t+t_{\tau }}\left(e_i^{\mathsf{T}}\left(\xi \right)Q_i{e}_i\left(\xi \right)+\sum _{p\in \{N_i,i\}}{\mu }_p^{*\mathsf{T}}\left(\xi \right)R_{ip}{\mu }_p^{*}\left(\xi \right)\right)\mathrm{d}\xi \hfill \\ \hfill & +C_i^{*}\left(e_i(t+t_{\tau })\right)-C_i^{*}\left(e_i\left(t\right)\right).\hfill \end{array}$$

(14)

Compared to (7), the policy evaluation (14) is not required for the accurate system dynamics in (1).

Theorem 1.

Let $C_i^{\left(k\right)}\left(e_i\right)\ge 0$, $C_i^{\left(k\right)}\left(0\right)=0$ and ${\mu }_i^{\left(k\right)}\in \mathcal{A}\left({\mathsf{\Omega }}_i\right)$. $C_i^{\left(k\right)}\left(e_i\right)$ is the solution of the integral Bellman equation

$$\begin{array}{cc}\hfill 0=& {\int }_t^{t+t_{\tau }}{e}_i^{\mathsf{T}}\left(\xi \right)Q_i{e}_i\left(\xi \right)\mathrm{d}\xi +{\int }_t^{t+t_{\tau }}\sum _{p\in \{N_i,i\}}{\mu }_p^{(k-1)\mathsf{T}}\left(\xi \right)R_{ip}{\mu }_p^{(k-1)}\left(\xi \right)\mathrm{d}\xi \hfill \\ \hfill & +C_i^{\left(k\right)}\left(e_i(t+t_{\tau })\right)-C_i^{\left(k\right)}\left(e_i\left(t\right)\right),\hfill \end{array}$$

(15)

if and only if $C_i^{\left(k\right)}\left(e_i\right)$ is the only solution of (11).

Proof of Theorem  1.

Considering (11), the time derivative of $C_i^{\left(k\right)}\left(e_i\right)$ corresponding to (4) is transformed as

$$\begin{array}{cc}\hfill \frac{\mathrm{d}{C}_i^{\left(k\right)}\left(e_i\right)}{\mathrm{d}t}=& \nabla C_i^{\left(k\right)}\left(e_i\right)\left(F_i+c_i{g}_i\left(x_i\right){\mu }_i^{(k-1)}+\sum _{p\in N_i}{a}_{ip}{g}_p\left(x_p\right){\mu }_p^{(k-1)}\right)\hfill \\ \hfill =& -e_i^{\mathsf{T}}{Q}_i{e}_i-\sum _{p\in \{N_i,i\}}{\mu }_p^{(k-1)\mathsf{T}}{R}_{ip}{\mu }_p^{(k-1)}.\hfill \end{array}$$

(16)

Integrate on both sides of (16) within $[t,t+t_{\tau }]$, that is

$$\begin{array}{cc}\hfill C_i^{\left(k\right)}\left(e_i(t+t_{\tau })\right)-C_i^{\left(k\right)}\left(e_i\left(t\right)\right)=& -{\int }_t^{t+t_{\tau }}{e}_i^{\mathsf{T}}\left(\xi \right)Q_i{e}_i\left(\xi \right)\mathrm{d}\xi \hfill \\ \hfill & -{\int }_t^{t+t_{\tau }}\sum _{p\in \{N_i,i\}}{\mu }_p^{(k-1)\mathsf{T}}\left(\xi \right)R_{ip}{\mu }_p^{(k-1)}\left(\xi \right)\mathrm{d}\xi .\hfill \end{array}$$

(17)

According to the derivation of (16) and (17), if $C_i^{\left(k\right)}\left(e_i\right)$ is the solution of (11), $C_i^{\left(k\right)}\left(e_i\right)$ satisfies the integral Bellman Equation (15). Next, we verify the uniqueness of the solution $C_i^{\left(k\right)}\left(e_i\right)$.

Supposing that ${\mathsf{{\rm Y}}}_i^{\left(k\right)}\left(e_i\right)$ is another solution of (11) with ${\mathsf{{\rm Y}}}_i^{\left(k\right)}\left(0\right)=0$. Similar to the mathematical operation of (16), we have

$$\begin{array}{c}\hfill \frac{\mathrm{d}{\mathsf{{\rm Y}}}_i^{\left(k\right)}\left(e_i\right)}{\mathrm{d}t}=-e_i^{\mathsf{T}}{Q}_i{e}_i-\sum _{p\in \{N_i,i\}}{\mu }_p^{(k-1)\mathsf{T}}{R}_{ip}{\mu }_p^{(k-1)}.\end{array}$$

(18)

Subtracting (16) into (18) yields

$$\frac{\mathrm{d}}{\mathrm{d}t}\left({\mathsf{{\rm Y}}}_i^{\left(k\right)}\left(e_i\right)-C_i^{\left(k\right)}\left(e_i\right)\right)=0.$$

(19)

Solving (19), we have ${\mathsf{{\rm Y}}}_i^{\left(k\right)}\left(e_i\right)-C_i^{\left(k\right)}\left(e_i\right)={\varsigma }_i$ with ${\varsigma }_i\in \mathbb{R}$ a real constant. For $e_i=0$, we have ${\varsigma }_i={\mathsf{{\rm Y}}}_i^{\left(k\right)}\left(0\right)-C_i^{\left(k\right)}\left(0\right)=0$. That is to say, ${\mathsf{{\rm Y}}}_i^{\left(k\right)}\left(e_i\right)=C_i^{\left(k\right)}\left(e_i\right)$. One can derive that $C_i^{\left(k\right)}\left(e_i\right)$ is the unique solution. In summary, $C_i^{\left(k\right)}\left(e_i\right)$ is the unique solution of (15) if and only if $C_i^{\left(k\right)}\left(e_i\right)$ is the only solution of (11). □

Theorem 1 reveals that the IRL algorithm with (15) and (12) theoretically equals to the model-based PI algorithm, whose relevant convergence analysis was provided in [15]. Hence, the IRL algorithm can be guaranteed to be convergent.

Theorem 2.

Considering the nonlinear MAS with partially unknown dynamic as (1), the local neighborhood containment error dynamic as (4) and the optimal value function $C_i^{*}\left(e_i\right)$ as (8), the closed-loop containment error system is guaranteed to be asymptotically stable under the local OCC protocol (10). Furthermore, the containment control is achieved with a set of the OCC protocols $\{{\mu }_1^{*},{\mu }_2^{*},\dots ,$${\mu }_N^{*}\}$ if there is a spanning tree in the directed graph.

Proof of Theorem 2.

Selecting the Lyapunov function candidate as $C_i^{*}\left(e_i\right)$. Combining (7), (8) and (10), then

$$\begin{array}{cc}\hfill \nabla C_i^{*\mathsf{T}}\left(e_i\right)F_i=& -\nabla C_i^{*\mathsf{T}}\left(e_i\right)\left(c_i{g}_i\left(x_i\right){\mu }_i^{*}+\sum _{p\in N_i}{a}_{ip}{g}_p\left(x_p\right){\mu }_p^{*}\right)\hfill \\ \hfill & -e_i^{\mathsf{T}}{Q}_i{e}_i-\sum _{p\in \{N_i,i\}}{\mu }_p^{*\mathsf{T}}{R}_{ip}{\mu }_p^{*}.\hfill \end{array}$$

(20)

Substituting (20) into the time derivative of $V_i^{*}\left(e_i\right)$, then

$$\begin{array}{cc}\hfill {\dot{C}}_i^{*}\left(e_i\right)=& \nabla C_i^{*\mathsf{T}}\left(e_i\right)\left(F_i+c_i{g}_i\left(x_i\right){\mu }_i^{*}+\sum _{p\in N_i}{a}_{ip}{g}_p\left(x_p\right){\mu }_p^{*}\right)\hfill \\ \hfill =& -e_i^{\mathsf{T}}{Q}_i{e}_i-\sum _{p\in \{N_i,i\}}{\mu }_p^{*\mathsf{T}}{R}_{ip}{\mu }_p^{*}.\hfill \end{array}$$

Therefore, ${\dot{C}}_i^{*}\left(e_i\right)\le 0$. One can conclude that the closed-loop containment error system (4) is asymptotically stable with the local OCC protocol (10). Since a spanning tree exists in the directed graph, the containment control of the nonlinear MAS with partially unknown dynamic can be achieved. □

3.3. Critic NN Implementation

Based on the Stone–Weierstrass approximation theorem, on the compact set ${\mathsf{\Omega }}_i$, the optimal function $C_i^{*}\left(e_i\right)$ and its partial gradient can be established by a critic NN as

$$\begin{array}{cc}\hfill C_i^{*}\left(e_i\right)=& {\varphi }_i^{*\mathsf{T}}{\sigma }_i\left(e_i\right)+{\omega }_i\left(e_i\right),\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill \nabla C_i^{*}\left(e_i\right)=& \nabla {\sigma }_i^{\mathsf{T}}\left(e_i\right){\varphi }_i^{*}+\nabla {\omega }_i\left(e_i\right),\hfill \end{array}$$

(22)

where ${\varphi }_i^{*}\in {\mathbb{R}}^{l_i}$ represents the ideal weight, ${\sigma }_i(·)\in {\mathbb{R}}^{l_i}$ represents the activation function, $l_i$ represents the number of hidden neurons and ${\omega }_i\left(e_i\right)$ stands for the reconstruction error.

Since the ideal weight vector is unknown, the approximation of $C_i^{*}\left(e_i\right)$ and $\nabla C_i^{*}\left(e_i\right)$ are expressed as

$$\begin{array}{cc}\hfill {\widehat{C}}_i\left(e_i\right)& ={\widehat{\varphi }}_i^{\mathsf{T}}{\sigma }_i\left(e_i\right),\hfill \\ \hfill \nabla {\widehat{C}}_i\left(e_i\right)& =\nabla {\sigma }_i^{\mathsf{T}}\left(e_i\right){\widehat{\varphi }}_i,\hfill \end{array}$$

(23)

where $\nabla {\sigma }_i\left(e_i\right)=\partial {\sigma }_i\left(e_i\right)/\partial e_i$ and ${\widehat{\varphi }}_i\in {\mathbb{R}}^{l_i}$ represents the estimation of ${\varphi }_i^{*}$. Then, the local OCC protocol (10) can be approximated by

$${\widehat{\mu }}_i\left(e_i\right)=-\frac{1}{2}{c}_i{R}_{ii}^{-1}{g}_i^{\mathsf{T}}\left(x_i\right)\nabla {\sigma }_i^{\mathsf{T}}\left(e_i\right){\widehat{\varphi }}_i.$$

(24)

The approximate local Hamiltonian is

$$\begin{array}{cc}\hfill e_{ci}=& {\int }_t^{t+t_{\tau }}\left(e_i^{\mathsf{T}}\left(\xi \right)Q_i{e}_i\left(\xi \right)+\sum _{p\in \{N_i,i\}}{\widehat{\mu }}_p^{\mathsf{T}}\left(\xi \right)R_{ij}{\widehat{\mu }}_p\left(\xi \right)\right)\mathrm{d}\xi \hfill \\ \hfill & +{\widehat{\varphi }}_i^{\mathsf{T}}\underset{{\theta }_i}{\underbrace{\left({\sigma }_i\left(e_i(t+t_{\tau })\right)-{\sigma }_i\left(e_i\left(t\right)\right)\right)}}.\hfill \end{array}$$

(25)

Combining (14) and (21) with (25) yields

$$\begin{array}{cc}\hfill e_{ci}=& {\int }_t^{t+t_{\tau }}\left(e_i^{\mathsf{T}}\left(\xi \right)Q_i{e}_i\left(\xi \right)+\sum _{p\in \{N_i,i\}}{\widehat{\mu }}_p^{\mathsf{T}}\left(\xi \right)R_{ip}{\widehat{\mu }}_p\left(\xi \right)\right)\mathrm{d}\xi \hfill \\ & -{\int }_t^{t+t_{\tau }}\left(e_i^{\mathsf{T}}\left(\xi \right)Q_i{e}_i\left(\xi \right)+\sum _{p\in \{N_i,i\}}{\mu }_p^{*\mathsf{T}}\left(\xi \right)R_{ip}{\mu }_p^{*}\left(\xi \right)\right)\mathrm{d}\xi \hfill \\ \hfill & +{\widehat{\varphi }}_i^{\mathsf{T}}{\theta }_i-{\varphi }_i^{*\mathsf{T}}{\theta }_i-{\omega }_i\left(e_i(t+t_{\tau })\right)+{\omega }_i\left(e_i\left(t\right)\right)\hfill \\ \hfill =& {\int }_t^{t+t_{\tau }}\sum _{p\in \{N_i,i\}}{\left({\widehat{\mu }}_p\left(\xi \right)+{\mu }_p^{*}\left(\xi \right)\right)}^{\mathsf{T}}{R}_{ip}\left({\widehat{\mu }}_p\left(\xi \right)-{\mu }_p^{*}\left(\xi \right)\right)\mathrm{d}\xi \hfill \\ \hfill & -{\tilde{\varphi }}_i{\theta }_i-{\omega }_i\left(e_i(t+t_{\tau })\right)+{\omega }_i\left(e_i\left(t\right)\right)\hfill \\ \hfill =& -{\tilde{\varphi }}_i{\theta }_i+{\mathsf{\Phi }}_i,\hfill \end{array}$$

(26)

where ${\tilde{\varphi }}_i={\varphi }_i^{*}-{\widehat{\varphi }}_i$ represents the weight estimation error and ${\mathsf{\Phi }}_i={\int }_t^{t+t_{\tau }}{\displaystyle \sum _{p\in \{N_i,i\}}}{\left({\widehat{\mu }}_p\left(\xi \right)+{\mu }_p^{*}\left(\xi \right)\right)}^{\mathsf{T}}{R}_{ip}\left({\widehat{\mu }}_p\left(\xi \right)-{\mu }_p^{*}\left(\xi \right)\right)\mathrm{d}\xi -{\omega }_i\left(e_i(t+t_{\tau })\right)+{\omega }_i\left(e_i\left(t\right)\right)$.

Assumption 2.

${\mathsf{\Phi }}_i$ is bounded by ${\eta }_i$, i.e., $\parallel {\mathsf{\Phi }}_i\parallel \le {\eta }_i$ with ${\eta }_i>0$.

In order to tune ${\widehat{\varphi }}_i$, the steepest descent algorithm is employed to minimize $E_{ci}=\frac{1}{2}{e}_{ci}^2$. A modified updating law of ${\widehat{\varphi }}_i$ is

$${\dot{\widehat{\varphi }}}_i=-l_{ci}\frac{{\theta }_i}{{(1+{\theta }_i^{\mathsf{T}}{\theta }_i)}^2}\left(e_{ci}-{\widehat{\eta }}_i\right)$$

(27)

where $l_{ci}>0$ and ${\widehat{\eta }}_i$, the estimation of ${\eta }_i$, can be updated by

$${\dot{\widehat{\eta }}}_i=l_{si}\frac{{\tilde{\varphi }}_i^{\mathsf{T}}{\theta }_i}{{(1+{\theta }_i^{\mathsf{T}}{\theta }_i)}^2},$$

(28)

where $l_{si}>0$ is a design constant. Considering (26) and (27), the weight estimation error is updated by

$${\dot{\tilde{\varphi }}}_i=-l_{ci}\frac{{\theta }_i}{{(1+{\theta }_i^{\mathsf{T}}{\theta }_i)}^2}\left({\tilde{\varphi }}^{\mathsf{T}}{\theta }_i-{\mathsf{\Phi }}_i+{\widehat{\eta }}_i\right).$$

(29)

Theorem 3.

Considering the nonlinear MAS with partially unknown dynamic as (1), the local neighborhood containment error dynamic as (4) and the critic NN with the modified updating laws (27) and (28), then ${\tilde{\varphi }}_i$ is guaranteed to be asymptotically stable.

Proof of Theorem 3.

Define ${\tilde{\eta }}_i={\eta }_i-{\widehat{\eta }}_i$. Choose the Lyapunov function candidate as

$${\mathsf{\Xi }}_{ci}=\frac{1}{2l_{ci}}{\tilde{\varphi }}_i^{\mathsf{T}}{\tilde{\varphi }}_i+\frac{1}{2l_{si}}{\tilde{\eta }}_i^2.$$

(30)

According to (28), ${\tilde{\eta }}_i$ is updated by

$${\dot{\tilde{\eta }}}_i=-l_{si}\frac{{\tilde{\varphi }}_i^{\mathsf{T}}{\theta }_i}{{(1+{\theta }_i^{\mathsf{T}}{\theta }_i)}^2}.$$

(31)

Considering (29) and (31), the time derivative of (30) is

$$\begin{array}{cc}\hfill {\dot{\mathsf{\Xi }}}_{ci}=& \frac{1}{l_{ci}}{\tilde{\varphi }}_i^{\mathsf{T}}{\dot{\tilde{\varphi }}}_i+\frac{1}{l_{si}}{\tilde{\eta }}_i{\dot{\tilde{\eta }}}_i\hfill \\ \hfill =& -\frac{{\tilde{\varphi }}_i^{\mathsf{T}}{\theta }_i}{{(1+{\theta }_i^{\mathsf{T}}{\theta }_i)}^2}\left({\tilde{\varphi }}^{\mathsf{T}}{\theta }_i-{\mathsf{\Phi }}_i+{\widehat{\eta }}_i\right)-\frac{{\tilde{\varphi }}_i^{\mathsf{T}}{\theta }_i}{{(1+{\theta }_i^{\mathsf{T}}{\theta }_i)}^2}{\tilde{\eta }}_i\hfill \\ \hfill =& -{\tilde{\varphi }}_i^{\mathsf{T}}{\mathsf{\Psi }}_i{\tilde{\varphi }}_i+\frac{{\tilde{\varphi }}_i^{\mathsf{T}}{\theta }_i}{{(1+{\theta }_i^{\mathsf{T}}{\theta }_i)}^2}({\mathsf{\Phi }}_i-{\widehat{\eta }}_i-{\tilde{\eta }}_i),\hfill \end{array}$$

(32)

where ${\mathsf{\Psi }}_i={\theta }_i{\theta }_i^{\mathsf{T}}/{(1+{\theta }_i^{\mathsf{T}}{\theta }_i)}^2$. According to Assumption 2, (32) is derived as

$$\begin{array}{cc}\hfill {\dot{\mathsf{\Xi }}}_{ci}\le & -{\lambda }_{min}\left({\mathsf{\Psi }}_i\right){\parallel {\tilde{\varphi }}_i\parallel }^2+\frac{{\tilde{\varphi }}_i^{\mathsf{T}}{\theta }_i}{{(1+{\theta }_i^{\mathsf{T}}{\theta }_i)}^2}\left(\parallel {\mathsf{\Phi }}_i\parallel -{\eta }_i\right)\hfill \\ \hfill \le & -{\lambda }_{min}\left({\mathsf{\Psi }}_i\right){\parallel {\tilde{\varphi }}_i\parallel }^2.\hfill \end{array}$$

It indicates ${\dot{\mathsf{\Xi }}}_{ci}\le 0$. Therefore, one can conclude that ${\tilde{\varphi }}_i$ is ensured to be asymptotically stable. □

Under the framework of the critic-only architecture, the IRL-based OCC scheme is presented. For each follower, the local neighborhood containment error (3) is established by communicating with its neighbors and the leaders. The value function of each follower is approximated by the critic NN (23), whose weights are tuned by a modified weight updating law (27). Based on (1), (3) and (23), the local OCC protocol (24) is obtained. The structural diagram of the developed IRL-based OCC scheme is shown in Figure 1.

Remark 1. 

In the actor–critic architecture, the optimal value function and the optimal control policy are approximated by a critic NN and an actor NN, respectively. While for the critic-only architecture, the optimal value function is approximated by a critic NN and the optimal control policy is directly obtained by combining (10) and (22). Hence, the critic-only architecture keeps the same performance as the actor–critic one. In contrast, the critic-only architecture utilizes a single critic NN only, which implies that the control structure is simplified and the computation burden is reduced.

3.4. Stability Analysis

Assumption 3.

${\varphi }_i^{*}$, ${\tilde{\varphi }}_i$, $\nabla {\sigma }_i(·)$ and $\nabla {\omega }_i(·)$ are norm-bounded, i.e.,

$$\begin{array}{c}\hfill \parallel {\varphi }_i^{*}\parallel \le {\varphi }_{i\mathrm{M}},\parallel {\tilde{\varphi }}_i\parallel \le {\overline{\varphi }}_{i\mathrm{M}},\parallel \nabla {\sigma }_i(·)\parallel \le {\overline{\sigma }}_{i\mathrm{M}},\parallel \nabla {\omega }_i(·)\parallel \le {\overline{\omega }}_{i\mathrm{M}},\parallel g_i(·)\parallel \le {\overline{g}}_{i\mathrm{M}},\end{array}$$

where ${\varphi }_{i\mathrm{M}}$, ${\overline{\varphi }}_{i\mathrm{M}}$, ${\overline{\sigma }}_{i\mathrm{M}}$, ${\overline{\omega }}_{i\mathrm{M}}$ and ${\overline{g}}_{i\mathrm{M}}$ are positive constants.

Theorem 4. 

Considering the nonlinear MAS with partially unknown dynamics as (1), the local neighborhood containment error dynamic as (4), the optimal value function as (8) and the critic NN which is updated by (27) and (28), the local containment control protocol (24) can guarantee the closed-loop containment error system (4) to be UUB.

Proof of Theorem 4.

The Lyapunov function candidate is chosen as

$${\mathsf{\Xi }}_i=C_i^{*}\left(e_i\right).$$

(33)

Considering (20), (21) and Assumption 3, the time derivative of (33) corresponding to (4) is

$$\begin{array}{cc}\hfill {\dot{\mathsf{\Xi }}}_i=& {\dot{C}}_i^{*}\left(e_i\right)\hfill \\ \hfill =& \nabla C_i^{*\mathsf{T}}\left(e_i\right)\left(F_i+c_i{g}_i\left(x_i\right){\widehat{\mu }}_i+\sum _{p\in N_i}{a}_{ip}{g}_p\left(x_p\right){\widehat{\mu }}_p\right)\hfill \\ \hfill =& \nabla C_i^{*\mathsf{T}}\left(e_i\right)\left(c_i{g}_i\left(x_i\right)({\widehat{\mu }}_i-{\mu }_i^{*})+\sum _{p\in N_i}{a}_{ip}{g}_p\left(x_p\right)({\widehat{\mu }}_p-{\mu }_p^{*})\right)-e_i^{\mathsf{T}}{Q}_i{e}_i-\sum _{p\in \{N_i,i\}}{\mu }_p^{*\mathsf{T}}{R}_{ip}{\mu }_p^{*}\hfill \\ \hfill \le & ∥\nabla C_i^{*\mathsf{T}}\left(e_i\right)∥\left(∥c_i{g}_i\left(x_i\right)({\widehat{\mu }}_i-{\mu }_i^{*})∥+\sum _{p\in N_i}∥a_{ip}{g}_p\left(x_p\right)({\widehat{\mu }}_p-{\mu }_p^{*})∥\right)-{\lambda }_{min}\left(Q_i\right){\parallel e_i\parallel }^2\hfill \\ \hfill \le & \left({\overline{\sigma }}_{i\mathrm{M}}{\varphi }_{i\mathrm{M}}+{\overline{\omega }}_{i\mathrm{M}}\right)\left(c_i{\overline{g}}_{i\mathrm{M}}\parallel {\widehat{\mu }}_i-{\mu }_i^{*}\parallel +\sum _{p\in N_i}{a}_{ip}{\overline{g}}_{p\mathrm{M}}∥{\widehat{\mu }}_p-{\mu }_p^{*}∥\right)-{\lambda }_{min}\left(Q_i\right){\parallel e_i\parallel }^2.\hfill \end{array}$$

(34)

Notice that

$$\begin{array}{cc}\hfill \parallel {\widehat{\mu }}_i-{\mu }_i^{*}\parallel =& \parallel -\frac{1}{2}{\mathcal{R}}_{ii}^{-1}{c}_i{g}_i^{\mathsf{T}}\left(x_i\right)\nabla {\sigma }_i^{\mathsf{T}}\left(e_i\right){\widehat{\varphi }}_i+\frac{1}{2}{\mathcal{R}}_{ii}^{-1}{c}_i{g}_i^{\mathsf{T}}\left(x_i\right)\left(\nabla {\sigma }_i^{\mathsf{T}}\left(e_i\right){\varphi }_i^{*}+\nabla {\omega }_i\left(e_i\right)\right)\parallel \hfill \\ \hfill =& \parallel \frac{1}{2}{\mathcal{R}}_{ii}^{-1}{c}_i{g}_i^{\mathsf{T}}\left(x_i\right)\left(\nabla {\sigma }_i^{\mathsf{T}}\left(e_i\right){\tilde{\varphi }}_i+\nabla {\omega }_i\left(e_i\right)\right)\parallel \hfill \\ \hfill \le & \frac{c_i{\overline{g}}_{iM}}{2\parallel {\mathcal{R}}_{ii}\parallel }\left({\overline{\sigma }}_{i\mathrm{M}}{\overline{\varphi }}_{i\mathrm{M}}+{\overline{\omega }}_{i\mathrm{M}}\right).\hfill \end{array}$$

Then, (34) becomes

$$\begin{array}{cc}\hfill {\dot{\mathsf{\Xi }}}_i\le & \left({\overline{\sigma }}_{i\mathrm{M}}{\varphi }_{i\mathrm{M}}+{\overline{\omega }}_{i\mathrm{M}}\right)\left(\frac{c_i^2{\overline{g}}_{iM}^2}{2\parallel {\mathcal{R}}_{ii}\parallel }\left({\overline{\sigma }}_{i\mathrm{M}}{\overline{\varphi }}_{i\mathrm{M}}+{\overline{\omega }}_{i\mathrm{M}}\right)+\sum _{p\in N_i}\frac{c_p{a}_{ip}{\overline{g}}_{pM}^2}{2\parallel {\mathcal{R}}_{pp}\parallel }\left({\overline{\sigma }}_{p\mathrm{M}}{\overline{\varphi }}_{p\mathrm{M}}+{\overline{\omega }}_{p\mathrm{M}}\right)\right)\hfill \\ \hfill & -{\lambda }_{min}\left(Q_i\right){\parallel e_i\parallel }^2.\hfill \end{array}$$

(35)

Let ${\mathsf{\Pi }}_{i1}=\frac{c_i^2{\overline{g}}_{iM}^2}{2\parallel R_{ii}\parallel }\left({\overline{\sigma }}_{i\mathrm{M}}{\overline{\varphi }}_{i\mathrm{M}}+{\overline{\omega }}_{i\mathrm{M}}\right)+{\sum }_{p\in N_i}\frac{c_p{a}_{ip}{\overline{g}}_{pM}^2}{2\parallel R_{pp}\parallel }$$\left({\overline{\sigma }}_{p\mathrm{M}}{\overline{\varphi }}_{p\mathrm{M}}+{\overline{\omega }}_{p\mathrm{M}}\right)$. Thus, (35) turns to

$$\begin{array}{cc}\hfill {\dot{\mathsf{\Xi }}}_i\le & \underset{{\mathsf{\Pi }}_{i2}}{\underbrace{\left({\overline{\sigma }}_{i\mathrm{M}}{\varphi }_{i\mathrm{M}}+{\overline{\omega }}_{i\mathrm{M}}\right){\mathsf{\Pi }}_{i1}}}-{\lambda }_{min}\left(Q_i\right){\parallel e_i\parallel }^2\hfill \\ \hfill =& {\mathsf{\Pi }}_{i2}-{\lambda }_{min}\left(Q_i\right){\parallel e_i\parallel }^2.\hfill \end{array}$$

It shows ${\dot{\mathcal{L}}}_{i2}<0$ if $e_i$ lies outside the compact set

$${\mathsf{\Omega }}_{e_i}=\left\{e_i:\parallel e_i\parallel \le \sqrt{\frac{{\mathsf{\Pi }}_{i2}}{{\lambda }_{min}\left(Q_i\right)}}\right\}.$$

Therefore, the closed-loop containment error system (4) is UUB under the local containment control protocol (24). □

Remark 2.

In Assumption 1, we know that the nonlinear functions $f\left(x\right)$ and $g_i\left(x\right)$ are Lipschitz continuous on a compact set ${\mathsf{\Omega }}_i$ containing the origin, $f\left(0\right)=0$. It indicates that the developed control scheme is effective in a compact set ${\mathsf{\Omega }}_i$. If the system states are outside this compact set, this scheme might be invalid. In Theorem 4, we analyzed the system stability within such a compact set via the Lyapunov direct method, which means the closed-loop system is stable in the compact set under the developed IRL-based OCC scheme.

4. Simulation Study

This section provides two simulation examples to support the developed IRL-based OCC scheme.

4.1. Example 1

Consider a six-node graph network connected by three leader nodes. The directed topology of the graph is displayed in Figure 2.

As displayed in Figure 2, nodes 1–3 stand for the leaders 1–3 and nodes 4–6 represent the followers 1–3. In (3), the edge weights and pinning gains were set to 0.5. The node dynamic of the jth leader is described as ${\dot{r}}_j=\overline{A}{r}_j$, where $r_j={[r_{j1},r_{j2}]}^{\mathsf{T}}\in {\mathbb{R}}^2$ represents the state vector, $j=1,2,3$ and

$$\overline{A}=\left[\begin{array}{cc}0.1& -1\\ 1& -0.1\end{array}\right].$$

For the ith follower, the node dynamic is formulated as ${\dot{x}}_i=\overline{A}{x}_i+{\overline{B}}_i{\mu }_i$, where $x_i={[x_{i1},x_{i2}]}^{\mathsf{T}}\in {\mathbb{R}}^2$ and ${\mu }_i\in \mathbb{R}$ with $i=1,2,3$, ${\overline{B}}_1={[-1.5,1]}^{\mathsf{T}}$, ${\overline{B}}_2={[-1,1]}^{\mathsf{T}}$ and ${\overline{B}}_3={[-1,-0.5]}^{\mathsf{T}}$. The local neighborhood containment error vector $e_i={[e_{i1},e_{i2}]}^{\mathsf{T}}\in {\mathbb{R}}^2$ is calculated by (3).

In the simulation, $C_i\left(e_i\right)$ was reconstructed by a critic NN with a 2–5–1 structure. The activation function was described as ${\sigma }_i\left(e_i\right)={[e_{i1}^2,e_{i1}{e}_{i2},e_{i2}^2,e_{i1}^2{e}_{i2},e_{i2}^2{e}_{i1}]}^{\mathsf{T}}$. The initialization of the node dynamics were characterized as $x_1\left(0\right)={[0.50,-1.00]}^{\mathsf{T}}$, $x_2\left(0\right)={[1.00,-0.50]}^{\mathsf{T}}$, $x_3\left(0\right)={[0.80,-0.30]}^{\mathsf{T}}$, $r_1\left(0\right)={[0.62,0.83]}^{\mathsf{T}}$, $r_2\left(0\right)={[0.45,0.40]}^{\mathsf{T}}$ and $r_3\left(0\right)={[0.30,0.22]}^{\mathsf{T}}$. The related parameters were chosen as $Q_i=5I_2$, $R_{ip}=R_{ii}=1$, $l_{ci}=0.1$ and $l_{si}=0.1$.

The simulation results are shown in Figure 3, Figure 4 and Figure 5 using the developed IRL-based OCC protocols. The evolution procedure of the local neighborhood containment errors for triple followers is shown in Figure 3, which indicates that the local neighborhood containment errors were regulated to zero under the developed control protocols. Thus, the containment control of MAS could be reached. Figure 4 and Figure 5 depict the state curves of the leaders and the followers, where all followers moved and stayed within the region formed by the envelope curves. It implies that the satisfactory performance of the containment control was acquired. The state curves of the followers and the leaders are displayed as 2-D phase plane plot in Figure 6 and the region enveloped by the three leaders ${\upsilon }_1,{\upsilon }_2$ and ${\upsilon }_3$ is shown at three different instants ($t=16.0\mathrm{s},20.3\mathrm{s}$ and $25.0\mathrm{s}$). We can observe from Figure 6 that the followers converged to the convex hull.

4.2. Example 2

Consider the nonlinear MAS consisting of three single-link robot arms and triple leader nodes. A rigid link is attached to each robot arm via a gear train to a direct current motor [28]. In Figure 2, the directed topology among these robot arms is shown. We chose the values of all edge weights and pinning gain as 1.

The state trajectories of the leaders is given by $r_1={[0.6sin\left(t\right),0.6cos\left(t\right)]}^{\mathsf{T}}$, $r_2={[0.4sin(t+\frac{\pi }{6}),0.4cos(t+\frac{\pi }{6})]}^{\mathsf{T}}$ and $r_3={[0.2sin(t-\frac{\pi }{6}),0.2cos(t-\frac{\pi }{6})]}^{\mathsf{T}}$. The single-link robot arm for each follower can be described as

$$\mathcal{J}{\ddot{z}}_i+\overline{B}{\dot{z}}_i+\overline{M}glsin\left(z_i\right)=u_i,$$

(36)

where $\mathcal{J}=9\mathrm{kg}·\mathrm{m}{}^2$, $\overline{B}=30.5$, $\overline{M}=1\mathrm{kg}$, $l=1\mathrm{m}$, $g=9.8\mathrm{m}/\mathrm{s}{}^2$ and $i=1,2,3$. The notations of the model (36) are defined in

Table 1. Notations of the single-link robot arm.

Symbol	Notation
$z_i$	Link angle
${\dot{z}}_i$	Angular velocity of the link
$\mathcal{J}$	Total rotational inertia of the link and motor
$\overline{B}$	Overall damping coefficient
$\overline{M}$	Total mass of the link
l	Distant from joint axis to mass center of the link
$u_i$	Command generator

.

Define $x_i={[x_{i1},x_{i2}]}^{\mathsf{T}}={[z_i,{\dot{z}}_i]}^{\mathsf{T}}\in {\mathbb{R}}^2$ and ${\mu }_i=u_i$. For the ith follower, the model (36) can be rewritten as

$$\begin{array}{cc}\hfill \left[\begin{array}{c}{\dot{x}}_{i1}\\ {\dot{x}}_{i2}\end{array}\right]=& \left[\begin{array}{c}{x}_{i2}\\ -\frac{\overline{M}gl}{\mathcal{J}}sin\left(x_{i1}\right)-\frac{\overline{B}}{\mathcal{J}}{x}_{i2}\end{array}\right]+\left[\begin{array}{c}0\\ \frac{1}{\mathcal{J}}\end{array}\right]{\mu }_i.\hfill \end{array}$$

(37)

Similar to Example Section 4.1, the local neighborhood containment error vector was given as $e_i={[e_{i1},e_{i2}]}^{\mathsf{T}}\in {\mathbb{R}}^2$.

The critic NN structures and the related activation functions were initialized as in Example Section 4.1. The critic NN weights were initialized as the random values within $(0,36)$ and the parameters of initialization and control were chosen as $r_1\left(0\right)={[0,0.6]}^{\mathsf{T}}$, $r_2\left(0\right)={[0.4sin\left(\frac{\pi }{6}\right),0.4cos\left(\frac{\pi }{6}\right)]}^{\mathsf{T}}$, $r_3\left(0\right)={[0.2sin(-\frac{\pi }{6}),0.2cos(-\frac{\pi }{6})]}^{\mathsf{T}}$, $x_1\left(0\right)={[0.8,0.1]}^{\mathsf{T}}$, $x_2\left(0\right)={[0.6,0.5]}^{\mathsf{T}}$, $x_3\left(0\right)={[0.7,-0.3]}^{\mathsf{T}}$, $Q_{ip}=18I_n$, $R_{ip}=5$, $t_{\tau }=0.1\mathrm{s}$, $l_{ci}=0.1$ and $l_{si}=0.1$.

Figure 7, Figure 8, Figure 9, Figure 10 and Figure 11 show the simulation results. The local neighborhood containment errors converged to a small region around zero as depicted in Figure 7, which shows that the containment control of the nonlinear MAS was achieved. In Figure 8 and Figure 9, it can be found that the state trajectories of single-link robot arms (36) entered and stayed within the region enveloped by the leader nodes as the time progressed, which indicated the satisfactory performance of the developed scheme. The evolution curves of all agents are illustrated as the 2-D phase plane plot in Figure 10. We can see that the convex hull formed by the leaders ${\upsilon }_1,{\upsilon }_2$ and ${\upsilon }_3$ contains the followers at the time instants $t=5.0\mathrm{s},10.0\mathrm{s},14.5\mathrm{s}$ and $26.0\mathrm{s}$, which implies that the followers converged to the convex hull. Figure 11 describes the curves of the containment control inputs, which shows the regulation process of the containment error system.

5. Conclusions

This paper investigated the OCC problem of nonlinear MASs with partially unknown dynamics via the IRL method. Based on the IRL method, the integral Bellman equation was constructed to relax the requirement of the drift dynamics. The proposed control algorithm was guaranteed to converge by analyzing the convergence of IRL. With the aid of the universal approximation capability of the NN, the solution of the HJBE was acquired by a critic NN with a modified weight-updating law which guaranteed the asymptotical stability of the weight error dynamics. By using the Lyapunov stability theorem, we showed that the closed-loop containment error system was UUB. From the simulation results of two examples, the effectiveness of the proposed IRL-based OCC scheme was illustrated. In the considered MASs, the information among all agents was transmitted by a desired communication network, which is always confronted with some security issues, such as attacks and packet dropouts. The focus of our future work is to develop a novel distributed resilient containment control for the MASs subjected to attacks and packet dropouts.