Zero-Shot Sim2Real Transfer of Deep Reinforcement Learning Controller for Tower Crane System

Mohammed B. Mohiuddin; Abdel Gafoor Haddad; Igor Boiko; Yahya Zweiri

doi:10.1016/j.ifacol.2023.10.867

Zero-Shot Sim2Real Transfer of Deep Reinforcement Learning Controller for Tower Crane System

Mohammed B. Mohiuddin^*
, Abdel Gafoor Haddad
, Igor Boiko
, Yahya Zweiri

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

11 Scopus citations

Abstract

Control of nonlinear systems is a challenging task that often requires linearization, which limits the operating envelope. Moreover, designing a controller for such nonlinear systems requires complex tuning rules and expert system knowledge. Deep Reinforcement Learning (DRL) has been used as a controller for nonlinear systems. However, its applicability is hindered by the Sim2Real gap, an issue where the DRL agent trained in simulation does not transfer to reality. In this work, we propose using DRL for the control of nonlinear systems across the Sim2Real gap by utilizing a detailed dynamic model of the nonlinear system. To demonstrate the proposed methodology, we use a tower crane as the nonlinear system of interest. The control policy of the DRL agent is learned rather than explicitly programmed. To highlight the effectiveness of the proposed DRL-based controller, the results are compared with the PI controller. The experimental results presented demonstrate the successful Sim2Real transfer and effectiveness of the proposed approach.

Original language	English
Title of host publication	IFAC-PapersOnLine
Editors	Hideaki Ishii, Yoshio Ebihara, Jun-ichi Imura, Masaki Yamakita
Publisher	Elsevier B.V.
Pages	10016-10020
Number of pages	5
Edition	2
ISBN (Electronic)	9781713872344
DOIs	https://doi.org/10.1016/j.ifacol.2023.10.867
State	Published - 1 Jul 2023
Externally published	Yes
Event	22nd IFAC World Congress - Yokohama, Japan Duration: 9 Jul 2023 → 14 Jul 2023

Publication series

Name	IFAC-PapersOnLine
Number	2
Volume	56
ISSN (Electronic)	2405-8963

Conference

Conference	22nd IFAC World Congress
Country/Territory	Japan
City	Yokohama
Period	9/07/23 → 14/07/23

Bibliographical note

Publisher Copyright:
Copyright © 2023 The Authors. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/)

Keywords

Control of constrained systems
Mechatronic systems
Reinforcement learning and deep learning in control
Simulation to reality transfer

ASJC Scopus subject areas

Control and Systems Engineering

Access to Document

10.1016/j.ifacol.2023.10.867

Cite this

@inproceedings{29246a00cabc464a88c58ee957f939c8,

title = "Zero-Shot Sim2Real Transfer of Deep Reinforcement Learning Controller for Tower Crane System",

abstract = "Control of nonlinear systems is a challenging task that often requires linearization, which limits the operating envelope. Moreover, designing a controller for such nonlinear systems requires complex tuning rules and expert system knowledge. Deep Reinforcement Learning (DRL) has been used as a controller for nonlinear systems. However, its applicability is hindered by the Sim2Real gap, an issue where the DRL agent trained in simulation does not transfer to reality. In this work, we propose using DRL for the control of nonlinear systems across the Sim2Real gap by utilizing a detailed dynamic model of the nonlinear system. To demonstrate the proposed methodology, we use a tower crane as the nonlinear system of interest. The control policy of the DRL agent is learned rather than explicitly programmed. To highlight the effectiveness of the proposed DRL-based controller, the results are compared with the PI controller. The experimental results presented demonstrate the successful Sim2Real transfer and effectiveness of the proposed approach.",

keywords = "Control of constrained systems, Mechatronic systems, Reinforcement learning and deep learning in control, Simulation to reality transfer",

author = "Mohiuddin, \{Mohammed B.\} and Haddad, \{Abdel Gafoor\} and Igor Boiko and Yahya Zweiri",

note = "Publisher Copyright: Copyright {\textcopyright} 2023 The Authors. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/); 22nd IFAC World Congress ; Conference date: 09-07-2023 Through 14-07-2023",

year = "2023",

month = jul,

day = "1",

doi = "10.1016/j.ifacol.2023.10.867",

language = "English",

series = "IFAC-PapersOnLine",

publisher = "Elsevier B.V.",

number = "2",

pages = "10016--10020",

editor = "Hideaki Ishii and Yoshio Ebihara and Jun-ichi Imura and Masaki Yamakita",

booktitle = "IFAC-PapersOnLine",

edition = "2",

}

Mohiuddin, MB, Haddad, AG, Boiko, I & Zweiri, Y 2023, Zero-Shot Sim2Real Transfer of Deep Reinforcement Learning Controller for Tower Crane System. in H Ishii, Y Ebihara, J Imura & M Yamakita (eds), IFAC-PapersOnLine. 2 edn, IFAC-PapersOnLine, no. 2, vol. 56, Elsevier B.V., pp. 10016-10020, 22nd IFAC World Congress, Yokohama, Japan, 9/07/23. https://doi.org/10.1016/j.ifacol.2023.10.867

Zero-Shot Sim2Real Transfer of Deep Reinforcement Learning Controller for Tower Crane System. / Mohiuddin, Mohammed B.; Haddad, Abdel Gafoor; Boiko, Igor et al.
IFAC-PapersOnLine. ed. / Hideaki Ishii; Yoshio Ebihara; Jun-ichi Imura; Masaki Yamakita. 2. ed. Elsevier B.V., 2023. p. 10016-10020 (IFAC-PapersOnLine; Vol. 56, No. 2).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Zero-Shot Sim2Real Transfer of Deep Reinforcement Learning Controller for Tower Crane System

AU - Mohiuddin, Mohammed B.

AU - Haddad, Abdel Gafoor

AU - Boiko, Igor

AU - Zweiri, Yahya

PY - 2023/7/1

Y1 - 2023/7/1

N2 - Control of nonlinear systems is a challenging task that often requires linearization, which limits the operating envelope. Moreover, designing a controller for such nonlinear systems requires complex tuning rules and expert system knowledge. Deep Reinforcement Learning (DRL) has been used as a controller for nonlinear systems. However, its applicability is hindered by the Sim2Real gap, an issue where the DRL agent trained in simulation does not transfer to reality. In this work, we propose using DRL for the control of nonlinear systems across the Sim2Real gap by utilizing a detailed dynamic model of the nonlinear system. To demonstrate the proposed methodology, we use a tower crane as the nonlinear system of interest. The control policy of the DRL agent is learned rather than explicitly programmed. To highlight the effectiveness of the proposed DRL-based controller, the results are compared with the PI controller. The experimental results presented demonstrate the successful Sim2Real transfer and effectiveness of the proposed approach.

AB - Control of nonlinear systems is a challenging task that often requires linearization, which limits the operating envelope. Moreover, designing a controller for such nonlinear systems requires complex tuning rules and expert system knowledge. Deep Reinforcement Learning (DRL) has been used as a controller for nonlinear systems. However, its applicability is hindered by the Sim2Real gap, an issue where the DRL agent trained in simulation does not transfer to reality. In this work, we propose using DRL for the control of nonlinear systems across the Sim2Real gap by utilizing a detailed dynamic model of the nonlinear system. To demonstrate the proposed methodology, we use a tower crane as the nonlinear system of interest. The control policy of the DRL agent is learned rather than explicitly programmed. To highlight the effectiveness of the proposed DRL-based controller, the results are compared with the PI controller. The experimental results presented demonstrate the successful Sim2Real transfer and effectiveness of the proposed approach.

KW - Control of constrained systems

KW - Mechatronic systems

KW - Reinforcement learning and deep learning in control

KW - Simulation to reality transfer

UR - https://www.scopus.com/pages/publications/85178163505

U2 - 10.1016/j.ifacol.2023.10.867

DO - 10.1016/j.ifacol.2023.10.867

M3 - Conference contribution

AN - SCOPUS:85178163505

T3 - IFAC-PapersOnLine

SP - 10016

EP - 10020

BT - IFAC-PapersOnLine

A2 - Ishii, Hideaki

A2 - Ebihara, Yoshio

A2 - Imura, Jun-ichi

A2 - Yamakita, Masaki

PB - Elsevier B.V.

T2 - 22nd IFAC World Congress

Y2 - 9 July 2023 through 14 July 2023

ER -

Zero-Shot Sim2Real Transfer of Deep Reinforcement Learning Controller for Tower Crane System

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Fingerprint

Cite this