Hand-Object Interaction Detection based on Visual Attention for Independent Rehabilitation Support

Adnan Rachmat Anom Besari; Azhar Aulia Saputra; Wei Hong Chin; Naoyuki Kubota; Kurnianingsih

doi:10.1109/IJCNN55064.2022.9892903

Hand-Object Interaction Detection based on Visual Attention for Independent Rehabilitation Support

Adnan Rachmat Anom Besari
, Azhar Aulia Saputra
, Wei Hong Chin
, Naoyuki Kubota
, Kurnianingsih

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

6 Scopus citations

Abstract

Hand rehabilitation in post-stroke patients with visual impairment is currently not supported by the availability of a cyber-physical-social system (CPSS) that can monitor physical development during daily activities. This paper discusses how to extract hand activity information on objects based on visual attention in the task-specific reach-to-grasp cycle. We used perception-based egocentric vision to observe hand-object interactions (HOI) in grasping tasks. Our approach combines object detection with hand skeletal model estimation and visual attention to validate HOI detection. We choose a multilayer Gated Recurrent Unit (GRU) based on Recurrent Neural Networks (RNN) architecture to classify the four main activities when the hand interacts with an object (wonder-reach-grasp-release). We evaluated the algorithm quantitatively on the new dataset we introduced for cup grasping activity. This method can validate the HOI detection with 97.0% precision with less training time for small data. Further research will use these results to increase self-efficacy for independent hand-eye coordination rehabilitation support in community-centric systems. The code and dataset are available at https://github.com/anom-tμhoi-attention/.

Original language	English
Title of host publication	2022 International Joint Conference on Neural Networks, IJCNN 2022 - Proceedings
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781728186719
DOIs	https://doi.org/10.1109/IJCNN55064.2022.9892903
State	Published - 2022
Externally published	Yes
Event	2022 International Joint Conference on Neural Networks, IJCNN 2022 - Padua, Italy Duration: 18 Jul 2022 → 23 Jul 2022

Publication series

Name	Proceedings of the International Joint Conference on Neural Networks
ISSN (Print)	2161-4393
ISSN (Electronic)	2161-4407

Conference

Conference	2022 International Joint Conference on Neural Networks, IJCNN 2022
Country/Territory	Italy
City	Padua
Period	18/07/22 → 23/07/22

Bibliographical note

Publisher Copyright:
© 2022 IEEE.

Keywords

egocentric vision
hand gesture
hand-eye coordination
post-stroke rehabilitation
reach-to-grasp cycle

ASJC Scopus subject areas

Software
Artificial Intelligence

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1109/IJCNN55064.2022.9892903

Cite this

Besari, A. R. A., Saputra, A. A., Chin, W. H., Kubota, N., & Kurnianingsih (2022). Hand-Object Interaction Detection based on Visual Attention for Independent Rehabilitation Support. In 2022 International Joint Conference on Neural Networks, IJCNN 2022 - Proceedings (Proceedings of the International Joint Conference on Neural Networks). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IJCNN55064.2022.9892903

Besari, Adnan Rachmat Anom ; Saputra, Azhar Aulia ; Chin, Wei Hong et al. / Hand-Object Interaction Detection based on Visual Attention for Independent Rehabilitation Support. 2022 International Joint Conference on Neural Networks, IJCNN 2022 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2022. (Proceedings of the International Joint Conference on Neural Networks).

@inproceedings{819b478aa7e0431bae8e323545f38e14,

title = "Hand-Object Interaction Detection based on Visual Attention for Independent Rehabilitation Support",

abstract = "Hand rehabilitation in post-stroke patients with visual impairment is currently not supported by the availability of a cyber-physical-social system (CPSS) that can monitor physical development during daily activities. This paper discusses how to extract hand activity information on objects based on visual attention in the task-specific reach-to-grasp cycle. We used perception-based egocentric vision to observe hand-object interactions (HOI) in grasping tasks. Our approach combines object detection with hand skeletal model estimation and visual attention to validate HOI detection. We choose a multilayer Gated Recurrent Unit (GRU) based on Recurrent Neural Networks (RNN) architecture to classify the four main activities when the hand interacts with an object (wonder-reach-grasp-release). We evaluated the algorithm quantitatively on the new dataset we introduced for cup grasping activity. This method can validate the HOI detection with 97.0\% precision with less training time for small data. Further research will use these results to increase self-efficacy for independent hand-eye coordination rehabilitation support in community-centric systems. The code and dataset are available at https://github.com/anom-tμhoi-attention/.",

keywords = "egocentric vision, hand gesture, hand-eye coordination, post-stroke rehabilitation, reach-to-grasp cycle",

author = "Besari, \{Adnan Rachmat Anom\} and Saputra, \{Azhar Aulia\} and Chin, \{Wei Hong\} and Naoyuki Kubota and Kurnianingsih",

note = "Publisher Copyright: {\textcopyright} 2022 IEEE.; 2022 International Joint Conference on Neural Networks, IJCNN 2022 ; Conference date: 18-07-2022 Through 23-07-2022",

year = "2022",

doi = "10.1109/IJCNN55064.2022.9892903",

language = "English",

series = "Proceedings of the International Joint Conference on Neural Networks",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "2022 International Joint Conference on Neural Networks, IJCNN 2022 - Proceedings",

address = "United States",

}

Besari, ARA, Saputra, AA, Chin, WH, Kubota, N & Kurnianingsih 2022, Hand-Object Interaction Detection based on Visual Attention for Independent Rehabilitation Support. in 2022 International Joint Conference on Neural Networks, IJCNN 2022 - Proceedings. Proceedings of the International Joint Conference on Neural Networks, Institute of Electrical and Electronics Engineers Inc., 2022 International Joint Conference on Neural Networks, IJCNN 2022, Padua, Italy, 18/07/22. https://doi.org/10.1109/IJCNN55064.2022.9892903

Hand-Object Interaction Detection based on Visual Attention for Independent Rehabilitation Support. / Besari, Adnan Rachmat Anom; Saputra, Azhar Aulia; Chin, Wei Hong et al.
2022 International Joint Conference on Neural Networks, IJCNN 2022 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2022. (Proceedings of the International Joint Conference on Neural Networks).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Hand-Object Interaction Detection based on Visual Attention for Independent Rehabilitation Support

AU - Besari, Adnan Rachmat Anom

AU - Saputra, Azhar Aulia

AU - Chin, Wei Hong

AU - Kubota, Naoyuki

AU - Kurnianingsih,

PY - 2022

Y1 - 2022

N2 - Hand rehabilitation in post-stroke patients with visual impairment is currently not supported by the availability of a cyber-physical-social system (CPSS) that can monitor physical development during daily activities. This paper discusses how to extract hand activity information on objects based on visual attention in the task-specific reach-to-grasp cycle. We used perception-based egocentric vision to observe hand-object interactions (HOI) in grasping tasks. Our approach combines object detection with hand skeletal model estimation and visual attention to validate HOI detection. We choose a multilayer Gated Recurrent Unit (GRU) based on Recurrent Neural Networks (RNN) architecture to classify the four main activities when the hand interacts with an object (wonder-reach-grasp-release). We evaluated the algorithm quantitatively on the new dataset we introduced for cup grasping activity. This method can validate the HOI detection with 97.0% precision with less training time for small data. Further research will use these results to increase self-efficacy for independent hand-eye coordination rehabilitation support in community-centric systems. The code and dataset are available at https://github.com/anom-tμhoi-attention/.

AB - Hand rehabilitation in post-stroke patients with visual impairment is currently not supported by the availability of a cyber-physical-social system (CPSS) that can monitor physical development during daily activities. This paper discusses how to extract hand activity information on objects based on visual attention in the task-specific reach-to-grasp cycle. We used perception-based egocentric vision to observe hand-object interactions (HOI) in grasping tasks. Our approach combines object detection with hand skeletal model estimation and visual attention to validate HOI detection. We choose a multilayer Gated Recurrent Unit (GRU) based on Recurrent Neural Networks (RNN) architecture to classify the four main activities when the hand interacts with an object (wonder-reach-grasp-release). We evaluated the algorithm quantitatively on the new dataset we introduced for cup grasping activity. This method can validate the HOI detection with 97.0% precision with less training time for small data. Further research will use these results to increase self-efficacy for independent hand-eye coordination rehabilitation support in community-centric systems. The code and dataset are available at https://github.com/anom-tμhoi-attention/.

KW - egocentric vision

KW - hand gesture

KW - hand-eye coordination

KW - post-stroke rehabilitation

KW - reach-to-grasp cycle

UR - https://www.scopus.com/pages/publications/85140776760

U2 - 10.1109/IJCNN55064.2022.9892903

DO - 10.1109/IJCNN55064.2022.9892903

M3 - Conference contribution

AN - SCOPUS:85140776760

T3 - Proceedings of the International Joint Conference on Neural Networks

BT - 2022 International Joint Conference on Neural Networks, IJCNN 2022 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2022 International Joint Conference on Neural Networks, IJCNN 2022

Y2 - 18 July 2022 through 23 July 2022

ER -

Besari ARA, Saputra AA, Chin WH, Kubota N, Kurnianingsih. Hand-Object Interaction Detection based on Visual Attention for Independent Rehabilitation Support. In 2022 International Joint Conference on Neural Networks, IJCNN 2022 - Proceedings. Institute of Electrical and Electronics Engineers Inc. 2022. (Proceedings of the International Joint Conference on Neural Networks). doi: 10.1109/IJCNN55064.2022.9892903

Hand-Object Interaction Detection based on Visual Attention for Independent Rehabilitation Support

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

UN SDGs

Access to Document

Fingerprint

Cite this