Towards Machine Learning-Based Emotion Recognition from Multimodal Data

Md Faiyaz Shahriar; Md Safkat Azad Arnab; Munia Sarwat Khan; Safwon Sadif Rahman; Mufti Mahmud; M. Shamim Kaiser

doi:10.1007/978-981-19-5191-6_9

Towards Machine Learning-Based Emotion Recognition from Multimodal Data

Md Faiyaz Shahriar, Md Safkat Azad Arnab, Munia Sarwat Khan, Safwon Sadif Rahman, Mufti Mahmud^*, M. Shamim Kaiser

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

8 Scopus citations

Abstract

Understanding human emotion is vital to communicate effectively with others, monitor patients, analyse behaviour, and keep an eye on those who are vulnerable. Emotion recognition is essential to achieve a complete human-machine interoperability experience. Artificial intelligence, mainly machine learning (ML), have been used in recent years to improve the model for recognising emotions from a single type of data. A multimodal system has been proposed that uses text, facial expressions, and speech signals to identify emotions in this work. The MobileNet architecture is used to predict emotion from facial expressions, and different ML classifiers are used to predict emotion from text and speech signals in the proposed model. The Facial Expression Recognition 2013 (FER2013) dataset has been used to recognise emotion from facial expressions, whilst the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset was used for both text and speech emotion recognition. The proposed ensemble technique consisting of random forest, extreme gradient boosting, and multi-layer perceptron achieves an accuracy of 70.67%, which is better than the unimodal approaches used.

Original language	English
Title of host publication	Frontiers of ICT in Healthcare - Proceedings of EAIT 2022
Editors	Jyotsna Kumar Mandal, Debashis De
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	99-109
Number of pages	11
ISBN (Print)	9789811951909
DOIs	https://doi.org/10.1007/978-981-19-5191-6_9
State	Published - 2023
Externally published	Yes
Event	7th International Conference on Emerging Applications of Information Technology, EAIT 2022 - kolkata, India Duration: 27 Mar 2022 → 28 Mar 2022

Publication series

Name	Lecture Notes in Networks and Systems
Volume	519 LNNS
ISSN (Print)	2367-3370
ISSN (Electronic)	2367-3389

Conference

Conference	7th International Conference on Emerging Applications of Information Technology, EAIT 2022
Country/Territory	India
City	kolkata
Period	27/03/22 → 28/03/22

Bibliographical note

Publisher Copyright:
© 2023, The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

Keywords

Emotion recognition
Machine learning
MobileNet
Multimodal
Prediction

ASJC Scopus subject areas

Control and Systems Engineering
Signal Processing
Computer Networks and Communications

Access to Document

10.1007/978-981-19-5191-6_9

Cite this

Shahriar, M. F., Arnab, M. S. A., Khan, M. S., Rahman, S. S., Mahmud, M., & Kaiser, M. S. (2023). Towards Machine Learning-Based Emotion Recognition from Multimodal Data. In J. K. Mandal, & D. De (Eds.), Frontiers of ICT in Healthcare - Proceedings of EAIT 2022 (pp. 99-109). (Lecture Notes in Networks and Systems; Vol. 519 LNNS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-981-19-5191-6_9

@inproceedings{1d305ec76040454c8afb4b2830626a92,

title = "Towards Machine Learning-Based Emotion Recognition from Multimodal Data",

abstract = "Understanding human emotion is vital to communicate effectively with others, monitor patients, analyse behaviour, and keep an eye on those who are vulnerable. Emotion recognition is essential to achieve a complete human-machine interoperability experience. Artificial intelligence, mainly machine learning (ML), have been used in recent years to improve the model for recognising emotions from a single type of data. A multimodal system has been proposed that uses text, facial expressions, and speech signals to identify emotions in this work. The MobileNet architecture is used to predict emotion from facial expressions, and different ML classifiers are used to predict emotion from text and speech signals in the proposed model. The Facial Expression Recognition 2013 (FER2013) dataset has been used to recognise emotion from facial expressions, whilst the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset was used for both text and speech emotion recognition. The proposed ensemble technique consisting of random forest, extreme gradient boosting, and multi-layer perceptron achieves an accuracy of 70.67\%, which is better than the unimodal approaches used.",

keywords = "Emotion recognition, Machine learning, MobileNet, Multimodal, Prediction",

author = "Shahriar, \{Md Faiyaz\} and Arnab, \{Md Safkat Azad\} and Khan, \{Munia Sarwat\} and Rahman, \{Safwon Sadif\} and Mufti Mahmud and Kaiser, \{M. Shamim\}",

note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.; 7th International Conference on Emerging Applications of Information Technology, EAIT 2022 ; Conference date: 27-03-2022 Through 28-03-2022",

year = "2023",

doi = "10.1007/978-981-19-5191-6\_9",

language = "English",

isbn = "9789811951909",

series = "Lecture Notes in Networks and Systems",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "99--109",

editor = "Mandal, \{Jyotsna Kumar\} and Debashis De",

booktitle = "Frontiers of ICT in Healthcare - Proceedings of EAIT 2022",

address = "Germany",

}

Shahriar, MF, Arnab, MSA, Khan, MS, Rahman, SS, Mahmud, M & Kaiser, MS 2023, Towards Machine Learning-Based Emotion Recognition from Multimodal Data. in JK Mandal & D De (eds), Frontiers of ICT in Healthcare - Proceedings of EAIT 2022. Lecture Notes in Networks and Systems, vol. 519 LNNS, Springer Science and Business Media Deutschland GmbH, pp. 99-109, 7th International Conference on Emerging Applications of Information Technology, EAIT 2022, kolkata, India, 27/03/22. https://doi.org/10.1007/978-981-19-5191-6_9

Towards Machine Learning-Based Emotion Recognition from Multimodal Data. / Shahriar, Md Faiyaz; Arnab, Md Safkat Azad; Khan, Munia Sarwat et al.
Frontiers of ICT in Healthcare - Proceedings of EAIT 2022. ed. / Jyotsna Kumar Mandal; Debashis De. Springer Science and Business Media Deutschland GmbH, 2023. p. 99-109 (Lecture Notes in Networks and Systems; Vol. 519 LNNS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Towards Machine Learning-Based Emotion Recognition from Multimodal Data

AU - Shahriar, Md Faiyaz

AU - Arnab, Md Safkat Azad

AU - Khan, Munia Sarwat

AU - Rahman, Safwon Sadif

AU - Mahmud, Mufti

AU - Kaiser, M. Shamim

PY - 2023

Y1 - 2023

N2 - Understanding human emotion is vital to communicate effectively with others, monitor patients, analyse behaviour, and keep an eye on those who are vulnerable. Emotion recognition is essential to achieve a complete human-machine interoperability experience. Artificial intelligence, mainly machine learning (ML), have been used in recent years to improve the model for recognising emotions from a single type of data. A multimodal system has been proposed that uses text, facial expressions, and speech signals to identify emotions in this work. The MobileNet architecture is used to predict emotion from facial expressions, and different ML classifiers are used to predict emotion from text and speech signals in the proposed model. The Facial Expression Recognition 2013 (FER2013) dataset has been used to recognise emotion from facial expressions, whilst the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset was used for both text and speech emotion recognition. The proposed ensemble technique consisting of random forest, extreme gradient boosting, and multi-layer perceptron achieves an accuracy of 70.67%, which is better than the unimodal approaches used.

AB - Understanding human emotion is vital to communicate effectively with others, monitor patients, analyse behaviour, and keep an eye on those who are vulnerable. Emotion recognition is essential to achieve a complete human-machine interoperability experience. Artificial intelligence, mainly machine learning (ML), have been used in recent years to improve the model for recognising emotions from a single type of data. A multimodal system has been proposed that uses text, facial expressions, and speech signals to identify emotions in this work. The MobileNet architecture is used to predict emotion from facial expressions, and different ML classifiers are used to predict emotion from text and speech signals in the proposed model. The Facial Expression Recognition 2013 (FER2013) dataset has been used to recognise emotion from facial expressions, whilst the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset was used for both text and speech emotion recognition. The proposed ensemble technique consisting of random forest, extreme gradient boosting, and multi-layer perceptron achieves an accuracy of 70.67%, which is better than the unimodal approaches used.

KW - Emotion recognition

KW - Machine learning

KW - MobileNet

KW - Multimodal

KW - Prediction

UR - http://www.scopus.com/inward/record.url?scp=85161422300&partnerID=8YFLogxK

U2 - 10.1007/978-981-19-5191-6_9

DO - 10.1007/978-981-19-5191-6_9

M3 - Conference contribution

AN - SCOPUS:85161422300

SN - 9789811951909

T3 - Lecture Notes in Networks and Systems

SP - 99

EP - 109

BT - Frontiers of ICT in Healthcare - Proceedings of EAIT 2022

A2 - Mandal, Jyotsna Kumar

A2 - De, Debashis

PB - Springer Science and Business Media Deutschland GmbH

T2 - 7th International Conference on Emerging Applications of Information Technology, EAIT 2022

Y2 - 27 March 2022 through 28 March 2022

ER -

Shahriar MF, Arnab MSA, Khan MS, Rahman SS, Mahmud M, Kaiser MS. Towards Machine Learning-Based Emotion Recognition from Multimodal Data. In Mandal JK, De D, editors, Frontiers of ICT in Healthcare - Proceedings of EAIT 2022. Springer Science and Business Media Deutschland GmbH. 2023. p. 99-109. (Lecture Notes in Networks and Systems). doi: 10.1007/978-981-19-5191-6_9