Convolutional MKL based multimodal emotion recognition and sentiment analysis

Soujanya Poria; Iti Chaturvedi; Erik Cambria; Amir Hussain

doi:10.1109/ICDM.2016.0055

Convolutional MKL based multimodal emotion recognition and sentiment analysis

Soujanya Poria
, Iti Chaturvedi
, Erik Cambria
, Amir Hussain

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

587 Scopus citations

Abstract

Technology has enabled anyone with an Internet connection to easily create and share their ideas, opinions and content with millions of other people around the world. Much of the content being posted and consumed online is multimodal. With billions of phones, tablets and PCs shipping today with built-in cameras and a host of new video-equipped wearables like Google Glass on the horizon, the amount of video on the Internet will only continue to increase. It has become increasingly difficult for researchers to keep up with this deluge of multimodal content, let alone organize or make sense of it. Mining useful knowledge from video is a critical need that will grow exponentially, in pace with the global growth of content. This is particularly important in sentiment analysis, as both service and product reviews are gradually shifting from unimodal to multimodal. We present a novel method to extract features from visual and textual modalities using deep convolutional neural networks. By feeding such features to a multiple kernel learning classifier, we significantly outperform the state of the art of multimodal emotion recognition and sentiment analysis on different datasets.

Original language	English
Title of host publication	Proceedings - 16th IEEE International Conference on Data Mining, ICDM 2016
Editors	Francesco Bonchi, Josep Domingo-Ferrer, Ricardo Baeza-Yates, Zhi-Hua Zhou, Xindong Wu
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	439-448
Number of pages	10
ISBN (Electronic)	9781509054725
DOIs	https://doi.org/10.1109/ICDM.2016.0055
State	Published - 2 Jul 2016
Externally published	Yes
Event	16th IEEE International Conference on Data Mining, ICDM 2016 - Barcelona, Catalonia, Spain Duration: 12 Dec 2016 → 15 Dec 2016

Publication series

Name	Proceedings - IEEE International Conference on Data Mining, ICDM
Volume	0
ISSN (Electronic)	2374-8486

Conference

Conference	16th IEEE International Conference on Data Mining, ICDM 2016
Country/Territory	Spain
City	Barcelona, Catalonia
Period	12/12/16 → 15/12/16

Bibliographical note

Publisher Copyright:
© 2016 IEEE.

Keywords

Convolutional neural networks
Deep learning
Multimodal sentiment analysis
Multiple kernel learning

ASJC Scopus subject areas

General Engineering

Access to Document

10.1109/ICDM.2016.0055

Cite this

Poria, S., Chaturvedi, I., Cambria, E., & Hussain, A. (2016). Convolutional MKL based multimodal emotion recognition and sentiment analysis. In F. Bonchi, J. Domingo-Ferrer, R. Baeza-Yates, Z.-H. Zhou, & X. Wu (Eds.), Proceedings - 16th IEEE International Conference on Data Mining, ICDM 2016 (pp. 439-448). Article 7837868 (Proceedings - IEEE International Conference on Data Mining, ICDM; Vol. 0). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICDM.2016.0055

Poria, Soujanya ; Chaturvedi, Iti ; Cambria, Erik et al. / Convolutional MKL based multimodal emotion recognition and sentiment analysis. Proceedings - 16th IEEE International Conference on Data Mining, ICDM 2016. editor / Francesco Bonchi ; Josep Domingo-Ferrer ; Ricardo Baeza-Yates ; Zhi-Hua Zhou ; Xindong Wu. Institute of Electrical and Electronics Engineers Inc., 2016. pp. 439-448 (Proceedings - IEEE International Conference on Data Mining, ICDM).

@inproceedings{d9f6d299401349af9260edbbf1e32328,

title = "Convolutional MKL based multimodal emotion recognition and sentiment analysis",

abstract = "Technology has enabled anyone with an Internet connection to easily create and share their ideas, opinions and content with millions of other people around the world. Much of the content being posted and consumed online is multimodal. With billions of phones, tablets and PCs shipping today with built-in cameras and a host of new video-equipped wearables like Google Glass on the horizon, the amount of video on the Internet will only continue to increase. It has become increasingly difficult for researchers to keep up with this deluge of multimodal content, let alone organize or make sense of it. Mining useful knowledge from video is a critical need that will grow exponentially, in pace with the global growth of content. This is particularly important in sentiment analysis, as both service and product reviews are gradually shifting from unimodal to multimodal. We present a novel method to extract features from visual and textual modalities using deep convolutional neural networks. By feeding such features to a multiple kernel learning classifier, we significantly outperform the state of the art of multimodal emotion recognition and sentiment analysis on different datasets.",

keywords = "Convolutional neural networks, Deep learning, Multimodal sentiment analysis, Multiple kernel learning",

author = "Soujanya Poria and Iti Chaturvedi and Erik Cambria and Amir Hussain",

note = "Publisher Copyright: {\textcopyright} 2016 IEEE.; 16th IEEE International Conference on Data Mining, ICDM 2016 ; Conference date: 12-12-2016 Through 15-12-2016",

year = "2016",

month = jul,

day = "2",

doi = "10.1109/ICDM.2016.0055",

language = "English",

series = "Proceedings - IEEE International Conference on Data Mining, ICDM",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "439--448",

editor = "Francesco Bonchi and Josep Domingo-Ferrer and Ricardo Baeza-Yates and Zhi-Hua Zhou and Xindong Wu",

booktitle = "Proceedings - 16th IEEE International Conference on Data Mining, ICDM 2016",

address = "United States",

}

Poria, S, Chaturvedi, I, Cambria, E & Hussain, A 2016, Convolutional MKL based multimodal emotion recognition and sentiment analysis. in F Bonchi, J Domingo-Ferrer, R Baeza-Yates, Z-H Zhou & X Wu (eds), Proceedings - 16th IEEE International Conference on Data Mining, ICDM 2016., 7837868, Proceedings - IEEE International Conference on Data Mining, ICDM, vol. 0, Institute of Electrical and Electronics Engineers Inc., pp. 439-448, 16th IEEE International Conference on Data Mining, ICDM 2016, Barcelona, Catalonia, Spain, 12/12/16. https://doi.org/10.1109/ICDM.2016.0055

Convolutional MKL based multimodal emotion recognition and sentiment analysis. / Poria, Soujanya; Chaturvedi, Iti; Cambria, Erik et al.
Proceedings - 16th IEEE International Conference on Data Mining, ICDM 2016. ed. / Francesco Bonchi; Josep Domingo-Ferrer; Ricardo Baeza-Yates; Zhi-Hua Zhou; Xindong Wu. Institute of Electrical and Electronics Engineers Inc., 2016. p. 439-448 7837868 (Proceedings - IEEE International Conference on Data Mining, ICDM; Vol. 0).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Convolutional MKL based multimodal emotion recognition and sentiment analysis

AU - Poria, Soujanya

AU - Chaturvedi, Iti

AU - Cambria, Erik

AU - Hussain, Amir

PY - 2016/7/2

Y1 - 2016/7/2

N2 - Technology has enabled anyone with an Internet connection to easily create and share their ideas, opinions and content with millions of other people around the world. Much of the content being posted and consumed online is multimodal. With billions of phones, tablets and PCs shipping today with built-in cameras and a host of new video-equipped wearables like Google Glass on the horizon, the amount of video on the Internet will only continue to increase. It has become increasingly difficult for researchers to keep up with this deluge of multimodal content, let alone organize or make sense of it. Mining useful knowledge from video is a critical need that will grow exponentially, in pace with the global growth of content. This is particularly important in sentiment analysis, as both service and product reviews are gradually shifting from unimodal to multimodal. We present a novel method to extract features from visual and textual modalities using deep convolutional neural networks. By feeding such features to a multiple kernel learning classifier, we significantly outperform the state of the art of multimodal emotion recognition and sentiment analysis on different datasets.

AB - Technology has enabled anyone with an Internet connection to easily create and share their ideas, opinions and content with millions of other people around the world. Much of the content being posted and consumed online is multimodal. With billions of phones, tablets and PCs shipping today with built-in cameras and a host of new video-equipped wearables like Google Glass on the horizon, the amount of video on the Internet will only continue to increase. It has become increasingly difficult for researchers to keep up with this deluge of multimodal content, let alone organize or make sense of it. Mining useful knowledge from video is a critical need that will grow exponentially, in pace with the global growth of content. This is particularly important in sentiment analysis, as both service and product reviews are gradually shifting from unimodal to multimodal. We present a novel method to extract features from visual and textual modalities using deep convolutional neural networks. By feeding such features to a multiple kernel learning classifier, we significantly outperform the state of the art of multimodal emotion recognition and sentiment analysis on different datasets.

KW - Convolutional neural networks

KW - Deep learning

KW - Multimodal sentiment analysis

KW - Multiple kernel learning

UR - https://www.scopus.com/pages/publications/85014552954

U2 - 10.1109/ICDM.2016.0055

DO - 10.1109/ICDM.2016.0055

M3 - Conference contribution

AN - SCOPUS:85014552954

T3 - Proceedings - IEEE International Conference on Data Mining, ICDM

SP - 439

EP - 448

BT - Proceedings - 16th IEEE International Conference on Data Mining, ICDM 2016

A2 - Bonchi, Francesco

A2 - Domingo-Ferrer, Josep

A2 - Baeza-Yates, Ricardo

A2 - Zhou, Zhi-Hua

A2 - Wu, Xindong

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 16th IEEE International Conference on Data Mining, ICDM 2016

Y2 - 12 December 2016 through 15 December 2016

ER -

Poria S, Chaturvedi I, Cambria E, Hussain A. Convolutional MKL based multimodal emotion recognition and sentiment analysis. In Bonchi F, Domingo-Ferrer J, Baeza-Yates R, Zhou ZH, Wu X, editors, Proceedings - 16th IEEE International Conference on Data Mining, ICDM 2016. Institute of Electrical and Electronics Engineers Inc. 2016. p. 439-448. 7837868. (Proceedings - IEEE International Conference on Data Mining, ICDM). doi: 10.1109/ICDM.2016.0055

Convolutional MKL based multimodal emotion recognition and sentiment analysis

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Fingerprint

Cite this