A soft voting ensemble learning-based approach for multimodal sentiment analysis

Abstract

It is possible to determine people's feelings and opinions about a subject or product from social media posts via sentiment analysis. With the pervasive usage of the Internet and smart devices, the data produced daily by users can be in different modalities such as text, image, audio, and video. Multimodal sentiment analysis is to reveal the feeling of the user's posts by analyzing the data in different modalities as a whole. One of the major challenges of multimodal sentiment is how the sentiment obtained on different modalities is combined to ensure sentiment and meaning integrity of the post. Also, many studies use the same classifying methods to analyze different modalities. In fact, each classifier can be effective in different feature sets. In this study, a soft voting-based ensemble model is proposed that takes advantage of the effective performance of different classifiers on different modalities. In the proposed model, deep feature extraction was made with deep learning methods (BiLSTM, CNN) from the multimodal datasets. After the feature selection was conducted on the features which are a fusion of text and image features, the final feature sets were classified with the soft voting-based ensemble learning model. The performance of the proposed model has been tested on two different benchmark datasets consisting of text-image pairs. As a result of the experimental studies, it was revealed that the proposed model outperformed multiple adversary models on the same datasets.