A systematic review of trimodal affective computing approaches: Text, audio, and visual integration in emotion recognition and sentiment analysis

Abstract

At the heart of affective computing lies the crucial task of decoding human emotions, a field that expertly intertwines emotion identification with the nuances of sentiment analysis. This dynamic discipline harnesses an array of data sources, from the intricacies of textual information to the subtleties of auditory signals and the dynamic realm of visual cues. One of its primary challenges is discerning emotions from physical cues like facial expressions and vocal tones, especially when these emotions are subtly concealed. The precise information yielded by physiological signals is invaluable, yet the complexity of their acquisition in real-world settings remains a formidable challenge. Our comprehensive systematic review marks a significant foray into trimodal affective computing, integrating text, audio, and visual data to provide a holistic understanding. We analyzed over 410 research articles from prominent conferences and journals spanning the last two decades. This extensive study categorizes and critically evaluates a spectrum of affect recognition methods, from unimodal to multimodal approaches, including bimodal and trimodal, offering profound insights into their structural composition and practical effectiveness. In concluding our exploration, we highlight the pivotal aspects of affective computing and chart a course for future groundbreaking research. This includes refining data integration techniques, overcoming challenges in emotion recognition, and addressing the critical ethical dimensions inherent in this field.