The Role of Attention Mechanism in Generating Image Captions: An Innovative Approach with Neural Network-Based Seq2seq Model

Abstract

This study addresses important contributions to generating text from images, aiming to create meaning in various fields such as entertainment, communication, commerce, security, and education by establishing a connection between visual and textual content. This process aims to increase the accessibility, understandability, and processability of content by converting image data into meaningful text. Therefore, advances and studies in this field are extremely important. This study focuses on the effect of the combination of deep neural network models and attention mechanisms in creating more meaningful captions from images. Experiments performed on the Flickr8k dataset highlight the abilities of Seq2seq and VGG19 models to generate titles compatible with reference sentences. By using the dynamic focusing feature of the attention mechanism, the model effectively captures detailed aspects of images. The findings of this study have the potential to push the boundaries of multimodal data processing and representation with the effective integration of visual and textual information by adding information that the attention mechanism works more effectively together with the Seq2seq model. © 2024, Sakarya University. All rights reserved.