Development of an Image Captioning Model to Assist The Activities of Visually Impaired Pedestrians in Urban Environments
DOI:
https://doi.org/10.26418/jp.v11i2.95754Keywords:
Image captioning, Visually Impaired, Urban Environment, Hybrid Method, Image Augmentation, Transfer LearningAbstract
Visual impairment is a global issue with significant impacts on the mobility and safety of individuals, especially in urban environments. Artificial intelligence solutions, such as image captioning, promise assistance for people with visual impairments to aid their daily activities. However, the field of image captioning in this context still has performance limitations. To address this, this study proposes a hybrid method combining image feature extraction from VGG16, ResNet50, and YOLO on the encoder side with LSTM and BiGRU on the decoder side to generate descriptions that have proven to enhance model performance on the Flickr8k dataset in previous research. By adapting this method to the Visual Assistance dataset, incorporating image augmentation through a combination of rotation and zoom, and applying transfer learning to address the dataset size limitation, this study successfully improved the model"™s performance in supporting the activities of visually impaired pedestrians in urban environments. Evaluation results showed significant improvements in several evaluation metrics. Overall, this model shows improvement compared to previous research, where Sharma et al. (2022) reported that the InceptionV3-BiLSTM model with Adaptive Attention achieved a BLEU-4 score of only 0.266 on the Visual Assistance dataset. This study achieved a 60.53% increase in BLEU-4 score compared to previous research on the Visual Assistance dataset. Overall, this study provides a positive contribution to developing more effective and accurate solutions for visually impaired navigation users in urban environments.References
Bourne, R, R, A., Flaxman, S, R., Braithwaite, T., Cicinelli, M, V., Das, A., 2017, Vision Loss Expert Group. Magnitude, temporal trends, and projections of the global prevalence of blindness and distance and near vision impairment: A systematic review and meta-analysis. Lancet Glob. Health 2017, 5, e888–e897.
Manduchi, R., PhD* Kurniawan, S. 2011, Mobility-Related Accidents Experienced by People with Visual Impairment, University of Santa Cruz Santa Cruz, CA, pp.10
Sáez, J.M., Escolano, F., Lozano, M.A. Aerial Obstacle Detection With 3-D Mobile Devices. IEEE J. Biomed. Health Inform. 2015, 19, 74–80.
Ahmed, F., and M. Yeasin, 2017, Optimization and Evaluation of deep architectures for ambient awareness on a sidewalk, in Neural Networks (IJCNN), 2017 International Joint Conference on. IEEE, pp. 2692–2697.
Liu, L., Wang, Y., and Zhao, H., 2019, "An Image Segmentation Method for the blind sidewalks recognition by using the convolutional neural network U-net," 2019 IEEE International Conference on Signal, Information and Data Processing (ICSIDP), Chongqing, China, pp. 1-4, doi: 10.1109/ICSIDP47821.2019.9172970.
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., Bengio. Y., 2015, Show, attend and tell: Neural Image caption generation with visual attention, in: Proceedings of the International conference on machine learning, pp. 2048–2057.
Chen, L., Zhang, H., Xiao, J., Nie, L., Shao, J., Liu, W., Chua, T., 2017, SCA-CNN: Spatial and channel-wise attention in convolutional networks for Image captioning, in: Proceedings of the IEEE Conference on Computer vision and Pattern Recognition (CVPR), pp. 6298–6306.
You, Q., Jin, H., Wang, Z., Fang, C., Luo, J 2016 Image captioning with semantic attention. In proceedings of the IEEE conference on computer vision and pattern recognition, 4651–4659
Sharma, D., Dhiman, C., and Kumar, D., 2022, "Automated Image Caption Generation Framework using Adaptive Attention and Bi-LSTM," 2022 IEEE Delhi Section Conference (DELCON), New Delhi, India, pp. 1-5.
Kaur, M., Kaur, H., 2023, An efficient deep learning based hybrid model for image caption generation, International Journal of Advanced Computer Science and Applications, 14(3)
Shorten, C., & Khoshgoftaar, T. M. 2019, A survey on Image Data Augmentation for Deep Learning. Journal of Big Data, 6(1).
Zhuang, F., Qi, Z., Duan, K., Xi, D., Zhu, Y., Zhu, H., Xiong, H., & He, Q. 2020, A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1), 43-76.
Han, M., Chen, W., & Moges, A.D., 2019, Fast image captioning using LSTM. Cluster Comput 22 (Suppl 3), 6143–6155. https://doi.org/10.1007/s10586-018-1885-9
Al-Malla, M.A., Jafar, A., & Ghneim, N., 2022, Image captioning model using attention and object features to mimic human image understanding. J Big Data 9, 20. https://doi.org/10.1186/s40537-022-00571-w
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J., 2001, BLEU, Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, Philadelphia
