Scopus

A MULTIMODAL AI FRAMEWORK FOR CONTEXT-AWARE LANGUAGE LEARNING: INTEGRATING VISION TRANSFORMER AND GPT-2

Năm XB 2026 Tạp chí / Hội thảo Advances in Information and Communication Technology: Proceedings of the 4th International Conference, ICTA 2025, Volume 2 Volume 2, pp. 356-364 Đơn vị ICTU DOI / Link https://link.springer.com/chapter/10.1007/978-3-032-18159-6_38 ↗

Tác giả

Quach Xuan Truon

Tóm tắt

Amidst globalization and digital transformation, foreign language acquisition has become a fundamental cognitive skill. Traditional vocabulary-focused methods, lacking contextual grounding and sensory integration, fail to develop communicative competence. This paper introduces an intelligent, multimodal learning system that integrates visual inputs with deep learning models. By combining Vision Transformer (ViT) for visual feature extraction and GPT-2 for contextual text generation, the system produces multilingual, semantically rich descriptions. It also employs text-to-speech for pronunciation support and includes interactive modules to reinforce retention. Operable offline, the system ensures data privacy and accessibility, promoting personalized, context-aware language learning through multimodal AI. This work contributes to a paradigm shift in computer-assisted language learning (CALL), emphasizing semantic grounding, learner autonomy, and cognitive engagement. Experimental results suggest promising usability in resource-constrained educational settings.

Tài liệu tham khảo

[1] Zhang, Z., Wu, Q., Li, Y.: A survey on knowledge-enhanced multimodal learning. Artif. Intell. Rev. (2024)

[2] Zhou, H., Li, X.: Graph neural networks in vision-language image understanding: a comprehensive review. Vis. Comput. (2024)

[3] Singh, P., Müller, K.: Evaluating the effectiveness of automatic image captioning for accessibility on the web. Univ. Access Inf. Soc. 21(1), 145–158 (2022)

[4] Zhang, H., Shang, J.: Natural language processing and applications. In: Natural Language Processing and Applications, pp. 15–32. Springer Singapore (2025)

[5] Kalyuga, S., Sweller, J.: Learning across media in a second language: effects of verbal and visual input. Eur. J. Psychol. Educ. 37(2), 401–418 (2022)

[6] Chen, Y., Li, J.: The role of live transcripts in synchronous online 12 classrooms. Educ. Inf. Technol. 28(5), 6029–6048 (2023)

[7] Li, Y., Xu, H.: Foundation models for speech, images, videos, and control. In: Foundations of Multimedia, pages 221–243. Springer (2023)

[8] Richard, E.M.: The effects of using an auto-subtitle system in educational videos. Educ. Inf. Technol. 28(1), 123–137 (2023)

[9] Richard, F.S., Robert, M.B.: Using commonly available technologies to create online multimedia learning environments. Educ. Tech. Res. Dev. 70(4), 967–985 (2022)

[10] Ms coco (microsoft common objects in context) dataset. https:// paperswithcode.com/dataset/coco. Accessed 2025

[11] Anonymous. Dgic collates popular existing datasets from five domains: common domain sourced from mscoco, assistive domain sourced from vizwiz ... NeurIPS Datasets and Benchmarks Track (2024)

[12] Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. In: International Conference on Learning Representations (ICLR). OpenReview (2021)

[13] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners. OpenAI Technical Report (2019). https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

[14] Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., Yuille, A.: Deep captioning with multimodal recurrent neural networks (m-rnn). In: International Conference on Learning Representations (ICLR) (2015)

[15] Donahue, J., et al.: Long-term recurrent convolutional networks for visual recognition and description. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

[16] Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

[17] Xu, K., et al.: Show, attend and tell: neural image caption generation with visual attention. In: Proceedings of the 32nd International Conference on Machine Learning (ICML) (2015)

Ghi chú

Source ID: 225; Author classification: ICTU - Main Author; First listed author: Quang-Quy Tran; Contact author: Quach Xuan Truong; Total authors: 6; Springer matched from provided ICTA 2025 volumes

← Quay lại danh sách bài báo