A Cloud-Based Cross-Modal Transformer for Emotion Recognition and Adaptive Human–Computer Interaction
DOI:
https://doi.org/10.66238/fsrma93Keywords:
Multimodal Emotion Recognition; Cross-Modal Transformer; Cloud Computing; Human-Computer Interaction; ViT; Wav2Vec2; BERT; Affective Computing.., Multimodal Emotion Recognition, Cross-Modal Transformer, Cloud Computing, Human-Computer Interaction, ViT, Wav2Vec2, BERT, Affective ComputingAbstract
Emotion recognition is a fundamental component of next-generation human-computer interaction (HCI), enabling machines to perceive, understand, and respond to users’affective states. However, existing systems often rely on single-modality analysis-such as facial expressions, speech tone, or textual sentiment-resulting in limited robustness and poor generalization in real-world environments. To address these challenges, this study proposes a Cloud-Based Cross-Modal Transformer (CMT) framework for multimodal emotion recognition and adaptive human-computer interaction. The proposed model integrates visual, auditory, and textual signals using pretrained encoders-Vision Transformer (ViT), Wav2Vec2, and BERT-and employs a cross-modal attention mechanism to capture complex interdependencies among heterogeneous features. By leveraging cloud computing infrastructure with distributed training (Kubernetes and TensorFlow Serving), the system enables scalable, low-latency emotion recognition for large-scale user interactions. Experiments conducted on benchmark datasets including IEMOCAP, MELD, and AffectNet demonstrate that the proposed Cloud-Based Cross-Modal Transformer (CMT) achieves state-of-the-art performance, improving the F1-score by 3.0% and reducing cross-entropy loss by 12.9% compared to the best fusion-based baselines. Additionally, cloud deployment evaluations show an average response latency of 128 ms, representing a 35% reduction compared with conventional transformer-based fusion systems. These results confirm that the proposed CMT framework enables efficient, real-time emotion recognition and adaptive feedback in interactive applications such as intelligent customer service, virtual tutoring, and affective computing interfaces.These results highlight the effectiveness and scalability of the proposed CMT framework, marking an important step toward cloud-native affective computing and emotionally intelligent interactive systems.
References
Mishra D, Mishra R K, Agarwal R. Human–computer interaction and user experience in the artificial intelligence era[J]. Trends in Computer Science and Information Technology Research,, 2025: 31-54.
Lian H, Lu C, Li S, et al. A survey of deep learning-based multimodal emotion recognition: Speech, text, and face[J]. Entropy, 2023, 25(10): 1440.
Kalateh S, Estrada-Jimenez L A, Nikghadam-Hojjati S, et al. A systematic review on multimodal emotion recognition: building blocks, current state, applications, and challenges[J]. IEEE Access, 2024, 12: 103976-104019.
Muhammad F, Hussain M, Aboalsamh H. A bimodal emotion recognition approach through the fusion of electroencephalography and facial sequences[J]. Diagnostics, 2023, 13(5): 977.
Papakostas M, Spyrou E, Giannakopoulos T, et al. Deep visual attributes vs. hand-crafted audio features on multidomain speech emotion recognition[J]. Computation, 2017, 5(2): 26.
Busso C, Bulut M, Lee C, et al. IEMOCAP: Interactive emotional dyadic motion capture database[J]. Language resources and evaluation, 2008, 42(4): 335-359.
Poria S, Hazarika D, Majumder N, et al. Meld: A multimodal multi-party dataset for emotion recognition in conversations[J]. arXiv preprint arXiv:1810.02508, 2018.
Mollahosseini A, Hasani B, Mahoor M H. Affectnet: A database for facial expression, valence, and arousal computing in the wild[J]. IEEE Transactions on Affective Computing, 2017, 10(1): 18-31.
Qin X, Wu Z, Zhang T, et al. Bert-erc: Fine-tuning bert is enough for emotion recognition in conversation[C]//Proceedings of the AAAI conference on artificial intelligence. 2023, 37(11): 13492-13500.
Bousaid R, El Hajji M, Es-Saady Y. Facial emotions recognition using vit and transfer learning[C]//2022 5th International Conference on Advanced Communication Technologies and Networking (CommNet). IEEE, 2022: 1-6.
Huang J, Tao J, Liu B, et al. Multimodal transformer fusion for continuous emotion recognition[C]//ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020: 3507-3511.
Meudt S, Schmidt-Wack M, Honold F, et al. Going further in affective computing: how emotion recognition can improve adaptive user interaction[M]//Toward Robotic Socially Believable Behaving Systems-Volume I: Modeling Emotions. Cham: Springer International Publishing, 2016: 73-103.
Kubernetes T. Kubernetes[J]. Kubernetes. Retrieved May, 2019, 24: 2019.
Toor, M. J., Shabbir, S., Rauf, A., Nisar, A., Safdar, S., & Zaib-Un-Nisa. (n.d.). The role of structured-communication-technique in developing pragmatic language capability among ASD-children. https://doi.org/10.63878/cjssr.v3i2.785
Toor, M. J., Mehmood, H., Yasmeen, A., Honey, Q., Waseem, M., & Wasif, M. (n.d.). Exploring the effect of verbal encouraging phrases (VEP) on Chinese language learners’ academic achievement at university level.
Toor, M. J., Arafat, M., Ayoub, M., Ali, M., & Alessi Longa, F. E. (n.d.). From coherent self to fragmented persona: Digital identity fragmentation and academic self-concept among Generation Z, with comparative insights from Generation Alpha. https://doi.org/10.63878/cjssr.v4i2.2318
Katsitadze, N., Kharadze, N., & Pirtskhalaishvili, D. (2025). The impact of social media on students’ effective time management. Baltic Journal of Economic Studies, 11(1), 200–210. https://doi.org/10.30525/2256-0742/2025-11-1-200-210
Toor, M. J., Zafar, S., Omer, F., Sarwar, M., & Hussain, A. K. (n.d.). Teachers’ perspectives on effectiveness of translanguaging in enhancing English language learning at secondary school level.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Ziwen Zhong, Zhitao Shu, Yue Zhao
This work is licensed under a Creative Commons Attribution 4.0 International License.
License Terms (Recommended – CC BY 4.0)
All articles published in Fundamental Scientific Reports in Multidisciplinary Areas are licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0).
This license permits unrestricted use, distribution, and reproduction in any medium, provided the original author(s) and source are properly cited. Users are allowed to copy, redistribute, remix, transform, and build upon the material for any purpose, including commercial use, as long as appropriate credit is given.
Authors retain copyright of their work while granting the journal the right of first publication. The journal provides immediate open access to its content on the principle that making research freely available supports a greater global exchange of knowledge.
For more information about this license, visit:
https://creativecommons.org/licenses/by/4.0/
