Graph-MARL-CloudLLM: A Graph-enhanced Multi-Agent Reinforcement Learning Framework for Adaptive Resource Scheduling and Workload Optimization of Large Language Model Inference Services across Multi-Cloud Platforms
DOI:
https://doi.org/10.66238/fsrma115Keywords:
Multi-Cloud Computing, Multi-Agent Reinforcement Learning, Graph Neural Network, Adaptive Resource Scheduling, Cloud AI Infrastructure, Workload Optimization, Large Language Model InferenceAbstract
The increasing adoption of Large Language Models (LLMs) in enterprise artificial intelligence applications has created significant challenges for inference service management across multi-cloud platforms, where heterogeneous computing resources, dynamic workloads, network fluctuations, and strict quality-of-service (QoS) requirements hinder efficient resource utilization and stable service delivery. This paper aims to develop an intelligent scheduling framework for cross-cloud LLM inference services that can achieve adaptive workload orchestration, low-latency inference, and cost-efficient resource allocation. We propose Graph-MARL-CloudLLM (Graph-enhanced Multi-Agent Reinforcement Learning for Cross-Cloud Large Language Model Inference Optimization), a novel framework that integrates Graph Neural Networks (GNNs) with Multi-Agent Reinforcement Learning (MARL) to model complex cloud resource dependencies and perform autonomous scheduling decisions. Specifically, Graph-MARL-CloudLLM constructs a dynamic resource graph containing cloud nodes, GPU clusters, LLM instances, and inference requests, and employs cooperative agents for cloud selection, model selection, GPU allocation, and request routing. A QoS-aware reinforcement learning strategy is further designed to jointly optimize latency, throughput, cost, and resource utilization. Experimental results on a simulated heterogeneous multi-cloud LLM inference environment demonstrate that Graph-MARL-CloudLLM achieves superior scheduling performance compared with existing approaches. Specifically, the proposed framework reduces P99 latency from 271 ms to 194 ms, improves GPU utilization to 86.9%, and decreases inference cost by 17.7% compared with the MARL-based baseline. These results validate the effectiveness of graph-enhanced state representation and cooperative multi-agent optimization for adaptive LLM inference scheduling.
References
[1] Kwon, Woosuk, et al. "Efficient memory management for large language model serving with pagedattention." Proceedings of the 29th symposium on operating systems principles. 2023.
[2] Wu, Bingyang, et al. "Fast distributed inference serving for large language models." arXiv preprint arXiv:2305.05920 (2023).
[3] Sun, Biao, et al. "Llumnix: Dynamic scheduling for large language model serving." 18th USENIX symposium on operating systems design and implementation (OSDI 24). 2024.
[4] Cong, Peizhuang, et al. "MICO: efficient query scheduling for multi-cloud deployed LLM inference service." Science China Information Sciences 69.3 (2026): 132102.
[5] Li, Tianran, Hanwu Li, and Yutong Zhou. "E-commerce Sentiment Analysis Using Fine-tuned LLaMA3 Models: A QLoRA-based Approach." Journal of Technology Innovation and Engineering 1.4 (2025).
[6] Aron, Rajni, and Ajith Abraham. "Resource scheduling methods for cloud computing environment: The role of meta-heuristics and artificial intelligence." Engineering Applications of Artificial Intelligence 116 (2022): 105345.
[7] Zhou, Guangyao, et al. "Deep reinforcement learning-based methods for resource scheduling in cloud computing: A review and future directions." Artificial Intelligence Review 57.5 (2024): 124.
[8] Mirhoseini, Azalia, et al. "Device placement optimization with reinforcement learning." International conference on machine learning. PMLR, 2017.
[9] Mao, Hongzi, et al. "Resource management with deep reinforcement learning." Proceedings of the 15th ACM workshop on hot topics in networks. 2016.
[10] Mao, Hongzi, et al. "Learning scheduling algorithms for data processing clusters." Proceedings of the ACM special interest group on data communication. 2019. 270-288.
[11] Yang, Shiyu, et al. "GART: Graph Neural Network-based Adaptive and Robust Task Scheduler for Heterogeneous Distributed Computing." IEEE Access 13 (2025): 200196-200216.
[12] Zeng, Jun, et al. "NeuroSchedule2. 0: A Novel GNN-based Scheduling Method with RL-based Preprocessing Optimization for High-level Synthesis." IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (2026).
[13] Chen, Sen, et al. "Adaptive Multi-Agent Reinforcement Learning with Graph Neural Networks for Dynamic Optimization in Sports Buildings." Buildings 15.14 (2025): 2554.
[14] Gao, Ziyang, Yuanliang Qu, and Yi Han. "Cross-Lingual Sponsored Search via Dual-Encoder and Graph Neural Networks for Context-Aware Query Translation in Advertising Platforms." arXiv preprint arXiv:2510.22957 (2025).
[15] Xu, Chengde, Chongye Wang, and Zeyu Li. "H2-LBM: A Hierarchical Hybrid Deep Reinforcement Learning Framework for L7 Load Balancing and Global Traffic Scheduling in Multi-Cloud LLM Serving." Fundamental Scientific Reports in Multidisciplinary Areas 2.02 (2026): 97-108.
[16] Yao, Yilin, Wenchao Zhang, and Mengtong Li. "Cloud-Edge Federated Incremental Learning Framework for PMSM Efficiency Optimization with Lightweight CNN-LSTM Models and OTA Differential Deployment." Journal of Computer Science and Frontier Technologies 3.1 (2026): 98-111.
[17] Zhang, Jingsi, and Jixiang Ding. "CA-PPO-AXI: Coverage-Aware Deep Reinforcement Learning for Adaptive AXI Bus Protocol Verification." Fundamental Scientific Reports in Multidisciplinary Areas 2.02 (2026): 159-169.
[18] Zhong, Ziwen, Zhitao Shu, and Yue Zhao. "A Cloud-Based Cross-Modal Transformer for Emotion Recognition and Adaptive Human–Computer Interaction." Fundamental Scientific Reports in Multidisciplinary Areas 2.02 (2026): 86-96.
[19] Zhang, Wenchao, Mengtong Li, and Xing Chen. "Distributed Edge-Fog-Cloud Architecture for Intelligent Fault Diagnosis of Permanent Magnet Synchronous Motors Using Temporal Fusion Transformers and Federated Learning." Fundamental Scientific Reports in Multidisciplinary Areas 2.02 (2026): 121-131.
[20] Zhao, Yue, and Zhimo Han. "Safe and Adaptive Cloud Healing: Verifying LLM-Generated Recovery Plans with a Neural-Symbolic World Model." 2025 7th International Academic Exchange Conference on Science and Technology Innovation (IAECST). IEEE, 2025.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Min Qiu Trine, Jinxiang Yu, Ningyun Wang
This work is licensed under a Creative Commons Attribution 4.0 International License.
License Terms (Recommended – CC BY 4.0)
All articles published in Fundamental Scientific Reports in Multidisciplinary Areas are licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0).
This license permits unrestricted use, distribution, and reproduction in any medium, provided the original author(s) and source are properly cited. Users are allowed to copy, redistribute, remix, transform, and build upon the material for any purpose, including commercial use, as long as appropriate credit is given.
Authors retain copyright of their work while granting the journal the right of first publication. The journal provides immediate open access to its content on the principle that making research freely available supports a greater global exchange of knowledge.
For more information about this license, visit:
https://creativecommons.org/licenses/by/4.0/
