Sensing-Communication-Computation-Control" (S3C)Co-optimization architecture for industrial Cyber-Physical System (Image by SIA)

Framework of the Large Language Model-Enhanced Multi-Agent Transfer Reinforcement Learning algorithm (Image by SIA)

Industrial Cyber-Physical Systems (ICPS) represent a quintessential deep coupling system integrating Sensing, Communication, Computation, and Control (S3C). They have long faced challenges such as high dimensionality, non-convexity, and partial observability. To address these challenges, the research team from Shenyang Institute of Automation(SIA) of Chinese Academy of Sciences, proposed a novel algorithmic framework: LLM-Enhanced Multi-Agent Transfer Reinforcement Learning (LLMPT-MADRL).

This research, titled LLM-Enhanced Multi-Agent Transfer Reinforcement Learning for Sensing, Communication, Computing, and Control Co-Optimization in Cyber-Physical Systems, has been published in the authoritative automation journal IEEE/CAA Journal of Automatica Sinica. Master's student ZHANG Junyuan is the first author, with ResearcherXU Chi serving as the corresponding author.

Traditional optimization algorithms struggle to achieve joint S3C optimization, while conventional Multi-Agent Deep Reinforcement Learning (MADRL) methods suffer from a lack of prior knowledge, low exploration efficiency, and a tendency to fall into local optima.

The research team formulated the joint optimization of “sensing, communication, computing, and control” as a partially observable Markov decision process and designed an algorithm architecture consisting of an LLM module, a multi-agent deep reinforcement learning (MADRL) module, and a policy selection module. Within this architecture, the LLM module produces high-quality guiding policies through prompts; the MADRL module adopts an Actor-Critic structure for policy optimization; and the policy selection module dynamically fuses the two types of outputs via a probabilistic mechanism, enabling smooth policy transition during training.

In addition, the researchers devised an adaptive transfer loss function to overcome the network-scale heterogeneity between the LLM and the reinforcement learning networks, thereby ensuring effective knowledge transfer. With the objective of minimizing system latency and control error, the team carried out joint optimization simulations covering sampling coefficients, communication bandwidth, computing frequency, and control inputs across the sensing-communication-computing-control chain. They selected MADDPG, MATD3, and MAPPO as baselines and adopted a centralized training with decentralized execution paradigm for validation.

The experimental results show that LLMPT-MADRL outperforms conventional multi-agent algorithms in all metrics. Compared with the baseline methods, the proposed framework achieves a 16.8% improvement in composite reward, a 24.5% reduction in system latency, and a 10.4% decrease in control error. These findings demonstrate that large models can accelerate policy learning, suppress policy oscillations, and significantly enhance system stability and control performance.