Hierarchical deep reinforcement learning for multi-timescale Robotic Mobile Fulfillment System optimization

Robotic Mobile Fulfillment Systems (RMFS) rely on autonomous mobile robots (AMRs) to transport inventory pods to workstations; however, optimizing decisions across tightly coupled components and multiple time scales remains challenging under stochastic demand. We propose a Hierarchical Multi-Timescale Deep Reinforcement Learning (HMT-DRL) framework that separates shift-level strategic planning from real-time tactical control. A strategic agent determines workstation role transitions over a several-hour shift-level horizon, while a tactical agent manages real-time AMR task allocation, routing, and pod assignment. The agents exchange context via a Spatio-Temporal Attention (STAtt) module and are trained with a sequential update scheme and advantage decomposition to improve stability over simultaneous multi-agent PPO updates. We evaluate HMT-DRL in RAWSim-O (an open-source simulator) across three warehouse layouts and stochastic demand profiles. Compared with strong baselines (heuristics, SOQN, and RL variants SAPPO, MAPPO, MRTAgent and MADQN), HMT-DRL reduces long-term operating cost by 4.17% (small), 33.0% (medium), and 57.2% (large) relative to an optimal fixed heuristic and consistently lowers weekly average cost and station queue lengths over 100 instances. These results demonstrate that explicitly decoupling and coordinating multi-timescale decisions yields robust cost reductions for dynamic RMFS operations.