AxisRL Unifies Agentic RL Post-Training Across SGLang and Megatron 🔗
The framework coordinates multi-turn rollouts, tool use, and weight sync for billion-parameter agent workflows.
AxisRL is an agentic reinforcement learning post-training framework built on SGLang for high-throughput rollout and Megatron for large-scale distributed training. It addresses the growing complexity of LLM post-training where agents interact with environments over hundreds of turns, call tools, and receive delayed rewards—scenarios that break traditional single-turn fine-tuning pipelines. By integrating rollout, training, weight synchronization, data movement, and reproducible debugging into one system, AxisRL reduces idle time and improves consistency between rollout and trainer components.
It supports configurable policy optimization objectives including PPO, GRPO/GRPO2, GSPO, TOPR, and TIS, and works with both white-box agent environments and black-box setups via an OpenAI-compatible proxy. Features like handle-based data movement, context packing, routing replay, and spike replay help maintain alignment between rollout-generated samples and training updates, critical for stabilizing training at extreme scales. The project is already seeing use in agent RL workflows with 300+ turn trajectories and models reaching hundreds of billions of parameters.
The catch: AxisRL assumes deep integration with SGLang and Megatron, limiting flexibility for teams using alternative serving or training stacks like vLLM or PyTorch FSDP.
Why this leads today AxisRL reduces engineering complexity in deploying long-horizon AI agents by integrating SGLang, Megatron, and real-world workflows into a production-ready RL framework, enabling scalable, reliable agent training and deployment for builders.
- Train LLM agents with multi-turn tool use and delayed rewards
- Scale PPO-style RLHF to hundred-billion-parameter models
- Synchronize rollout data and training weights in distributed agent workflows
Source: XYZ-AI-Lab/axrl — based on the project README.