AI Papers Reader

Personalized digests of latest AI research

View on GitHub

When More Data Changes the Rules of Reinforcement Learning

Reinforcement learning for robot control has long been built around a scarcity assumption: an agent must wring as much as possible from a small reservoir of past experience. Massively parallel GPU simulators now overturn that assumption. But algorithms designed for scarce data still carry stabilizers that may hinder them when data is abundant. In a new study, researchers at Tianjin University, Shanxi University, and Imperial College London show that the value of those stabilizers depends on the data regime, and propose a family of algorithms, WarpSAC, that matches stabilizer choices to data availability.

Off-policy methods such as Soft Actor-Critic (SAC) reuse experience through a replay buffer. Two common stabilizers are parameter projection normalization, which constrains network weights to curb wild value estimates on unseen states, and clipped double-Q targets, in which the learner keeps two critics and uses the lower prediction to avoid overoptimism. These help when replay coverage is narrow. When thousands of parallel environments flood the buffer with diverse transitions, the researchers hypothesized, the bottleneck shifts from conservative exploration to efficiently exploiting abundant data.

To test this, they varied three components on top of FlashSAC, a scalable SAC baseline: replay weighting, normalization, and number of critics. A replay-weighting scheme called Sample Weight Decay (SWD), which gives recently collected transitions higher sampling probability, was used in all versions. Based on ablations across eight benchmark families, the team prescribes WarpSAC-L (normalization on, two critics) for data-limited CPU-scale training and WarpSAC-A (normalization off, one critic) for data-abundant GPU-parallel training.

In one illustrative case, on the simulated manipulation task UnitreeG1TransportBox-v1, WarpSAC-A raised success rate from 19.8% to 96.4%, a gain the authors attribute to relaxing conservatism when broad replay data already supplies exploration. Across nine CPU-scale environments, WarpSAC improved mean normalized score–step area under the curve (AUC), a combined measure of final performance and learning speed, by 4.5% over FlashSAC. Across fourteen GPU-parallel environments the improvement was 23.1%. On MuJoCo Playground, it gained 19.1% in mean normalized wall-time AUC. In a sim-to-real test, WarpSAC trained a Unitree G1 humanoid to walk in about 35 minutes on one NVIDIA A800 GPU, versus about 55 minutes for FlashSAC—a 36.4% wall-clock reduction.

Mechanism ablations showed SWD especially helped small networks. With one residual block, WarpSAC-L’s final return on humanoid-run was 467.39 versus 209.93 for FlashSAC, more than double; the gap narrowed but remained visible on harder HumanoidBench tasks as network size grew. In GPU-parallel settings, disabling normalization improved performance markedly at low capacity on Playground tasks such as G1 Flat and T1 Rough.

The authors caution that their prescription is chosen offline based on the target regime. A pipeline that starts with narrow CPU-scale data and later shifts to GPU-parallel fine-tuning would, they note, benefit from an online adaptive version that modulates normalization and critic count based on replay coverage. The study also focuses on FlashSAC and three axes; other stabilizers could behave differently. The results do not show that conservatism is useless—only that its value is conditional. As robot learning moves toward high-throughput simulation, knowing when to drop old stabilizers may be as important as inventing new ones.