Researchers developed a simulation benchmark measuring humanoid robots under realistic physical forces, revealing hidden performance gaps and offering a stronger foundation for future evaluations.
Researchers have introduced ThorArena, alongside the Force-Aware Tracking Score (FATS), to assess how well humanoid robots cope with realistic physical interactions rather than idealised laboratory conditions. The benchmark, described in a recent preprint, aims to provide a more accurate measure of robotic performance during everyday tasks involving external forces.
Current evaluation methods typically focus on movement in obstacle-free environments, where robots face little or no resistance. While such tests measure locomotion and balance, they often fail to capture how machines perform when lifting, pushing, pulling or carrying objects. As a result, robots that appear capable in conventional assessments may struggle once real-world forces come into play.
To address this limitation, the research team collected data from human volunteers wearing virtual reality headsets and motion trackers while completing common tasks. Specialised sensor-equipped hand tools recorded the direction and magnitude of forces applied during each activity. These measurements were then reproduced within a physics-based simulator, allowing virtual humanoid robots to perform the same actions under realistic force conditions.
The robots were evaluated using the Force-Aware Tracking Score, which measures tracking accuracy and stability while interacting with external loads. Four representative humanoid control systems were tested. Although all performed well in force-free scenarios, clear differences emerged when realistic forces were introduced. One system consistently maintained better balance and achieved the highest overall score, highlighting weaknesses that conventional benchmarks failed to identify.
The researchers believe the benchmark could improve the development of humanoid robots destined for factories, warehouses and other workplaces where physical interaction is unavoidable. Future work will expand the dataset with additional tasks and a wider range of force conditions before applying the evaluation framework to real-world robotic platforms.
