A wearable sensing system combines first-person video, body tracking, finger movements and tactile data to build a 100-hour dataset for training humanoid robots in real-world tasks.
Researchers at the Shanghai AI Laboratory, Zhejiang University and Shanghai Jiao Tong University have developed a wearable sensing system that captures human movements, hand interactions and tactile information to improve how humanoid robots learn physical tasks. The system, called EgoExplorer, combines multiple sensor inputs into a 100-hour dataset designed to help robotic systems coordinate walking, posture and object manipulation. The research could support the development of robots capable of performing complex tasks in industrial and everyday environments.
The research addresses a limitation in robot training: conventional datasets often capture only selected aspects of human movement, leaving gaps in how body motion and object interaction work together. The wearable setup combines a head-mounted GoPro HERO13 camera, five VIVE Ultimate Trackers and sensor-equipped gloves, including MANUS Metagloves or Wuji gloves. The camera records the wearer’s first-person view, while the trackers measure position and orientation at the head, hands and ankles. The gloves capture detailed finger movements, with the Wuji gloves also recording tactile information.
By synchronising these inputs, the system creates a multimodal record of human activity, linking visual information with whole-body movement, finger articulation and touch. Step-level descriptions provide additional context about the actions being performed. A key component is the Calibrate-Complete-Constrain motion-reconstruction pipeline. It processes sparse tracker measurements to estimate movements of body joints that are not directly monitored. The method uses learned movement patterns and calibration data, while applying constraints to maintain temporal smoothness, consistent body geometry and agreement with sensor measurements.
According to the researchers, the approach reduced body-position error by 71.9% compared with the strongest baseline evaluated in their study. Preliminary experiments also indicated that generating video, movement and tactile data together could improve results compared with generating each data type independently.
The team tested the resulting data in simulation by training a controller for a Unitree G1 humanoid robot to reproduce two human movement sequences. The simulated robot completed both sequences without falling, demonstrating the potential of transferring human demonstrations into robotic control. However, the experiments did not establish reliable object manipulation or performance on unfamiliar tasks, and the system has not yet been validated on physical robots. EgoExplorer could provide a foundation for training more capable humanoid robots by supplying detailed examples of how humans move through environments while handling objects. Further work will be needed to determine how effectively these learned behaviours transfer to real-world robotic applications.
