The framework lets robots learn skills, adapt to changes, and perform tasks, offering a new approach to improving robot capabilities in real-world settings.
The WildLMa framework, developed by a team from UC San Diego, MIT, and NVIDIA, addresses the limitations of existing mobile robot manipulation techniques by seamlessly integrating robust skill acquisition with efficient task planning.
The framework consists of two core components: WildLMa-Skill for skill learning and WildLMa-Planner for task execution. WildLMa-Skill enables robots to learn atomic, reusable skills through language-conditioned imitation learning. It uses pre-trained vision-language models like CLIP to translate language commands (e.g., “find the red bottle”) into visual representations. A reparameterization technique enhances this process by generating probability maps for greater precision.
Skills are acquired through virtual reality teleoperation, where human demonstrations of complex actions are captured using a learned low-level controller. This method expands the robot’s functional repertoire while reducing the cost and complexity of protests. Once skills are learned, WildLMa-Planner organizes them into a skill library and uses large language models to interpret human instructions, sequencing the skills to complete multi-step tasks effectively.
The framework was evaluated using a Unitree B1 quadruped robot equipped with a Z1 arm, a custom gripper, multiple cameras, and LiDAR for navigation and manipulation tasks. It was tested in two environments: in-distribution settings, where objects and layouts resembled the training data, and out-of-distribution (O.O.D.) scenarios, which introduced variations in object placement, textures, and backgrounds.
WildLMa was benchmarked against multiple baselines, including imitation learning, reinforcement learning, and zero-shot grasping methods. The results highlighted WildLMa’s superior success rates, particularly in O.O.D. settings, showcasing its ability to generalize skills effectively. It also excelled in long-horizon tasks and real-world applications, demonstrating resilience to environmental perturbations.
By open-sourcing their work, the team aims to inspire further research in robot manipulation and accelerate the development of practical, multitasking robots capable of assisting in real-world scenarios.
