Researchers have developed an AI-driven system that creates realistic virtual environments, enabling robots to practice complex tasks more efficiently before real-world deployment and testing.
Researchers at the Massachusetts Institute of Technology (MIT) and the Toyota Research Institute have developed SceneSmith, an AI-powered framework that automatically generates realistic virtual environments for training robots. The system aims to reduce the time, cost and effort required to collect real-world training data while improving robots’ ability to perform everyday tasks.
Robots typically learn through repeated physical interactions, making the training process labour-intensive and expensive. SceneSmith addresses this challenge by creating detailed 3D indoor environments where robots can safely practise activities before operating in real-world settings.
The framework relies on three AI agents working together throughout the design process. A “designer” creates the initial scene, a “critic” evaluates its realism and practicality, and an “orchestrator” manages revisions until the virtual environment meets the required standard. Once complete, the scenes are imported into physics simulation software that accurately models properties such as mass, friction and inertia.
According to the researchers, the system generated more than 1,300 unique indoor environments, including homes, offices and commercial spaces. Compared with previous scene-generation methods, the environments contained significantly more objects and interactive elements, enabling robots to practise tasks such as opening cupboards, placing items on shelves and moving objects between rooms.
The researchers also evaluated robot policies within these simulated environments before deployment. Human reviewers agreed with the AI system’s assessments more than 99 per cent of the time, helping identify unsuitable strategies before physical testing began. Additional experiments showed that robots could interact with cabinets, bottles and other movable objects while maintaining realistic physical behaviour.
User studies involving more than 200 participants found that the generated environments appeared realistic in over 90 per cent of cases and followed text prompts more accurately than earlier approaches. The researchers believe the technology could accelerate robotics development by providing scalable, diverse and highly realistic virtual training grounds while reducing dependence on costly real-world data collection.
