Researchers from MIT and Meta have created PlatoNeRF, a computer vision method that allows autonomous vehicles to detect hidden obstacles using shadows to build accurate 3D models.
Imagine you’re in an autonomous vehicle in a tunnel when traffic suddenly stops ahead due to a crash. Typically, your car would react based on the vehicle in front. But what if it could detect the halted traffic ahead and brake sooner?
Researchers from MIT and Meta have developed PlatoNeRF. This computer vision technique enables autonomous vehicles to detect obstacles beyond their direct line of sight by using shadows to create accurate 3D models of obscured areas. This method combines lidar with machine learning to improve 3D reconstructions, particularly in challenging lighting conditions. PlatoNeRF could also enhance AR/VR headsets and assist warehouse robots by allowing precise environment modelling without manual measurements.
Shedding light on the problem
The researchers have developed PlatoNeRF using single-photon lidar, which emits light pulses and measures their return time to map a high-resolution scene. This lidar illuminates a target point, and while some light returns directly, most scatter off other objects. PlatoNeRF captures this scattered light to gather more scene details, including depth and shadow information. It analyses secondary light rays that bounce from the target to other points, identifying shadowed areas to infer the geometry of hidden objects. By sequentially lighting 16 points and capturing multiple images, PlatoNeRF reconstructs the entire 3D scene.
A winning combination
PlatoNeRF hinges on the integration of multibounce lidar with a neural radiance field (NeRF), a type of machine-learning model that encodes a scene’s geometry into the weights of a neural network, enhancing its ability to interpolate or estimate novel views of a scene. Compared to two common alternatives—one using only lidar and the other only a NeRF with a colour image—PlatoNeRF outperformed both, mainly when the lidar sensor was of lower resolution. This makes PlatoNeRF more viable for real-world applications, where commercial devices often feature lower-resolution sensors.
In the future, the researchers plan to track more than two light bounces to enhance scene reconstructions and intend to incorporate deeper learning techniques and colour image data into PlatoNeRF to capture texture details.
