Friday, July 19, 2024

Safer Self-Driving Cars With New Vision Tech

- Advertisement -

Researchers from MIT and Meta have created PlatoNeRF, a computer vision method that allows autonomous vehicles to detect hidden obstacles using shadows to build accurate 3D models.

Plato-NeRF is a computer vision system that combines lidar measurements with machine learning to reconstruct a 3D scene, including hidden objects, from only one camera view by exploiting shadows. Here, the system accurately models the rabbit in the chair, even though that rabbit is blocked from view.
Credits:Credit: Courtesy of the researchers, edited by MIT News
Plato-NeRF is a computer vision system that combines lidar measurements with machine learning to reconstruct a 3D scene, including hidden objects, from only one camera view by exploiting shadows. Here, the system accurately models the rabbit in the chair, even though that rabbit is blocked from view. Credits:Credit: Courtesy of the researchers, edited by MIT News

Imagine you’re in an autonomous vehicle in a tunnel when traffic suddenly stops ahead due to a crash. Typically, your car would react based on the vehicle in front. But what if it could detect the halted traffic ahead and brake sooner?

Researchers from MIT and Meta have developed PlatoNeRF. This computer vision technique enables autonomous vehicles to detect obstacles beyond their direct line of sight by using shadows to create accurate 3D models of obscured areas. This method combines lidar with machine learning to improve 3D reconstructions, particularly in challenging lighting conditions. PlatoNeRF could also enhance AR/VR headsets and assist warehouse robots by allowing precise environment modelling without manual measurements.

- Advertisement -

Shedding light on the problem

The researchers have developed PlatoNeRF using single-photon lidar, which emits light pulses and measures their return time to map a high-resolution scene. This lidar illuminates a target point, and while some light returns directly, most scatter off other objects. PlatoNeRF captures this scattered light to gather more scene details, including depth and shadow information. It analyses secondary light rays that bounce from the target to other points, identifying shadowed areas to infer the geometry of hidden objects. By sequentially lighting 16 points and capturing multiple images, PlatoNeRF reconstructs the entire 3D scene.

A winning combination

PlatoNeRF hinges on the integration of multibounce lidar with a neural radiance field (NeRF), a type of machine-learning model that encodes a scene’s geometry into the weights of a neural network, enhancing its ability to interpolate or estimate novel views of a scene. Compared to two common alternatives—one using only lidar and the other only a NeRF with a colour image—PlatoNeRF outperformed both, mainly when the lidar sensor was of lower resolution. This makes PlatoNeRF more viable for real-world applications, where commercial devices often feature lower-resolution sensors.

In the future, the researchers plan to track more than two light bounces to enhance scene reconstructions and intend to incorporate deeper learning techniques and colour image data into PlatoNeRF to capture texture details.

Nidhi Agarwal
Nidhi Agarwal
Nidhi Agarwal is a journalist at EFY. She is an Electronics and Communication Engineer with over five years of academic experience. Her expertise lies in working with development boards and IoT cloud. She enjoys writing as it enables her to share her knowledge and insights related to electronics, with like-minded techies.

SHARE YOUR THOUGHTS & COMMENTS

Unique DIY Projects

Electronics News

Truly Innovative Tech

MOst Popular Videos

Electronics Components

Calculators