Build an AI Powered Advanced Driver Assistance Systems (ADAS)

In Advanced Driver Assistance Systems (ADAS), semantic segmentation is one of the most important computer vision technologies. Unlike traditional object detection, which only identifies and locates objects using rectangular bounding boxes, semantic segmentation analyses every pixel in an image. It classifies each pixel into a specific category such as road, lane marking, vehicle, pedestrian, sidewalk, traffic sign, vegetation, or building. This enables the system to understand not only what objects are present but also their exact shape and position within the scene.

In autonomous driving and ADAS, semantic segmentation plays a critical role in enabling scene understanding, which is the vehicle’s ability to interpret its surroundings in real time. It helps identify the exact drivable area by distinguishing the road surface from pavements, grass, barriers, and other non drivable regions. This information is essential for safe path planning, lane keeping, and navigation.

Semantic segmentation also improves object understanding by distinguishing between different types of obstacles, such as pedestrians, cyclists, vehicles, traffic cones, or static poles, while accurately identifying their boundaries. Unlike bounding boxes that often include empty background areas, pixel level segmentation outlines the actual shape of every object, making it easier for the vehicle to judge available space and maintain safe distances.

Another advantage is its ability to perform well in complex environments such as busy city roads, highways, construction zones, and challenging weather or lighting conditions. Since every frame is analysed in detail, the vehicle receives a richer understanding of its surroundings, helping reduce the chances of incorrect decisions caused by misinterpreting the environment.

By delivering fine grained, pixel wise predictions, semantic segmentation bridges the gap between raw camera images and the high level decision making algorithms used by autonomous vehicles. The result is a more reliable perception system that improves safety and reduces the risk of critical driving errors.

Project introduction

Here, the project uses real time semantic segmentation combined with object tracking for ADAS applications using Ultralytics YOLO models on a Raspberry Pi. It demonstrates how lightweight YOLO semantic segmentation models, such as YOLO26n sem, can run efficiently on resource constrained edge devices like the Raspberry Pi or single board computers (SBCs) such as the NVIDIA Orin. 

Combining semantic segmentation with multi-object tracking creates a perception pipeline that not only understands what is present in the scene but also follows moving objects across consecutive frames. Tracking allows the system to estimate how objects move over time, which is important for predicting possible collisions and planning safer driving paths.

Bill of material

Coding

The design uses the YOLO26 model for semantic segmentation through the Ultralytics framework. The application can run on a Raspberry Pi with 8GB RAM or higher. However, using an NVIDIA Orin SBC provides significantly better performance because of its dedicated Artificial Intelligence (AI) hardware. During development, the best performance was observed on an Apple Silicon MacBook Air, where the application achieved approximately 30 to 40 frames per second (FPS). A higher FPS value means the system processes more video frames every second, resulting in smoother and more responsive real time detection. 

To begin, install Ultralytics and its required software libraries. Open the terminal and run the following commands:

sudo  pip3 install ultralytics
sudo pip3 install numpy
sudo pip3 install opencv-pythonncv-python

After successfully installing all the required dependencies, download the YOLO26 semantic segmentation model. The official models and download links are available on the Ultralytics documentation page. : 

https://docs.ultralytics.com/models/yolo26#supported-tasks-and-modes.

The yolo26n sem.pt nano semantic model is recommended because it is optimised for edge devices and provides good real time performance while requiring relatively low computing power.

Once the model has been downloaded, create the Python script. First, import the required libraries such as OpenCV (cv2) for image processing, Ultralytics YOLO for loading the deep learning model, and NumPy for handling numerical operations.

Next, load the YOLO model by specifying the model filename, such as yolo26n sem.pt. If the model file is stored in the same directory as the Python script, only the filename is required. Otherwise, specify its complete file path (refer to Fig. 3).

OpenCV is then used to capture live video from the camera. A video stream is simply a continuous sequence of image frames captured many times every second. Each frame is immediately passed to the YOLO26 semantic segmentation model for processing.

Unlike conventional object detection, which simply draws rectangles around detected objects, semantic segmentation analyses every pixel in the frame. Each pixel is assigned to a predefined class such as road, vehicle, person, building, sidewalk, vegetation, sky, or traffic sign. This produces a much more detailed understanding of the scene because every visible region is labelled.

Internally, the model processes each image through several stages. A backbone network first extracts important visual features such as edges, textures, and shapes. These features are then combined at different scales by the neck network, allowing both large and small objects to be recognised accurately. Finally, the segmentation head generates a dense prediction map in which every pixel is assigned a probability for each available class. The class with the highest probability becomes the final label for that pixel.

The resulting output is known as a semantic mask. This mask is essentially a colour coded map where every object category is represented by a different colour. For example, roads may appear in one colour, pedestrians in another, and vehicles in a third colour. The mask is resized to match the original camera image and blended with the live video feed, creating an overlay that clearly highlights every detected object while still allowing the original scene to remain visible.

This visualisation enables the system to display roads, vehicles, pedestrians, buildings, traffic signs, and other scene elements with highly accurate boundaries in every frame. Such pixel level information is extremely valuable for ADAS because it allows accurate drivable area detection, improved obstacle recognition, better lane understanding, and more reliable scene interpretation than conventional bounding box based detection.

Connection

Connect the camera to the Camera Serial Interface (CSI) port of the Raspberry Pi, NVIDIA Orin, or the selected SBC. The CSI interface provides a direct high speed connection between the camera and the processor, allowing video frames to be transferred with low latency, which is important for real time vision applications. 

After connecting the camera, power the board using the 5V AC DC adapter. 

After creating the Python script, run the application. Within a few seconds, a camera window opens and displays the real time semantic segmentation output. Move the camera around or mount it on the front or rear of a vehicle to simulate driving conditions.

As the video stream is processed, every frame is analysed pixel by pixel. The model highlights roads, vehicles, pedestrians, trees, buildings, sidewalks, traffic signs, and other objects using different colours. Unlike conventional object detection, which draws rectangular boxes, semantic segmentation accurately outlines the actual shape of every object, providing a much richer understanding of the scene.

This detailed scene information helps distinguish drivable road areas from obstacles and can be further extended by adding path planning, object tracking, or drivable area detection for autonomous driving applications on platforms such as the Raspberry Pi or NVIDIA Orin.