YOLOE – YOLO For Everything!

This article describes how a YOLO-based model can be extended to detect both standard object classes and previously unseen or user-defined objects without retraining the model specifically for those categories. This approach enables more flexible and practical object detection for real-world applications.

YOLO (you only look once) is a fast and efficient object detection algorithm that identifies objects by processing an entire image in a single forward pass through a neural network. Instead of scanning an image multiple times, YOLO divides it into a grid, with each grid cell predicting bounding boxes and object classes. Every predicted bounding box is assigned a confidence score that reflects both the likelihood of an object being present and how accurately the box fits it. Because multiple predicted boxes may overlap, YOLO applies non-maximum suppression (NMS) to retain only the most accurate bounding box for each object. By performing all predictions in a single pass, YOLO delivers real-time performance, making it well-suited for edge devices and embedded AI applications.

However, most pre-trained YOLO models can detect only the object categories on which they were originally trained. They are generally unable to recognise user-defined objects, such as a particular MacBook charger, a unique tool, or any other object outside the trained dataset.

How YOLOE operates

Having understood how YOLO works, it is now useful to explore how YOLOE operates. YOLOE can be considered an extension of YOLO and is often described as ‘YOLO for Everything.’ While a conventional YOLO model is trained to detect a fixed set of object categories, YOLOE enables the detection of user-specified objects from a much broader range of visual concepts. In simple terms, if YOLO is designed to recognise a predefined collection of objects, YOLOE focuses only on the objects specified through prompts. Unlike YOLO, which performs detection without prompts, YOLOE uses visual prompts to guide the model towards the desired objects.

Traditionally, adding a new object to an object detection model requires collecting numerous images of that object from different viewpoints, annotating them, and retraining the model. This process is computationally expensive and time-consuming, often requiring hours of training even on high-end, server-class GPUs.

YOLOE eliminates this retraining step. The model is already pre-trained on a vast and diverse collection of visual concepts. Instead of learning a new object through training, it extracts the visual features and geometric characteristics of the object provided as a prompt. By supplying one or more example regions of the object from different viewpoints, YOLOE creates a visual representation and associates it with a user-defined object ID rather than an object name. During inference, whenever the model encounters an object with similar visual characteristics, it returns the corresponding object ID. This prompt-based object identification enables YOLOE to detect both predefined and user-defined objects without retraining.

Following the same principle, the model can be configured to identify a specific object, such as a particular MacBook charger, among many visually similar chargers. Similarly, the model can be configured to detect a particular person, for example, someone wearing glasses, or a person holding a smartphone. The key advantage of this approach is that the model remains lightweight while providing highly flexible, prompt-based object detection without requiring retraining.

In practical testing, this approach achieves an inference speed of approximately 7.2fps on a MacBook Air, which is impressive for real-time object detection. On a Raspberry Pi 5 (8GB RAM), performance drops slightly to around 5fps, but this remains sufficient for reliable real-time detection in most edge AI applications. Overall, the combination of a lightweight model and prompt-based object identification makes YOLOE an efficient solution for deployment on resource-constrained edge devices. Fig. 1 shows the tested prototype.

The system can be executed on a Raspberry Pi 4 or Raspberry Pi 5 for edge deployment, or it can be run directly on a laptop by installing Python and the required software packages, making development and testing straightforward.

In the prototype presented here, a Raspberry Pi 5 with 8GB RAM was used. The complete list of hardware components required to build the system is provided in Bill of Materials table.

Software and coding

For Python-based development, it is always recommended to work within a virtual environment. A virtual environment isolates system-specific packages from the system-wide Python installation, preventing dependency conflicts and ensuring that existing applications remain unaffected. Once development is complete, the virtual environment can simply be deactivated, allowing the Python environment to return to its normal configuration.

Create and activate a Python virtual environment