Face-Tracking System Using A Laptop, ESP32, And YOLOE

The system combines a laptop running the YOLOE (You Only Look Once – Everything) face-detection model with an ESP32 for servo control and sensor interfacing. This division of tasks enables face tracking at more than 50 frames per second (FPS), with the laptop handling computationally intensive AI inference and the ESP32 managing real-time hardware control and sensing.

This hybrid design combines the processing power of a laptop with the low-latency hardware control capabilities of the ESP32. Conventional implementations that run the detection model entirely on a Raspberry Pi 5, even with a Hailo-8L AI accelerator, typically achieve around 30 FPS, while software-only implementations generally operate at just 3-5 FPS and require a GUI, which further increases the processing load. In comparison, the same YOLOE or YOLOv8-Face models running on a mid-range laptop can consistently achieve 50+ FPS, with even higher frame rates when GPU acceleration is available.

While a laptop excels at AI inference, it is not well suited to direct, low-latency control of servos and sensors. To overcome this limitation, the ESP32 controls the pan-tilt servos, interfaces with a VL53L0X time-of-flight (ToF) distance sensor, and drives a 128×64 OLED display. The laptop and ESP32 communicate using the lightweight ESPConnect protocol, enabling the laptop to transmit face coordinates, servo commands, and face-count information, while the ESP32 returns distance measurements and performs real-time hardware control.

The system employs a closed-loop raster-scan algorithm. During idle operation, the camera performs rapid horizontal sweeps from 0° to 180°, incrementally adjusting the tilt angle before scanning back in the opposite direction. Once one or more faces are detected, the camera continuously tracks the centroid of the detected group instead of locking onto a single face. This behaviour resembles the auto-tracking used in professional video-conferencing cameras, helping keep all detected subjects within the camera’s field of view. An external USB webcam is mounted directly on the pan-tilt mechanism, allowing the camera to follow the servo movement.

Fig. 1 shows the authors’ prototype and the real-time testing setup for the 50+ FPS face-tracking system.

The main controller used in this system is the ESP32, which receives commands from the laptop over the ESPconnect and controls the servo motors accordingly. A VL53L0X time-of-flight (ToF) distance sensor continuously measures the distance to the detected object and sends the readings back to the laptop, where they are overlaid on the live video feed. The complete list of components required to build this system is provided in Bill of Materials table.

Circuit and working

Fig. 2 shows the circuit diagram of the 50+ FPS face-tracking system using the laptop-ESP32 duo. The system is built around the ESP32, which controls the pan-tilt mechanism, reads the VL53L0X ToF sensor, and updates the OLED display. A 7V-9V DC supply passes through a reverse-polarity protection diode and a 5V regulator. The regulated 5V rail powers the PCA9685 PWM driver and the two servos, while a 100µF capacitor helps minimise voltage fluctuations caused by servo movement. The ESP32, OLED, and VL53L0X are powered from their appropriate regulated supply pins, with all grounds connected to a common reference.

The ESP32 communicates with the PCA9685, VL53L0X, and OLED through the I²C bus, typically using GPIO 21 (SDA) and GPIO 15 (SCL). The PCA9685 receives position commands from the ESP32 and generates PWM signals for the pan and tilt servos through its first two channels. The pan servo provides horizontal movement, while the tilt servo provides vertical movement. The remaining 14 channels are available for future expansion. The VL53L0X measures the distance to the object and sends the data to the ESP32, while the OLED displays information such as distance and servo status.

A USB webcam is mounted on the pan-tilt mechanism and connected to the laptop. The laptop runs the YOLOE face-detection software, which determines the face position in the camera frame and sends corresponding movement commands to the ESP32. The ESP32 processes these commands and drives the servos through the PCA9685 to keep the detected face near the centre of the frame.

Thus, the laptop performs face detection, while the ESP32 handles servo control, distance measurement, and display functions. This division of tasks enables fast and responsive face tracking. The PCA9685’s additional channels also allow the system to be extended for multi-axis robotic applications.

Fig. 3 shows the pan-tilt camera mount used to provide two-axis movement to the camera. A USB camera is mounted on the top of the mechanism and can be rotated horizontally (pan) and vertically (tilt) using two servo motors. The servos are mounted in the black mechanical frame, providing controlled movement of the camera in both directions.