An open-source project uses a low-cost ESP32-S3 board to let an AI agent tap, swipe and navigate an iPhone without a jailbreak.
A low-cost ESP32-S3 development board can enable an artificial intelligence (AI) agent to operate a real iPhone, tapping buttons, navigating menus and entering text based on natural-language instructions. The open-source project, called ESP32 iPhone Agent, combines screen mirroring, a USB mouse-and-keyboard interface and a vision model to automate tasks without installing an app on the phone or jailbreaking it.
The project uses a Seeed Studio XIAO ESP32S3 board, which its author prices at about US$7. However, the AI model does not run on the microcontroller. Instead, a separate computer processes the phone’s screen, plans actions and sends commands to the ESP32-S3, which executes them through a USB connection.
The approach offers an alternative to mobile automation methods that depend on Apple’s development tools or applications installed on the device. The project is available under the MIT licence, allowing developers to inspect, modify and redistribute its code under the licence’s terms.
The hardware setup requires an ESP32-S3 development board with native USB support, an iPhone and a computer running macOS or Linux. USB-C iPhones can connect directly, while Lightning-equipped models require a Lightning to USB Camera Adapter. A USB-C hub with power pass-through can help keep the phone charging during operation.
The computer must run the agent’s server software and an AirPlay receiver, with both the phone and computer connected to the same network. Users also need access to a vision model through an OpenAI-compatible API, either from a cloud service or a locally hosted model. Although the development board is inexpensive, the overall cost depends on the computer, any additional hardware and the chosen AI service.
The system works by combining screen capture, AI decision-making and physical input. The iPhone mirrors its display over AirPlay to UxPlay, an open-source AirPlay receiver running on the computer. A GStreamer pipeline converts the incoming video stream into JPEG images that the vision model can analyse.
The software divides the task between a planner and an actor. The planner develops a sequence of steps to complete the requested task, while the actor examines the latest screenshot and selects the next action. The system can use the same model for both roles or configure separate models.
The ESP32-S3 acts as the agent’s hands. Connected to the iPhone over USB, it presents itself as a standard human interface device (HID), functioning as a mouse and keyboard. Apple’s AssistiveTouch accessibility feature enables the phone to display a pointer that can be moved and clicked to interact with on-screen controls.
The computer sends commands to the board over Wi-Fi. Because mouse movement is relative rather than based on absolute screen coordinates, the server tracks the pointer’s position and corrects alignment before clicking. It then checks the screen to determine whether the action produced the expected result, helping the agent identify missed taps and recover from errors.
Building the system involves flashing the firmware using arduino-cli and version 3.x of the ESP32 Arduino core. Developers must configure the Wi-Fi credentials and the computer’s IP address in the firmware settings before running the flashing script provided in the repository’s hardware directory.
On the computer, UxPlay and the required GStreamer components must be installed. The server software is set up in a Python virtual environment, with model details and API credentials configured through a .env file. Once the server starts, users can access the web interface through a browser on port 8890 by default.
The iPhone requires AssistiveTouch to be enabled under its accessibility settings. After connecting the board, users select the receiver named iPhone Agent from the Screen Mirroring menu in Control Centre. A calibration step aligns the pointer with the mirrored display. Tasks can then be entered through the web interface, with manual controls and an emergency stop available to help supervise operation.
The firmware and server code are available in the project’s GitHub repository under the MIT licence. Model configuration supports OpenAI-compatible endpoints, including compatible deployments using vLLM, SGLang, llama.cpp and Ollama, provided the selected model supports image inputs and the required API. This allows developers to experiment with cloud-based inference or local model hosting.
According to the author, a local setup using NVIDIA’s Qwen3.8-Flash-Next-NVFP4 vision model across two NVIDIA DGX Spark systems achieved approximately 1.3 seconds per action and two seconds per planning step, with the model’s thinking mode disabled. These results apply to that specific configuration and should not be treated as typical performance figures. The project does not report an overall task success rate.
The system has several limitations. iOS does not mirror certain screens, including passcode entry, Apple Pay and DRM-protected video, preventing the agent from seeing or interacting with them. Screen mirroring also stops when the phone locks, requiring automatic locking to be disabled during a session. Keyboard input is currently limited to ASCII characters, restricting direct text entry in Hindi and other Indian scripts. Landscape applications are also unsupported, and changing the AssistiveTouch tracking speed requires recalibration.
For Indian engineering students and makers, the project provides a practical way to explore embedded systems, USB HID, computer vision and AI agents using relatively inexpensive hardware. The board’s quoted US$7 price is not a verified Indian retail price, and the computer and AI inference requirements add to the overall cost. Local model hosting could reduce recurring API charges for institutions with suitable computing resources, while support for Indian-language text input remains an opportunity for further development.
For more information, click here.
