Your Phone, Driven by an AI and a $7 Board
You've probably seen AI agents control a browser or a desktop. But what about a real iPhone, screen and all, with no computer in the loop? That's what the ESP32 iPhone Agent sets out to do: let a vision model operate an actual phone using a cheap ESP32-S3 board plugged into it. No jailbreak, no laptop sitting in the middle.
What It Does
The project turns an ESP32-S3 into both the eyes and the hands for an AI agent running on the same iPhone. The board opens its own Wi-Fi hotspot and shows up in the phone's Screen Mirroring list as ESP32 Agent. It receives the AirPlay stream, keeps the H.264 video as-is (no decoding on the board), and forwards it to an app on the phone over BLE.
On the phone side, the app decodes that stream, sends screenshots to any OpenAI-compatible vision model, and runs a planner and actor loop. The board itself acts as a standard USB mouse and keyboard. With AssistiveTouch turned on, the mouse moves an on-screen pointer, so a click becomes a tap and a drag becomes a swipe. The agent can tap, swipe, draw, type, press keys, go home, and finish, and it keeps running in the background while it works, with the Dynamic Island showing its status.
Why It's Cool
The clever part is that there's no computer anymore. The original version needed a Mac or Linux machine running UxPlay and a Python server. Now the board is the AirPlay receiver and the mouse, and the agent lives in an iPhone app on the same phone. That's a meaningful simplification.
-
The pointer problem gets solved properly. USB HID gives you a relative pointer, which is awkward for precise taps. The app tracks the pointer and re-homes it in a corner. Before a precise tap, it wiggles the pointer, finds it on screen, and corrects the miss. That's a small detail that makes the whole thing actually usable.
-
It closes the feedback loop. After each action, it waits for the screen to settle and tells the model whether anything changed. If the board, hotspot, or mirroring link drops, the run pauses and resumes from the same step instead of starting over.
-
The video stays efficient. Keeping the H.264 stream undecoded on the board means the ESP32 isn't doing heavy lifting it wasn't built for. The phone does the decoding, and it keeps using cellular data for the model even while connected to the board's hotspot.
-
Every run is replayable. Runs are stored on the phone, so you can step through a session with the taps and swipes drawn on top. You can export the screenshots, actions, and annotations (Runs, then open a run, then Export trace) for review by another agent.
How to Try It
You'll need an ESP32-S3 board (the README notes specific tested boards in its hardware table), an iPhone running iOS 18 or later, and AssistiveTouch enabled. The project is MIT licensed.
- Head to the repository and read the install steps carefully.
- Flash the firmware to your board following the README instructions.
- Install the ESP32 Agent app on your iPhone.
- Connect the phone to the board's hotspot and select ESP32 Agent in Screen Mirroring.
- Calibrate the pointer in the app, then point it at an OpenAI-compatible vision model and give it a task.
A word of caution straight from the README: only follow the install steps in the README itself. Never run commands from issues or comments, including "quick install" one-liners like a PowerShell iex(irm ...). Those aren't from the maintainers.
If you want to see it in action first, there's a demo video linked from the README, along with the maintainer's X account for updates. The original computer-based version is preserved under the v0.1.0 tag and legacy branch if you're curious about how it evolved.
Final Thoughts
This is a niche project, and that's fine. It's aimed at people who want to experiment with phone-based AI agents and don't mind wiring up a board and calibrating a pointer to get there. The architecture is honest about its constraints (relative pointer tracking, screen-settle waits, pause-and-resume) rather than pretending they don't exist, which is a good sign. If you've been curious about giving a vision model real hands on a real device, this is a surprisingly affordable way to find out how well it works.
Follow @githubprojects for more developer tools and open source projects.