Android AI Agent
Describe what you want done on your phone and an agent does it, built first for blind and visually impaired users.
- Kind
- Automation agent with a vision model
- Year
- 2026
- Built with
- Python
- ADB
- Groq
- Llama 4 Scout
- Gemini
- WebSocket
- 1 more
The question
What if a phone could be operated by describing the job instead of finding the buttons?
The agent looks at the screen the way a person would, picks one step and checks the result before the next.
Why it exists
Built for blind and visually impaired users, who cannot rely on seeing a screen to operate a phone. The agent turns a plain-language command into the taps, swipes and typing that carry it out.
How it works
One pass of the loop
- 1
Screen
A screenshot and the list of accessibility elements are captured from the device.
- 2
Decision
The model sees the task, the history and the screen, and answers with exactly one action.
- 3
Check
The action is validated, so it cannot target an element that is not on screen.
- 4
Act
The action is run over ADB, announced aloud and marked by a haptic pattern.
Decisions
A free local device first
ADB against an emulator or a phone costs nothing and has no queue. The cloud option sits behind the same interface for when a device farm is needed.
Groq as the default model
The free tier, vision support and quick answers made it the default, and Gemini is one setting away.
The accessibility tree finds the elements
Element locations come from the accessibility tree instead of image recognition, and the screenshot still goes to the model so that it sees the screen as it looks.
The agent notices when it goes in circles
The loop detects repeated screens and tells the model which elements it has not tried yet.
Not built yet
- Device clouds other than AWS Device Farm
Details
- About 369 automated tests
Built with
Languages
- Python
AI and models
- Groq
- Llama 4 Scout
- Gemini
Infrastructure and services
- Docker
Systems and hardware
- ADB
Protocols and chains
- WebSocket