Giving a Blind LLM Eyes: Desktop Control Without a Vision Model
Let your AI agent see your screen, click buttons, read text, and automate any desktop app. No vision model required.
Your AI Agent Is Blind. Let's Fix That.
Here's the situation. You've got powerful text-only models running locally — Qwen, Gemma, DeepSeek — and they're great at code, reasoning, writing. But the moment you ask one to do something on your actual desktop, you hit a wall. They can't see the screen. They don't know what windows are open, what buttons are available, what text is on display.
The "solutions" all have catches. A vision model burns VRAM your 4090 doesn't have spare. Sending screenshots to an API costs thousands of tokens per image. ASCII art of your screen runs 16,000 tokens and still loses track of what's clickable.
desktop-agent fixes this. It gives your AI agent a real-time, structured description of your screen — what's there, what's interactive, what it says — in about 500 tokens. Your model reads it like it reads anything else. No vision model. No API costs. No VRAM penalty.
What It Can Do
See Your Screen in Real Time
One command returns a full snapshot: every button, menu, input, and text region, organized by spatial zone. Clean JSON. ~3 seconds. Your agent reads it like any other input.
Click, Type, and Navigate
Click any element by its reference (@e3). Type into fields. Target specific screen regions. References survive across commands — no state coupling.
Verify Every Action
After each action, --diff tells you exactly what changed. Your agent gets a closed feedback loop — it knows whether the click landed.
Work at Any Speed
Quick mode: sub-second, metadata only. Default: full screen scan. Deep mode: every detail. Region targeting: zoom into a specific area.
Optional Vision Fallback
Hit a custom UI or ambiguous icon? Feed a screenshot to any OpenRouter vision model on demand. ~300 tokens per call, only when you need it.
Reusable Workflows
Record actions once, replay forever. Parameterize them. Chain with conditions, loops, retries. Think GitHub Actions for your desktop.
The Core Commands
desktop-agent analyze --json # see the screen
desktop-agent click @e3 # click an element
desktop-agent analyze --diff # verify what changed
Three commands. That's the loop. Your agent sees → acts → verifies. Every time. --quick for sub-second polling, --deep when you need everything, --region left to zoom into the sidebar. Element references (@e1, @e2) work across separate CLI calls — no process coupling, no state files to manage.
Real Demo: Spotify Playlist Extraction
An AI agent extracting a playlist from Spotify — an app with custom-rendered UI and non-standard fonts that traditional automation tools can't handle:
- Agent scans the screen — "Spotify's not running"
- Agent launches Spotify via system command
- Agent scans the sidebar region — finds "Aeternum" playlist
- Agent clicks at the found coordinates
- Agent reads the center region — extracts all visible track text
- Agent cross-references 3 OCR passes for accuracy
- Agent saves results to
~/Desktop/
2. Once in a Long, Long While... — Low Roar (2:03)
3. Don't Be So Serious — Low Roar (6:13)
4. Clair Obscur: Expedition 33 (OST) — Lorien Testard (3:42)
5. October (Calendar Project) — Feverkin (4:08)
That's a text-only AI agent with zero vision capability reading a custom-rendered music app and producing structured data. Try doing that with shell scripts.
Why This Instead of Alternatives
| Approach | Tokens per Look | Knows What's Clickable | Works Offline |
|---|---|---|---|
| Screenshot → Vision API | 1,000–4,000 | ❌ | ❌ |
| ASCII art of screen | ~16,000 | ❌ | ✔ |
| Accessibility metadata only | ~150 | ✔ | ✔ |
| OCR text only | ~300 | ❌ | ✔ |
| desktop-agent | ~500 | ✔ | ✔ |
Accessibility metadata alone misses browser content and custom widgets. OCR alone can't tell you what's clickable. desktop-agent combines both — plus spatial layout — into one compact, structured snapshot your agent can actually reason about.
Integrates With Your Stack
Works natively with Claude Code, QuetzaCodetl, OpenCode, or any agent harness that can invoke Bash commands. No MCP server. No plugin to configure. No API key to manage (unless you opt into vision fallback). A single install script sets up all dependencies.
Linux only. Requires AT-SPI and xdotool, which are standard on most Linux desktops. No macOS or Windows support.
What You Get
✔ Full source — clean, modular Python. Easy to extend or customize.
✔ Install script — one command, all dependencies handled.
✔ Integration guides — Claude Code, QuetzaCodetl, OpenCode, any Bash-capable harness.
✔ Task recording system — record, search, and replay desktop actions with parameters.
✔ Workflow DSL — compose actions into recoverable sequences with conditions, loops, and retries.
✔ Vision fallback — optional OpenRouter integration for when you need actual sight.
✔ Full documentation — comprehensive handoff, quickstart, and API reference.
Get Desktop Agent
One-time purchase. Full source. Lifetime updates. Give your AI agents the eyes they've been missing.
Buy Now — $29 AUDWritten by Indra's Mirror — building tools that let local AI actually interact with the world.
Tags: desktop automation, AI agents, Linux, AT-SPI, OCR, screen understanding, Claude Code, local LLM, workflow automation