Vision agents vs API agents benchmark: token usage and performance comparison
Benchmark the agent and note that API agent uses 14x fewer tokens than vision agent; consider building an API surface for internal tools.
Build an API surface for internal tools to reduce token usage and variance.
Summary
The benchmark compared two approaches to let an AI agent operate a React‑admin style admin panel: a vision agent that interacts via screenshots and clicks, and an API agent that calls the app’s HTTP endpoints directly. Both agents used Claude Sonnet and the same pinned dataset, but the vision agent required 53 steps, 14 minutes, and 550k input tokens, while the API agent completed the task in 8 calls, 19.7 seconds, and 12k tokens. The vision agent struggled with pagination and had to be given an explicit 14‑step UI walkthrough to succeed. The API agent leveraged structured responses, eliminating the need to interpret pagination controls. Variance in the vision path was high (wall‑clock time 749–1257 s), whereas the API path was stable. Haiku failed the vision path due to its structured‑output schema. The benchmark demonstrates that building an API surface for internal tools can dramatically reduce token consumption and improve reliability. The study was conducted using the Reflex 0.9 plugin that auto‑generates HTTP endpoints from event handlers.
Key changes
- Vision agent required 53 steps, 14 min, 550k tokens; API agent used 8 calls, 19.7 s, 12k tokens.
- Vision agent failed to paginate without explicit UI walkthrough.
- API agent leveraged structured responses, eliminating pagination.
- Vision path variance high (749–1257 s); API path stable.
- Haiku model failed vision path due to structured‑output schema.
- Auto‑generated API surface via Reflex 0.9 reduces engineering cost.
- Vision agents suitable for third‑party SaaS; internal tools benefit from API approach.
- Token consumption difference: 14x fewer tokens for API agent.