← Back to all projects

Agent S

A desktop agent that pairs a reasoning model with a separate model trained only to find the thing on screen

Apache-2.0
Stars
12.8k
Forks
1.5k
Open issues
72
Last commit
24 Oct 2026

What is Agent S?

Agent S separates two jobs that a single model does badly at once: deciding what to do next, and working out where on the screen to click. A general model handles the planning; a smaller vision model, run separately, converts "the Save button" into coordinates. That split is why it scores as it does — Simular reports 72.60% on OSWorld with its best-of-several-attempts setting, against roughly 72% for humans. It is also why setup is more work than a pip install: you supply the grounding model as its own endpoint and tell the agent what resolution it reports coordinates in. By default it drives your actual desktop, one monitor, with no sandbox in between.

What can you do with Agent S?

  • Split planning from pointing — A general model decides the next step and a separate vision model turns a description into coordinates, which is more reliable than asking one model to do both.
  • Choose each model independently — The planner can be any of the major hosted models or one you serve yourself, and the grounding model is configured apart from it.
  • Trade cost for accuracy at run time — Running several attempts and selecting between them is what lifts the published score; it can be turned off when a single pass is enough.
  • Run on the desktop you already have — Linux, macOS and Windows are supported directly, so evaluating it does not require standing up a virtual machine first.
  • Let it run code when clicking is the wrong tool — An optional local execution mode lets the agent write and run a script for tasks like bulk file handling, instead of clicking through them.

Before you choose Agent S

  • The grounding model is required and is not bundled: you host it yourself or buy an endpoint, and you must pass the resolution it reports coordinates in or every click lands in the wrong place.
  • By default it controls your real desktop on a single monitor and, with local execution enabled, runs generated code on that machine — the project says as much, and the isolation is yours to add.

Star history

27 Aug to 30 Oct · +636

12.2k12.8k

Frequently asked questions

Is Agent S free for commercial use?

Agent S is released under the Apache-2.0 licence — OSI-approved open source, which permits commercial use.

How can Agent S be deployed?

Agent S is available as Runs locally / Self-hosted.