Agent S — Computer & Browser Use

What is Agent S?

Agent S separates two jobs that a single model does badly at once: deciding what to do next, and working out where on the screen to click. A general model handles the planning; a smaller vision model, run separately, converts "the Save button" into coordinates. That split is why it scores as it does — Simular reports 72.60% on OSWorld with its best-of-several-attempts setting, against roughly 72% for humans. It is also why setup is more work than a pip install: you supply the grounding model as its own endpoint and tell the agent what resolution it reports coordinates in. By default it drives your actual desktop, one monitor, with no sandbox in between.

What can you do with Agent S?

Split planning from pointing — A general model decides the next step and a separate vision model turns a description into coordinates, which is more reliable than asking one model to do both.
Choose each model independently — The planner can be any of the major hosted models or one you serve yourself, and the grounding model is configured apart from it.
Trade cost for accuracy at run time — Running several attempts and selecting between them is what lifts the published score; it can be turned off when a single pass is enough.
Run on the desktop you already have — Linux, macOS and Windows are supported directly, so evaluating it does not require standing up a virtual machine first.
Let it run code when clicking is the wrong tool — An optional local execution mode lets the agent write and run a script for tasks like bulk file handling, instead of clicking through them.

Before you choose Agent S

The grounding model is required and is not bundled: you host it yourself or buy an endpoint, and you must pass the resolution it reports coordinates in or every click lands in the wrong place.
By default it controls your real desktop on a single monitor and, with local execution enabled, runs generated code on that machine — the project says as much, and the isolation is yours to add.

Frequently asked questions

Is Agent S free for commercial use?

Agent S is released under the Apache-2.0 licence — OSI-approved open source, which permits commercial use.

How can Agent S be deployed?

Agent S is available as Runs locally / Self-hosted.

Documentation

Reproduced from the simular-ai/Agent-S README, published under Apache-2.0. Read the original ↗

💡 Introduction

Welcome to Agent S, an open-source framework designed to enable autonomous interaction with computers through Agent-Computer Interface. Our mission is to build intelligent GUI agents that can learn from past experiences and perform complex tasks autonomously on your computer.

Whether you’re interested in AI, automation, or contributing to cutting-edge agent-based systems, we’re excited to have you here!

🎯 Current Results

On OSWorld, Agent S3 alone reaches 66% in the 100-step setting, already exceeding the previous state of the art of 63.4% (GTA1 w/ GPT-5). With the addition of Behavior Best-of-N, performance climbs even higher to 72.6%, surpassing human-level performance on OSWorld (~72%)!

Agent S3 also demonstrates strong zero-shot generalization! On WindowsAgentArena, accuracy rises from 50.2% using only Agent S3 to 56.6% by selecting from 3 rollouts. Similarly on AndroidWorld, performance improves from 68.1% to 71.6%

🛠️ Installation & Setup

Prerequisites

Single Monitor: Our agent is designed for single monitor screens
Security: The agent runs Python code to control your computer - use with care
Supported Platforms: Linux, Mac, and Windows

Installation

To install Agent S3 without cloning the repository, run

pip install gui-agents

If you would like to test Agent S3 while making changes, clone the repository and install using

pip install -e .

Don’t forget to also brew install tesseract! Pytesseract requires this extra installation to work.

API Configuration

Option 1: Environment Variables

Add to your .bashrc (Linux) or .zshrc (MacOS):

export OPENAI_API_KEY=<YOUR_API_KEY>
export ANTHROPIC_API_KEY=<YOUR_ANTHROPIC_API_KEY>
export HF_TOKEN=<YOUR_HF_TOKEN>

Option 2: Python Script

import os
os.environ["OPENAI_API_KEY"] = "<YOUR_API_KEY>"

Supported Models

We support Azure OpenAI, Anthropic, Gemini, Open Router, and vLLM inference. See models.md for details.

Grounding Models (Required)

For optimal performance, we recommend UI-TARS-1.5-7B hosted on Hugging Face Inference Endpoints or another provider. See Hugging Face Inference Endpoints for setup instructions.

🚀 Usage

⚡️ Recommended Setup:
For the best configuration, we recommend using OpenAI gpt-5-2025-08-07 as the main model, paired with UI-TARS-1.5-7B for grounding.

CLI

Note, this is running Agent S3, our improved agent, without bBoN.

Run Agent S3 with the required parameters:

agent_s \
    --provider openai \
    --model gpt-5-2025-08-07 \
    --ground_provider huggingface \
    --ground_url http://localhost:8080 \
    --ground_model ui-tars-1.5-7b \
    --grounding_width 1920 \
    --grounding_height 1080

Local Coding Environment (Optional)

For tasks that require code execution (e.g., data processing, file manipulation, system automation), you can enable the local coding environment:

agent_s \
    --provider openai \
    --model gpt-5-2025-08-07 \
    --ground_provider huggingface \
    --ground_url http://localhost:8080 \
    --ground_model ui-tars-1.5-7b \
    --grounding_width 1920 \
    --grounding_height 1080 \
    --enable_local_env

⚠️ WARNING: The local coding environment executes arbitrary Python and Bash code locally on your machine. Only use this feature in trusted environments and with trusted inputs.

Required Parameters

--provider: Main generation model provider (e.g., openai, anthropic, etc.) - Default: “openai”
--model: Main generation model name (e.g., gpt-5-2025-08-07) - Default: “gpt-5-2025-08-07”
--ground_provider: The provider for the grounding model - Required
--ground_url: The URL of the grounding model - Required
--ground_model: The model name for the grounding model - Required
--grounding_width: Width of the output coordinate resolution from the grounding model - Required
--grounding_height: Height of the output coordinate resolution from the grounding model - Required

Optional Parameters

--model_temperature: The temperature to fix all model calls to (necessary to set to 1.0 for models like o3 but can be left blank for other models)

Grounding Model Dimensions

The grounding width and height should match the output coordinate resolution of your grounding model:

UI-TARS-1.5-7B: Use --grounding_width 1920 --grounding_height 1080
UI-TARS-72B: Use --grounding_width 1000 --grounding_height 1000

Optional Parameters

--model_url: Custom API URL for main generation model - Default: ""
--model_api_key: API key for main generation model - Default: ""
--ground_api_key: API key for grounding model endpoint - Default: ""
--max_trajectory_length: Maximum number of image turns to keep in trajectory - Default: 8
--enable_reflection: Enable reflection agent to assist the worker agent - Default: True
--enable_local_env: Enable local coding environment for code execution (WARNING: Executes arbitrary code locally) - Default: False

Local Coding Environment Details

The local coding environment enables Agent S3 to execute Python and Bash code directly on your machine. This is particularly useful for:

Data Processing: Manipulating spreadsheets, CSV files, or databases
File Operations: Bulk file processing, content extraction, or file organization
System Automation: Configuration changes, system setup, or automation scripts
Code Development: Writing, editing, or executing code files
Text Processing: Document manipulation, content editing, or formatting

When enabled, the agent can use the call_code_agent action to execute code blocks for tasks that can be completed through programming rather than GUI interaction.

Requirements:

Python: The same Python interpreter used to run Agent S3 (automatically detected)
Bash: Available at /bin/bash (standard on macOS and Linux)
System Permissions: The agent runs with the same permissions as the user executing it

Security Considerations:

The local environment executes arbitrary code with the same permissions as the user running the agent
Only enable this feature in trusted environments
Be cautious when the agent generates code for system-level operations
Consider running in a sandboxed environment for untrusted tasks
Bash scripts are executed with a 30-second timeout to prevent hanging processes

`gui_agents` SDK

First, we import the necessary modules. AgentS3 is the main agent class for Agent S3. OSWorldACI is our grounding agent that translates agent actions into executable python code.

import pyautogui
import io
from gui_agents.s3.agents.agent_s import AgentS3
from gui_agents.s3.agents.grounding import OSWorldACI
from gui_agents.s3.utils.local_env import LocalEnv  # Optional: for local coding environment

# Load in your API keys.
from dotenv import load_dotenv
load_dotenv()

current_platform = "linux"  # "darwin", "windows"

Next, we define our engine parameters. engine_params is used for the main agent, and engine_params_for_grounding is for grounding. For engine_params_for_grounding, we support custom endpoints like HuggingFace TGI, vLLM, and Open Router.

engine_params = {
  "engine_type": provider,
  "model": model,
  "base_url": model_url,           # Optional
  "api_key": model_api_key,        # Optional
  "temperature": model_temperature # Optional
}

# Load the grounding engine from a custom endpoint
ground_provider = "<your_ground_provider>"
ground_url = "<your_ground_url>"
ground_model = "<your_ground_model>"
ground_api_key = "<your_ground_api_key>"

# Set grounding dimensions based on your model's output coordinate resolution
# UI-TARS-1.5-7B: grounding_width=1920, grounding_height=1080
# UI-TARS-72B: grounding_width=1000, grounding_height=1000
grounding_width = 1920  # Width of output coordinate resolution
grounding_height = 1080  # Height of output coordinate resolution

engine_params_for_grounding = {
  "engine_type": ground_provider,
  "model": ground_model,
  "base_url": ground_url,
  "api_key": ground_api_key,  # Optional
  "grounding_width": grounding_width,
  "grounding_height": grounding_height,
}

Then, we define our grounding agent and Agent S3.

# Optional: Enable local coding environment
enable_local_env = False  # Set to True to enable local code execution
local_env = LocalEnv() if enable_local_env else None

grounding_agent = OSWorldACI(
    env=local_env,  # Pass local_env for code execution capability
    platform=current_platform,
    engine_params_for_generation=engine_params,
    engine_params_for_grounding=engine_params_for_grounding,
    width=1920,  # Optional: screen width
    height=1080  # Optional: screen height
)

agent = AgentS3(
    engine_params,
    grounding_agent,
    platform=current_platform,
    max_trajectory_length=8,  # Optional: maximum image turns to keep
    enable_reflection=True     # Optional: enable reflection agent
)

Finally, let’s query the agent!

# Get screenshot.
screenshot = pyautogui.screenshot()
buffered = io.BytesIO() 
screenshot.save(buffered, format="PNG")
screenshot_bytes = buffered.getvalue()

obs = {
  "screenshot": screenshot_bytes,
}

instruction = "Close VS Code"
info, action = agent.predict(instruction=instruction, observation=obs)

exec(action[0])

Refer to gui_agents/s3/cli_app.py for more details on how the inference loop works.

OSWorld

To deploy Agent S3 in OSWorld, follow the OSWorld Deployment instructions.

💬 Citations

If you find this codebase useful, please cite:

@misc{Agent-S3,
      title={The Unreasonable Effectiveness of Scaling Agents for Computer Use}, 
      author={Gonzalo Gonzalez-Pumariega and Vincent Tu and Chih-Lun Lee and Jiachen Yang and Ang Li and Xin Eric Wang},
      year={2025},
      eprint={2510.02250},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2510.02250}, 
}

@misc{Agent-S2,
      title={Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents}, 
      author={Saaket Agashe and Kyle Wong and Vincent Tu and Jiachen Yang and Ang Li and Xin Eric Wang},
      year={2025},
      eprint={2504.00906},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2504.00906}, 
}

@inproceedings{Agent-S,
    title={{Agent S: An Open Agentic Framework that Uses Computers Like a Human}},
    author={Saaket Agashe and Jiuzhou Han and Shuyu Gan and Jiachen Yang and Ang Li and Xin Eric Wang},
    booktitle={International Conference on Learning Representations (ICLR)},
    year={2025},
    url={https://arxiv.org/abs/2410.08164}
}

What is Agent S?

What can you do with Agent S?

Before you choose Agent S

Frequently asked questions

🥳 Updates

Table of Contents

💡 Introduction

🎯 Current Results

🛠️ Installation & Setup

Prerequisites

Installation

API Configuration

Option 1: Environment Variables

Option 2: Python Script

Supported Models

Grounding Models (Required)

🚀 Usage

CLI

Local Coding Environment (Optional)

Required Parameters

Optional Parameters

Grounding Model Dimensions

Optional Parameters

Local Coding Environment Details

gui_agents SDK

OSWorld

💬 Citations

Other options in Computer & Browser Use

`gui_agents` SDK