← Back to all projects

Windows Agent Arena

The Windows counterpart to OSWorld: real applications, real tasks, and a cloud runner so a full pass takes minutes

OfficialMIT
Stars
913
Forks
106
Open issues
40
Last commit
26 Oct 2026

What is Windows Agent Arena?

Almost every desktop-agent benchmark runs on Linux, and almost every desktop that matters commercially runs Windows. This is Microsoft's answer to that gap — a reproducible Windows environment with tasks across the applications people actually use, scored on whether the task was completed. Its useful trick is parallelism: runs can be spread across cloud machines so a few hundred tasks finish in minutes instead of over a day, which is what makes it usable during development rather than only at publication time. Read the last commit date before planning around it, though — this is a research artefact tied to a 2024 paper, and it has moved slowly since.

What can you do with Windows Agent Arena?

  • Test on the operating system your users have — Tasks run against real Windows applications, which is where a Linux-only benchmark stops predicting how an agent will do in the field.
  • Finish a full run in minutes — Runs can be spread across cloud machines in parallel, so a complete pass is something you do during development rather than once before publishing.
  • Raise the difficulty when scores saturate — A harder mode stops setting the task up for the agent, so it has to find and open the right application itself before the work even starts.
  • Start from a working reference agent — The repository ships the agent from the paper, including its screen-understanding setup, so a new agent has something to be measured against.

Before you choose Windows Agent Arena

  • This is a research artefact attached to a 2024 paper and has moved slowly since, so check recent activity before treating its task set as a current standard.
  • Running it means standing up Windows environments, and the parallel mode is documented against Microsoft's own cloud machine-learning service rather than infrastructure in general.

Star history

29 Aug to 30 Oct · +22

891913

Frequently asked questions

Is Windows Agent Arena free for commercial use?

Windows Agent Arena is released under the MIT licence — OSI-approved open source, which permits commercial use.

How can Windows Agent Arena be deployed?

Windows Agent Arena is available as Runs locally / Self-hosted.

Windows Agent Arena
Ask AI
GitHub