
Windows Agent Arena
The Windows counterpart to OSWorld: real applications, real tasks, and a cloud runner so a full pass takes minutes
What is Windows Agent Arena?
Almost every desktop-agent benchmark runs on Linux, and almost every desktop that matters commercially runs Windows. This is Microsoft's answer to that gap — a reproducible Windows environment with tasks across the applications people actually use, scored on whether the task was completed. Its useful trick is parallelism: runs can be spread across cloud machines so a few hundred tasks finish in minutes instead of over a day, which is what makes it usable during development rather than only at publication time. Read the last commit date before planning around it, though — this is a research artefact tied to a 2024 paper, and it has moved slowly since.
What can you do with Windows Agent Arena?
- Test on the operating system your users have — Tasks run against real Windows applications, which is where a Linux-only benchmark stops predicting how an agent will do in the field.
- Finish a full run in minutes — Runs can be spread across cloud machines in parallel, so a complete pass is something you do during development rather than once before publishing.
- Raise the difficulty when scores saturate — A harder mode stops setting the task up for the agent, so it has to find and open the right application itself before the work even starts.
- Start from a working reference agent — The repository ships the agent from the paper, including its screen-understanding setup, so a new agent has something to be measured against.
Before you choose Windows Agent Arena
- This is a research artefact attached to a 2024 paper and has moved slowly since, so check recent activity before treating its task set as a current standard.
- Running it means standing up Windows environments, and the parallel mode is documented against Microsoft's own cloud machine-learning service rather than infrastructure in general.
Star history
29 Aug to 30 Oct · +22
Frequently asked questions
Is Windows Agent Arena free for commercial use?
Windows Agent Arena is released under the MIT licence — OSI-approved open source, which permits commercial use.
How can Windows Agent Arena be deployed?
Windows Agent Arena is available as Runs locally / Self-hosted.