
OSWorld
Desktop-agent tasks run against a real operating system in a virtual machine, not a simulated interface
What is OSWorld?
It is the common yardstick the computer-use projects cite, and the reason it carries weight is that the environment is genuine: Ubuntu or Windows in a VM, running real applications, reachable through VMware, VirtualBox, Docker with KVM, or AWS, Azure and Modal for parallel runs. That realism is also the cost — adopting it means hypervisors, VM images and cleanup after interrupted runs. Results are version-sensitive too: the Verified update changed tasks and signals, and the project asks that comparisons be made against the re-run numbers.
What can you do with OSWorld?
- Drive an operating system, not a mock — Tasks run against Ubuntu or Windows with real applications installed, so an agent has to cope with actual dialogs, file managers and application quirks rather than a simplified stand-in built for the benchmark.
- Choose where the machine runs — VMware or VirtualBox on a workstation, Docker with KVM on a server, or AWS, Azure and Modal when runs need to happen in parallel — the environment layer was deliberately separated from the benchmark.
- Finish a full evaluation in about an hour — The Verified update added AWS support with parallelisation, which the project reports brings a complete evaluation down to within an hour rather than the day it previously took.
- Use the environment without the tasks — The desktop environment is published as its own package, so it can be used to build and score your own tasks rather than only to reproduce the published set.
- Compare against re-run baselines — Model results are maintained on the project site and were re-run for the Verified version, so there is a consistent set of numbers to place a new result beside.
Before you choose OSWorld
- The Verified update changed both the tasks and the evaluation signals, and the project asks that results be compared against the re-run numbers — scores from before it are not comparable with scores after.
- Running it is infrastructure work: a hypervisor or a KVM-capable host, virtual machine images to fetch, and containers left behind by interrupted runs that have to be cleared manually.
Star history
21 Aug to 30 Oct · +144
Frequently asked questions
Is OSWorld free for commercial use?
OSWorld is released under the Apache-2.0 licence — OSI-approved open source, which permits commercial use.
How can OSWorld be deployed?
OSWorld is available as Runs locally / Self-hosted.