
Midscene.js
Writes UI tests as sentences and drives the screen from the pixels, so there are no selectors to fix after a refactor
What is Midscene.js?
UI tests break for reasons that have nothing to do with the product: a class name changed, a wrapper was added, a component moved. Midscene removes the selector entirely. Steps are written as sentences, the model locates elements from the screenshot, and assertions can be about what a person would see — the row is highlighted, the layout has not collapsed — rather than about whether a node exists. That also reaches surfaces the DOM cannot: canvas drawings, icon-only buttons, native mobile applications, cross-origin frames. The trade is per-step cost and determinism, because each step is now a model call rather than a lookup, so a suite that runs in seconds with selectors will not run in seconds here.
What can you do with Midscene.js?
- Describe the step instead of locating the element — A sentence replaces the selector, so a refactor that renames classes or moves a component does not send you back through the test suite.
- Assert what a person would see — Checks can be about colour, highlighting and layout — the kinds of breakage that pass a test asserting only that an element exists.
- Reach the parts the DOM cannot describe — Canvas drawings, icon-only controls, native applications and cross-origin frames are all just pixels here, so they are as reachable as anything else.
- Use the same script on phone and browser — One API covers web, Android, iOS, HarmonyOS and desktop, so a flow tested on the site can be run against the app it mirrors.
- Pull structured data off a page — A query returns what is on screen as data rather than text, which turns a test tool into a way of extracting from interfaces with no API.
- Start in the browser before writing code — A Chrome extension runs natural-language instructions against a page, so the approach can be evaluated before any test file exists.
Before you choose Midscene.js
- Every step is a model call, so runs are slower and cost money per step, and two runs of the same suite can differ — which is a real change if your tests currently gate every merge.
- It needs a multimodal model that is good at locating things on screen; the project names the ones it works with, and a general-purpose model that is not on that list will place clicks badly.
Star history
27 Aug to 30 Oct · +727
Frequently asked questions
Is Midscene.js free for commercial use?
Midscene.js is released under the MIT licence — OSI-approved open source, which permits commercial use.
How can Midscene.js be deployed?
Midscene.js is available as Runs locally / Self-hosted.