← Back to all projects

Midscene.js

Writes UI tests as sentences and drives the screen from the pixels, so there are no selectors to fix after a refactor

MIT
Stars
15.4k
Forks
1.2k
Open issues
103
Last commit
30 Oct 2026

What is Midscene.js?

UI tests break for reasons that have nothing to do with the product: a class name changed, a wrapper was added, a component moved. Midscene removes the selector entirely. Steps are written as sentences, the model locates elements from the screenshot, and assertions can be about what a person would see — the row is highlighted, the layout has not collapsed — rather than about whether a node exists. That also reaches surfaces the DOM cannot: canvas drawings, icon-only buttons, native mobile applications, cross-origin frames. The trade is per-step cost and determinism, because each step is now a model call rather than a lookup, so a suite that runs in seconds with selectors will not run in seconds here.

What can you do with Midscene.js?

  • Describe the step instead of locating the element — A sentence replaces the selector, so a refactor that renames classes or moves a component does not send you back through the test suite.
  • Assert what a person would see — Checks can be about colour, highlighting and layout — the kinds of breakage that pass a test asserting only that an element exists.
  • Reach the parts the DOM cannot describe — Canvas drawings, icon-only controls, native applications and cross-origin frames are all just pixels here, so they are as reachable as anything else.
  • Use the same script on phone and browser — One API covers web, Android, iOS, HarmonyOS and desktop, so a flow tested on the site can be run against the app it mirrors.
  • Pull structured data off a page — A query returns what is on screen as data rather than text, which turns a test tool into a way of extracting from interfaces with no API.
  • Start in the browser before writing code — A Chrome extension runs natural-language instructions against a page, so the approach can be evaluated before any test file exists.

Before you choose Midscene.js

  • Every step is a model call, so runs are slower and cost money per step, and two runs of the same suite can differ — which is a real change if your tests currently gate every merge.
  • It needs a multimodal model that is good at locating things on screen; the project names the ones it works with, and a general-purpose model that is not on that list will place clicks badly.

Star history

27 Aug to 30 Oct · +727

14.7k15.4k

Frequently asked questions

Is Midscene.js free for commercial use?

Midscene.js is released under the MIT licence — OSI-approved open source, which permits commercial use.

How can Midscene.js be deployed?

Midscene.js is available as Runs locally / Self-hosted.