Introduction
How screenshots are compared
Pairing by name, the pixel comparison and its tolerances, ignore regions, size changes, the diff image, and the settings that change the result.
How a build flows says a screenshot is compared with its baseline. This page is the detail of that comparison: what is compared, how two images are judged different, and which knobs change the answer.
What counts as a screenshot
Section titled “What counts as a screenshot”The worker reads every uploaded file as a PNG. A file that isn't one, is empty, is corrupt or truncated, or is larger than 100 million pixels is marked Couldn't read, with its reason, and can't be approved. See How a build flows for what that does to the build.
Pairing by name
Section titled “Pairing by name”A build's screenshots are matched with the baseline build's by name (the file's path without .png). Each name falls into one case:
| Case | Rule |
|---|---|
| Added | The name is only in this build |
| Removed | The name is only in the baseline |
| Unchanged | Same name and byte-identical files (the same SHA-256). No pixels are read |
| Compared | Same name, different bytes: the pixel comparison below decides |
Which build is the baseline is explained in Which baseline?.
The pixel comparison
Section titled “The pixel comparison”For each pair that is compared:
- Both images go on one canvas. The canvas is as wide as the wider image and as tall as the taller one. The smaller image is placed at the top left and the rest is transparent.
- Ignore regions are blanked. Every ignore region of that screenshot name is made transparent on both images, clipped to the canvas (whole pixels). Pixels in those rectangles can't differ.
- Pixels are compared one by one with a perceptual color distance, not by comparing raw values. A pixel counts as changed when its distance goes past a tolerance of 0.1 (on a scale of 0 to 1), so imperceptible noise doesn't count. The tolerance is fixed: no setting or API field changes it. A semi-transparent pixel is judged as blended over a background.
- Anti-aliasing is left out. A pixel that differs only because an edge was smoothed differently isn't counted as changed.
- The score is the share of changed pixels: the number of changed pixels divided by the canvas's width times its height. A 0 means identical, and 0.01 means 1% of the canvas.
- The verdict: the screenshot is changed when its size differs from the baseline's, or when the score is greater than the project's diff threshold. Otherwise it's Unchanged.
A size difference of even one pixel is always a change, whatever the score, because the images don't line up. The page height of a full-page screenshot changing with content is the usual cause.
With the default diff threshold of 0%, a single changed pixel makes a screenshot changed. Raising it lets a share of the canvas differ: at 1%, a 1000 × 1000 screenshot may have up to 10,000 changed pixels and still count as unchanged. The check is on the share of the whole canvas, so the same few pixels weigh less on a tall page.
What is stored
Section titled “What is stored”For each compared screenshot the worker keeps the verdict, the score, the number of changed pixels, whether the size changed, and the diff image. A changed screenshot gets a diff image. An unchanged one has none, even when a few pixels differed within the threshold.
The pixels that ignore regions hid are counted too, with the same tolerance, and per region. That is what Insights shows as the effect of ignore regions. See Insights.
The diff image
Section titled “The diff image”The diff is a PNG the size of the canvas. It shows a faded copy of the screenshot (15% opacity) with the changed pixels drawn over it in red. The build page shows it in Diff, where the This build pill sets how much of the build shows under the red. See Reviewing a build.
Thumbnails
Section titled “Thumbnails”For lists, the worker makes a WebP thumbnail of each screenshot: the top of the image, at most 480 pixels wide and 720 tall, never enlarged.
The settings that change the result
Section titled “The settings that change the result”| Where | Setting | Effect on the comparison |
|---|---|---|
| Project settings › General › Comparison | Diff threshold | The share of the canvas that may differ before a screenshot counts as changed. Stored as a share from 0 to 1, shown as a percentage from 0 to 100. Default 0%. See Project settings |
Baseline page, build page, or circi admin ignore |
Ignore regions | Rectangles left out of the comparison, per screenshot name. See Baselines and ignore regions |
| CI | --name build name |
A baseline is only taken from a build with the same name, so each name has its own baseline history |
| Server | CIRCI_DIFF_ENGINE, CIRCI_DIFF_THREADS |
Where the pixels are compared, below. Not what counts as different |
The threshold is read when a build is processed.
The comparison engines
Section titled “The comparison engines”CIRCI_DIFF_ENGINE picks where the pixel comparison runs. Decoding and encoding the images are the same either way.
| Value | Runs |
|---|---|
native (default) |
A Rust addon the image ships, which uses several threads |
js |
The JavaScript comparison on the worker's event loop |
js-workers |
The same JavaScript comparison in a pool of worker threads, CIRCI_DIFF_THREADS of them (default one less than the cores, at least 1) |
If native can't load on the machine, or the js-workers script can't be found, the worker logs a warning at start and uses js. A js-workers thread that dies mid-comparison makes that comparison run in-process, with one warning. See Environment variables and Performance for throughput.