Screenshots over time
Triage and flaky screenshots
The project's triage queue for flaky screenshots, missing baselines and failed builds, who owns it each week, and how flakiness is decided.
Triage is the queue of problems with a project's default branch that no single pull request owns. Where a review is about one build, triage is about the screenshots and builds that keep causing trouble. It's a page of the project, in the sidebar under the open project, with a count of items waiting. Anyone who can see the project can work the queue.
What is in the queue
Section titled “What is in the queue”| Group | What it's |
|---|---|
| Flaky | A screenshot that keeps changing on the default branch without anyone meaning it to |
| No baseline | A screenshot with no approved baseline yet. The pane shows "First seen in build #N" |
| Failed | A default-branch build that failed to process. The pane shows its error |
Each row shows the screenshot or build, why it's here, the assignee, and how long ago it was first seen. The pane on the right adds the detail: for flaky items Why it's flaky, a strip of the last 10 commits on main with ~ ("changed in that build") marking the ones where the screenshot changed, and a preview (Where it changes) with Open in build; plus the assignee, how long the item has been in triage, and what it's Blocking ("The check on this commit is failing", or "3 pull requests show it as changed").
The count next to the title is the number of items; "2 handled" counts what you dealt with since the page opened. "Triage is clear" means nothing is waiting.
Work an item
Section titled “Work an item”Select a row and use the pane's buttons or the keys. Each action takes effect at once, and the toast offers Undo, except Re-run comparison.
| Action | Key | Does |
|---|---|---|
| Open in build | Enter (or double-click) | Opens the build with the screenshot selected |
| Re-run comparison | Failed items only. Compares the build again; it can't be undone | |
| Assign to me / Unassign me | m | Takes the item or gives it back |
| Snooze | h | Until next push, Tomorrow, Next week, or a Custom date |
| Unsnooze | h | On a snoozed item: brings it back into the queue |
| Dismiss | d | Takes the item out of the queue |
"Until next push" wakes the item when a new build arrives on the default branch. If you don't choose a time, a snooze lasts the number of days in the project's Snoozed items come back after setting. Right-click a row for the same actions.
Snoozed items
Section titled “Snoozed items”Snoozed items leave the queue and the count and wait in a Snoozed group at the end of the list, collapsed until you open it. Each row says when it wakes ("Snoozed until Mon", "until next push"). On a snoozed item the pane has Unsnooze where Snooze was, and h, the right-click menu and the command menu offer Unsnooze too. It acts at once and the toast, "Unsnoozed", has Undo. A link to a snoozed item opens the Snoozed group with that item selected, so a shared link still works. While the group is open, j and k walk its rows too.
When an item leaves the queue the next row is selected, so you can keep going with the keyboard.
The weekly rotation
Section titled “The weekly rotation”A project can have a triage rotation: its members take turns, one calendar week each (Monday to Sunday, UTC). The header shows a chip such as "You're on triage until Mon" when it's you, and the name of whoever is on triage otherwise. With Assign new items to whoever is on triage on, unassigned items are assigned to the person on triage, and move to the next person when the week changes. Items someone assigns or unassigns by hand stay as they left them. The person gets an Inbox notice, "Assigned to you as the person on triage". See Inbox and reviews.
The rotation also catches builds that no review rule matches: those are requested from whoever is on triage that week.
All of this is set by owners and admins in Project settings › Reviews and triage: the people in the rotation, the first week's start, whether new items are assigned automatically, the flaky threshold and the snooze default. See Project settings. The workspace sets the default for automatic assignment, which each project can override; see Workspace settings.
How a screenshot becomes flaky
Section titled “How a screenshot becomes flaky”CirCI looks at the last 10 commits on the default branch (several builds of the same commit count once). A screenshot is flaky when either of these holds:
- Flip-flops. Its image keeps returning to one it already had in that window: A, B, A is one flip-flop, A, B, A, B is two. A steady redesign (A, B, C, D) never counts, because nothing came back. It's flaky at 3 flip-flops by default; the project can set anywhere from 2 to 8.
- Same commit, different images. Two builds of the same commit produced different images, so one such commit is enough whatever the threshold.
Flaky screenshots show up in several places:
- the Flaky group (collapsed) on the build page, instead of Changed. See Reviewing a build;
- a banner on the project's builds page, and a chip on the screenshot's row on the Baselines page (when its triage item is snoozed, the chip's tooltip says until when). See Baselines and ignore regions;
- a Flaky chip on review requests;
- the "Flakiest screenshots" list and flaky chart in Insights;
- an Inbox notice, "Flaky screenshot found", for the person on triage that week.
Flaky changes and the merge gate
Section titled “Flaky changes and the merge gate”When every change in a pull request's build is to a flaky screenshot, and each change looks like variation already seen (it's an image the screenshot already had on the default branch, or it differs no more than the largest difference seen there, at the same size), the gate can let it pass: "Only flaky screenshots changed". Whether it does is the Flaky-only changes block merging setting. The workspace sets the default (blocking, unless changed), and each project can follow it or override it. A change that breaks the layout of a flaky screenshot still needs a person.