Skip to content

Operate

Monitoring

The health endpoints, what the logs say, what the Health page and the doctor check, and what is worth alerting on.

CirCI gives you three things to watch: two HTTP probes for the process, structured logs, and the checks behind circi admin doctor and Instance › Health. The background queue has its own numbers on the Health page.

Both answer without a sign-in and don't write a log line per request, because probes poll them every few seconds.

Endpoint Answers Use it for
GET /healthz 200 with { "status": "ok" } while the process serves. Liveness: restart the container when it stops answering.
GET /readyz 200 with { "status": "ok", "checks": { "database": "ok", "storage": "ok" } }, or 503 with "status": "fail" and fail against the check that failed. Readiness: take the instance out of a load balancer while a dependency is down.

/readyz runs the two checks at once. Each has a 2 second timeout, so a hung database or storage can't hang the probe. A failing check is also logged as a warning, readiness check failed, with the check's name and the error.

/readyz doesn't say whether a worker is running or whether jobs are piling up. Watch the queue on the Health page for that (below).

The image's own HEALTHCHECK calls /healthz every 15 seconds, with a 3 second timeout, 20 seconds of start period and 3 retries. It reads PORT (default 3000).

The server and the worker log JSON lines to standard output (they use pino), each with "service": "circi". In Docker Compose, docker compose logs -f circi follows them. The authorization and cookie request headers are removed from log lines.

LOG_LEVEL is one of fatal, error, warn, info (the default), debug and trace.

On a stop signal the process logs shutting down, closes, and exits 0. If closing fails or takes longer than 10 seconds it exits 1, logging error during shutdown or shutdown timed out, exiting.

Level Message Means
warn No accounts yet. Open <URL> and set up CirCI with the setup code <code> A fresh install: finish sign-in and setup.
warn The dashboard address changed from <old> to <new>. Finish the address-change checklist. PUBLIC_URL moved. The checklist is on Instance › Health.
warn lost a database connection The pool reconnects; /readyz reports an outage while it lasts.
warn readiness check failed /readyz returned 503; the line names the check.
warn lost the job notification connection; polling until it is back The worker is still working, by polling; jobs start a little later. job notifications are back follows.
warn lost the lease on a running job, could not extend a job lease A job ran longer than its lease, or the database was unreachable. Its result is dropped if another worker claimed it.
warn failed builds that stopped uploading Builds whose upload never completed were failed after UPLOAD_TIMEOUT_MINUTES.
warn failed jobs whose lease expired too many times A job kept killing its worker (for example out of memory).
error could not claim a job The worker can't read the queue; check the database.
error retention pass failed, review sweep failed, could not expire stalled builds, could not prune finished jobs A periodic sweep failed; the next run tries again.
error the give-up of an expired job failed A job past its retries couldn't be marked failed.
info retention pass finished, deleted finished jobs, told reviewers about overdue requests Routine sweeps.

Each job logs its queue and job ID, so a failing build can be followed from the build ID to its process-build job.

circi admin doctor and the Run checks again button on Instance › Health run the same checks. Each finding is ok, info (a note), warn (look at it) or fail (something CirCI needs doesn't work). The command prints one line per finding, with a fix: line under a warning, and exits 1 if any finding is a warning or a failure, so it can run in a deploy script or a scheduled job.

Terminal window
docker compose exec circi circi admin doctor
Check Fails or warns when
Database Fails if it isn't reachable, if fewer migrations are applied than this version ships (Run circi migrate, or restart: the server migrates on start unless --no-migrate is set), or if a newer version already migrated it.
Storage Fails if the storage isn't reachable and writable.
Disk space Local storage only: warns under 10% free or under 2 GB free.
Mail Fails if SMTP doesn't accept a login (nothing is sent: circi admin mail test --to <address> sends a real message). An info note if SMTP isn't set.
Dashboard address Warns if PUBLIC_URL is plain http, because sign-in cookies need https.
Domains and cookies Warns when a host sets cookies for a whole domain or serves content on the sites domain's apex. Best effort: it only checks hosts it can reach. See Site domains.

Instance › Health also shows Background jobs, per queue: waiting, running, the oldest waiting job, retrying (with the most retried) and failed. See Scaling the worker; each job and what it does is in Background jobs.

  • /readyz returning 503 for more than a minute or two: the database or the storage is down. Run the doctor to see which.
  • /healthz not answering: the process is down. The image's health check marks the container unhealthy.
  • Free space on local storage: the doctor warns below 10% or 2 GB, but alert earlier on your own volume. See Retention and garbage collection.
  • A growing queue: the oldest waiting job in a queue keeps getting older, or failed jobs rise. Check the worker is running and has room: Scaling the worker.
  • Log lines at error, and the warn lines for lost leases and expired leases above.
  • A failing scheduled circi admin doctor.

If something is failing and you don't know what, Troubleshooting goes from the symptom to the cause.