Measure a CI/CD Pipeline: Events, Metrics, and Bottlenecks

Measure a CI/CD pipeline to answer an operating question: why do changes wait, which stage causes rework, or how often does a production deployment need repair? Start with one application and one question. Collect enough evidence to choose a change, then measure whether it helped.

Build duration describes one part of the system. It does not include every wait before production, and a green build does not prove that a release works for users. Keep pipeline efficiency, production delivery performance, and financial return as separate views.

Define events before building a dashboard

Export events from the version-control system, CI runner, deployment system, and incident record. Keep timestamps in one time zone and join records using a stable service identifier, commit SHA, pipeline run ID, and deployment ID. Include the target environment so a staging deployment cannot be mistaken for production.

• Change: commit identifier and commit timestamp, with the mapping to the artifact that contains it.
• Pipeline: queued, started, and finished timestamps; outcome; and whether the run was cancelled or retried.
• Stage: stage name, start, finish, and outcome for build, test, security checks, approval, and deployment.
• Production deployment: artifact, environment, completion time, and validation outcome.
• Recovery: failed deployment reference, impact start, recovery confirmation, and whether the response was rollback or repair.

Retried jobs need their own attempt records under the original run. Otherwise successful retries can hide flaky tests or failed first attempts. Version the event definitions when you change what counts as a deployment or failure.

Separate waiting from execution

Calculate queue time as runner start minus queue entry, then execution time as finish minus runner start. Record approval waits separately. Report sample counts and a distribution, such as the median and 95th percentile, so a small number of slow runs does not disappear inside an average. With very few runs, inspect each one instead of overinterpreting percentiles.

In a hypothetical run, a change is committed at 09:00, queued at 09:02, starts at 09:10, finishes its automated work at 09:22, and reaches production after approval at 10:00. Queue time is eight minutes, execution is 12 minutes, and commit-to-production time is 60 minutes. Cutting execution by three minutes saves only three minutes of that total if the other waits stay unchanged.

Use the evidence to choose a response. Long queue waits suggest checking runner capacity and scheduling. A slow test stage suggests examining test setup and repeated work. Long approval waits need a workflow decision. Adding runners will not remove a manual approval delay.

Connect pipeline events to production delivery

DORA defines five software delivery performance metrics. Calculate them per application or service:

• Change lead time: elapsed time from commit to production, rather than from an idea or change request.
• Deployment frequency: production deployments over a stated period, or elapsed time between them.
• Failed deployment recovery time: time to recover from a deployment that fails and requires intervention, rather than every kind of service incident.
• Change fail rate: the proportion of deployments that need immediate intervention.
• Deployment rework rate: the proportion of deployments that are unplanned responses to production incidents.

For an illustrative period with 20 production deployments, two that require immediate intervention give a 10% change fail rate. A failed unit-test job that blocks a release is not itself a failed production deployment. Tag unplanned recovery deployments separately when calculating rework.

Define how batched commits, multiple artifacts, hotfixes, and rollbacks map to your service's deployment record. Retain missing events as a data-quality issue rather than silently assuming success. Do not combine unrelated services into a league table: different release patterns can make the comparison misleading.

Choose one improvement and protect service quality

Take a representative baseline and name the largest observed constraint. Change one part of the process, then compare equivalent periods. Track test reliability, deployment failures, and user-facing health alongside speed. Removing a check can shorten a pipeline while moving expensive failures into production.

For example, after changing dependency caching, compare the same stage across similar workloads and record cache misses. Include failed and cancelled runs in the investigation. Faster successful runs alone do not show whether the whole delivery process improved.

Assign a service owner to review the trend and investigate exceptions. Retain only fields needed for the question, restrict access to logs, and exclude credentials and customer payloads from analytics events. Tool-native reports or an existing observability system may be enough before adding another platform.

Allocate runner and supporting infrastructure costs through the cloud cost management process. Convert observed effort changes into an investment case with the DevOps ROI worked example. For delivery automation work, see platform engineering; for a broader process review, the DevOps maturity model.