Great! You’ve got your CICD pipeline up and running, so… what now?
Having a continuous deployment pipeline is now table stakes for any business with in-house development. The ability to manage your code and deploy it through the software development lifecycle with visibility and automation is now the new norm for most businesses.
But, now that you have your shiny new deployment pipeline setup – how do you know that it’s making the impact that you wanted it to? and how do you find ways to continuously improve its effectiveness?
In this article, we’ll look to break down the ways businesses can measure how effective their CICD pipeline is and whether they’re getting a return on the investment.
We’ll break it down into these parts:
- Why it’s important to have a CICD pipeline
- Importance of having CICD metrics
- What metrics should you be looking at
- What are some ways to collect this data
- How do you communicate that data to your stakeholders
The not-so-new way to deploy code
There’s no question that having a CICD pipeline is the standard way to develop software. Before the days of continuous integration – there were much bigger integration efforts that happened much less frequently, with much bigger chunks of code. This caused much anxiety for developers as it meant a potential plethora of bugs and outdated work as so much time had passed between releases.
This then gave birth to the idea of integrating smaller increments on a continuous basis, some have traced this idea back to the mid to late 90s. Ah the 90s… Spice Girls, Blockbuster, and Tamagotchis. What a time to be alive!
In saying all this though, CICD may not provide you as much value if you’re a smaller team of developers. A recent State of Continuous Delivery report by the Continuous Delivery foundation found that 32% of developers are already using continuous integration to automatically build and test code changes. The majority of this group will likely belong to enterprise organizations that require the automation to scale effectively. So, if you’re an enterprise organization with a team deploying manually – chances are that you’re going to start feeling the pain of scale, if you haven’t already.
No metrics = no data = maybe ROI?
First off, don’t measure anything for the sake of measuring something. There’s a constant barrage of thought leaders telling you that data is the new gold and to measure every little thing. While there is truth in that statement – data without intention that can’t be turned into insights and actions is just random numbers.
Before you take the plunge and immerse yourself in a never-ending ocean of data, make sure you have a well-defined business goal in mind that you want to achieve. It is important to remember that data is only useful if you are able to leverage it to support your overall business objectives.
Without a pre-established goal, it can be very easy to get lost in the data and forget the actual purpose of why you began the analysis in the first place. Additionally, without a clear plan of action, you may find yourself overwhelmed, and unable to effectively utilize the data to its full potential. So, before you jump in, make sure you have clear and concise goals in mind!
Yay! Metrics and data!
Remember to work your way down from the business objective. The pipeline metrics that you’re looking to track will depend on whether you’re trying to answer one of these questions:
- Are we trying to ship more features faster?
- What pains are our engineers feeling right now?
- Are we introducing too many defects to production?
These are just a few of the questions that you should be asking your stakeholders before investing in your CICD pipeline and data tools.
Without clear answers and intentions behind these questions, you’ll be left with a smorgasbord of dashboards and charts with no real actionable insights.
We won’t leave you empty-handed though. Here are some common metrics that most teams should be looking at. But, again, only in the context of how they support the overarching business objectives:
- Number of successful deployments – while this number alone won’t tell you much. It will allow you to compare to previous periods.
- Time in deployment stages – one of the goals of CICD is to improve efficiency. If your deployments are getting stuck at certain stages, then your pipeline isn’t working as intended.
- False-positive notifications – visibility is another value proposition of an effective CICD. Being able to notify the right teams at the right time is crucial for an effective pipeline. Having visibility into the number of false-positives over a period of time will tell you if this area is working as intended or not.
- Mean time to restore – this is the time it takes to restore service from an unplanned outage. The ability to roll changes forwards and back is crucial to your business continuity.
Measure your CICD pipeline and get that sweet sweet data
Now that you know what you need to track, how do you get your hands on that data? Well, there are a number of options out there. Most of the CICD tools have native reporting that will satisfy the many use cases out there. Here’s just a few you could already be using:
- Jenkins – a popular open-source tool for CI/CD, allowing automation of build, test and deployment processes, with a range of plugins for integration.
- TravisCI – cloud-based CI tool used by open-source projects. It integrates with GitHub and offers an easy setup process and plugins.
- GitLab CICD – provides a complete Continuous Integration and Continuous Deployment solution, integrated into the GitLab code repository and issue tracking system.
- AWS CodePipeline – Amazon’s cloud native solution for CICD. Providing a streamline and scalable solution that integrations with a range of AWS services.
For many enterprises though, you’ll likely already be aggregating data into monitoring and logging tools. Luckily, many of these tools offer a range of insights and analytics for CICD. Here’s a few that are particularly popular and provide deep insights into CICD:
- Datadog – provides real-time insights into infrastructure, apps, and logs via a cloud-based monitoring & analytics platform, integrating with a variety of tools & tech. Offers a unified view of your stack.
- New Relic – cloud platform to monitor and optimize apps, infra, and digital exp. New Relic provides real-time insights into performance, errors, and user behavior, and integrates with many technologies.
- Grafana – an open-source platform for visualizing and analyzing data from multiple sources such as Prometheus, InfluxDB and Elasticsearch, with a range of visuals and dashboards.
Conclusion and recap
Moral of the story: make sure to gather key metrics on your CICD pipeline. But, more importantly – make sure that these metrics align with your business objectives. There are far too many arbitrary metrics floating around today. While they may look great and form graphs that point in the right direction, they may not necessarily correlate to the bigger picture. Business Value.
Key takeaways
- Understand why it’s important to measure your CICD pipeline
- No measurement means no way of realizing business value
- Ensure that your metrics align with your business objectives
- There are many monitoring tool choices. Pick the right tools to get the data that fits your requirements
Is your business getting value from your CICD pipeline? Are you improving the development experience for your engineers? Are your customers getting high-quality features in their hands as quickly as possible. Answer these questions with your data and you’ll be on your way to operating a highly effective deployment lifecycle with your CICD pipeline.
If you’re looking to implement or improve CICD in your development lifecycle, reach out to us at Autimo! We live and breathe DevOps and Cloud. We can do a free 30 minute consultation and work out whether there’s opportunity for your team to leverage the benefits of CICD.
