Why can’t you measure DevOps by popular metrics?

Peter Drucker, a famous management consultant, once said, “If you can’t measure it, you can’t improve it.” This is true because without knowing where you are, you cannot move forward. When you order a DevOps services you want to see results. So how do you measure—and improve—the work of your DevOps team? In this article, I will share which key performance indicators you should check, and which will work against you.

I recently had a conversation with a friend who said his manager wanted to measure the team’s workload to make sure it was working at full capacity. In fact, the manager felt that if the teams didn’t keep up with their commitments, they were probably not working hard enough. I often hear this attitude from many companies. But let’s look at this in terms that most engineering managers should understand.

Imagine this scenario: you want to make sure DevOps team is “meeting their commitments”. They promised they would deliver something, and you have to hold them to it, right? But they seem to be failing to accomplish a good amount of work. In the last five sprints, the team committed 50 story points out of 100. That doesn’t look very productive, does it? Therefore, you will use this ratio to improve their performance. It appears to work. In this new sprint, the ‘lazy’ team decided to fulfill their commitments. They doubled their speed!

You are satisfied that you have managed the situation successfully. Now everyone is happy…except your customer, who is now dealing with a buggy application. What went wrong here? Let me show you. 

Key performance indicators you could measure… and why they will not work.

There are many Key Performance Indicators (KPI), but I will focus on the main ones that seem like they would be helpful…but instead can be misleading.

Lines of Code

You might think that if a DevOps team is typing more lines of code per day, they are more productive. Therefore, it seems as if you would want to measure lines of code, or the number of ‘commits’, per day for each developer. But in reality, elite developers spend more time on thinking than typing. 

Read also: How to reduce cloud-operating costs

The code is only the documentation of your DevOps’ planning converted to a format that a computer can understand. It is better to have a smaller number of high-quality lines, because such small enhancements can result in the deletion of over 5,000 lines of code. Impressively efficient, right? 

In addition, delve into DevOps Support Services.

In other words, if you measure your DevOps team’s productivity by lines of code, they could just add more code in order to look more productive. Copying and pasting older code is much easier than spending the time on developing a smarter business solution.

Number of Defects Fixed

It may seem that if we measure the number of defects fixed, there will be fewer defects as a result. But in reality, high-quality work results in a smaller number of defects in the final product. So, if you are focusing on the number of defect tickets the team has closed, you are prioritizing quantity over quality. Moreover, if you pay a bonus for defects fixed you create an opportunity for the team to create more defects, or waste time finding more minor defects instead of addressing larger critical issues. 

Read also: DevOps as a service

Committed vs. Completed:

If we measure DevOps team by how much they complete versus the schedule to which they commit, they will work hard to meet those deadlines and deliver above and beyond! Or at least, that is how this would work in a perfect world. In reality, your DevOps team may prioritize sticking to a schedule over producing quality code. Defects and underlying issues could be ignored. Measuring your DevOps team by ‘committed versus completed’ means that you care more about the plan than the value.

As Daniel Ariely, one of the founders of behavioral economics, has noted: “Human beings adjust behavior based on the metrics they’re held against. Anything you measure will impel a person to optimize his score on that metric. What you measure is what you’ll get. Period.”

Velocity/Throughput:

Are you still thinking that higher velocity/throughput means a DevOps team is more productive? Velocity or throughput give us an average of how many things the team have completed during a fixed period of time. But if you measure DevOps in this way, they could create more tasks or increase the number of story points per task just to satisfy your requirement. It may look good on paper, but in fact this leads to a higher number of mistakes and bugs in the long run.