Measuring Software Delivery Performance With DORA

Not sure when the last version of Windows shipped on a CD? In today’s fast-moving digital age, it would appear to be in the distant past. The current expansion of the CD acronym would be continuous delivery, not compact discs. That may give you an idea of how long it has been since software stopped being shipped on CDs.

But why do organizations continue to operate with shipping on a CD, aka the big bang release mindset, for their software products and applications?

Not sure why this practice of a bygone era continues to persist? Join a panel of Agile & DevOps experts and me to find out why and what to do about it at the upcoming DASA Enterprise Leadership Forum event. The event’s topic is Measuring Software Delivery Performance with DORA (DevOps Research Assessment) metrics.

Watch the webinar recording

 Measuring software delivery performance with DORA

Why software delivery performance matters and the urgent need to shift from the shipping-on-CD mindset to delivering software continuously mindset: In the bygone era of compact discs, the development of software products was separate from their operations. But then came the likes of Amazon, Google, Netflix, and other digital native companies. Software’s design, development, delivery, and operations phases stopped being distinct. Instead, it became a continuous cycle of software delivery, learning from the software’s usage and improving it based on the learning.

In tandem, driven by digital natives and the explosive growth of mobile devices, consumers increasingly sought to interact with organizations through digital means. And consumers’ digital experience expectations of all enterprises are now based on the expectations set by digital natives.

So, why is it necessary to shift from big bang releases to delivering software continuously? First, because it is possible. Both digital natives and high-performing incumbents, who have embraced digital age technological and cultural practices, routinely do it. Second, and more importantly, because it is a business imperative. 

Mastering software delivery at scale is the means of production in the digital age. So, any business that wants to operate at scale is a software business that needs the capabilities to build and operate software systems at scale. This profound insight of ING’s CIO, talking about ING’s Agile & DevOps transformation, sums up the urgent need to let go of the practices of the bygone era and embrace the practices of the digital age: “We came to the realization that ultimately, we are a technology company operating in the financial-services business.”[i]

Improving software delivery performance with DORA metrics and DevOps:

Measuring software delivery performance is the first step to improving it, but measuring it is fraught with challenges. 

The waterfall-era practice of measuring based on lines of code is flawed. This practice measures outputs, not outcomes.

Measuring team velocity per iteration became popular with agile software development. While it is helpful in specific contexts, it isn’t a good measure of software delivery performance. It is a team-dependent measure. Because it is team-focused, not system-focused, it inhibits collaboration across teams when used as a measure of software delivery performance. 

As the authors of the book Accelerate, [ii]building and scaling high-performing technology organizations, Nicole Forsgren, Jez Humble and Gene Kim, and the originators of the DORA metrics rightly point out, a successful measure of software delivery performance should have two key characteristics.

The first characteristic is that the performance measure should focus on a global outcome, not a local outcome. A local outcome risks pitting teams and functions against each other. A well-known example of local optimization is optimizing development for throughput and operations for stability.

The second characteristic of a good performance measure is it should focus on outcomes, not outputs.

The DORA metrics meet those two characteristics and are increasingly the de facto standard for measuring software delivery performance.

The four DORA metrics are:

  • Lead time: the time from code commit to code deploy 
  • Deployment frequencies: how often changes ( code, configurations) are deployed 
  • Mean time to restore service in case of a failure
  • Change fail percentage 

Improving an organization’s software delivery performance requires organizational investments. Organizations must invest in the categories of capabilities: technical process, cultural, and measurement capabilities to improve their software delivery performance continually.

DORA-capabilities-model

Source: https://www.devops-research.com/research.html

The essence of DevOps is to facilitate the people, process, and cultural transformation across the organization to achieve superior software delivery performance and organizational performance.

DevOps Antipatterns: However, in practice, DevOps has different meanings for different people. Due to this confusion, organizations don’t realize the benefits of DevOps. Here are some antipatterns.

Agile is for development teams, and DevOps is for the downstream teams: Wrong. DevOps is about building end-to-end product teams, who own the product and are responsible for it throughout its lifecycle. These are the teams Amazon and Spotify popularized: the two pizza teams with build-and-run responsibilities.

The DASA DevOps team competency model  illustrates the skills and knowledge areas of an end-to-end responsible DevOps team.

DASA DevOps Competence Model

DevOps is a CI/CD pipeline: Another common antipattern is to confuse a CI/CD pipeline with DevOps. DevOps is more than implementing a CI/CD pipeline. It is about embracing the technical and cultural practices that enable the smooth flow of small batches of work from development to operations, getting continuous feedback across the delivery pipeline, including from operations, and continually improving the process and the product based on feedback. 

The mythical DevOps engineers: This antipattern is widely prevalent and perhaps the most stressful for the unlucky person. Instead of the entire organization owning the improvement of software delivery performance, a person is responsible for it. Predictably, this doesn’t work. Instead of DevOps breaking silos, organizations create another silo and a bottleneck – the stressed-out DevOps engineer.

So, what is the antidote for these antipatterns? It is DevOps culture and transformational leadership. 

Culture is hard to measure and harder to change, so it gets neglected. Without the right culture, all the enormous technological investments will not improve software delivery and organizational performance. 

Thanks to Jez Humble et al., we have Westrum’s typology of organizational culture. We can use it to measure and change the culture. Ron Westrum defined these three types of organizational culture[iii].

westrums typology of organizational culture

Source: Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations Paperback

Through surveys, observations, and conversations, we can quickly identify our organization’s cultural type. If our organizational culture is pathological or bureaucratic, we need to work on transforming it into a generative culture. A generative culture is necessary for high software delivery performance and organizational performance.

The way to achieve cultural transformation is through practices. As the lean  movement in manufacturing has confirmed, the “way to change culture is not first to change how people think, but instead to start by changing how people behave – what they do.”[iv]

As research by the DORA report has confirmed, the technical practices of continuous deliveries and agile and lean management practices contribute to transforming an organization’s culture into a generative culture.

Westrum Organizational Culture's Drivers

Westrum Organizational Culture’s Drivers

Although, in popular perception, DevOps is associated with a grass root movement, transformational leaders with authority and budget are necessary for a successful DevOps transformation. The attributes of a transformational leader can be measured along five dimensions:

  •       Vision
  •       Inspirational communication
  •       Intellectual stimulation
  •       Supportive leadership
  •       Personal Recognition

So, here is a quick recap of what is in store at the upcoming ELF event.

  •       Why software delivery performance matters
  •       How to measure and improve software delivery performance
  •       The DevOps antipatterns
  •       The critical role of culture and transformational leadership in a successful DevOps transformation

[i][i] https://www.mckinsey.com/industries/financial-services/our-insights/ings-agile-transformation

[ii] Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations Paperback – Illustrated, March 27, 2018

[iii] Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations Paperback – Illustrated, March 27, 2018

[iv] Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations Paperback – Illustrated, March 27, 2018

Subscribe to Our Mailing List

Stay up to date with the latest happenings at the DevOps Agile Skills Association (DASA).

Irfan Shariff

DASA Research Director, Expert Program - Project Manager, Management trainer & Agile/DevOps coach, RTI Consulting Services

Irfan Shariff has extensive experience in managing strategic technology-based business initiatives in a variety of industries – Hi-tech, Financial, Retail. He has deep expertise and…