Principle-based DevOps Frameworks

Educational Foundations | Reading time: 11 minutes

In our first two articles in the DevOps track of Educational Foundations, we:

  • Dispelled some DevOps misconceptions and presented our own principle-based definition of DevOps
  • Presented the business case for DevOps, including the problems movements like Lean and Agile attempt to solve, as well as metrics from some of the latest research on high-performing DevOps organizations

In this article, we’ll discuss the following three principle-based DevOps frameworks, as well as the common themes we can take from them to apply to your own DevOps transformation:

  • The Three Ways as described in The Phoenix Project and The DevOps Handbook
  • Mature capabilities in technical and management practices found in high-performing DevOps teams, based on the research presented in Accelerate
  • The CALMS (Culture, Automation, Lean, Measurement, Sharing) framework for assessing DevOps

Framework #1: The Phoenix Project/DevOps Handbook’s Three Ways

If you’ve read either The Phoenix Project or The DevOps Handbook, you’ve been introduced to The Three Ways framework for DevOps:

  • The First Way: Principles of Flow
  • The Second Way: Principles of Feedback
  • The Third Way: Principles of Continuous Learning

The First Way: Principles of Flow

The First Way is mostly concerned with accelerating the “flow” of work throughout a process. Gene Kim also refers to the First Way as Systems Thinking in his article The Three Ways: Principles Underpinning DevOps. Whether you’re calling it Flow or Systems Thinking, the principles underpinning the First Way are working toward the same end: viewing the flow of work as one continuous system (unsiloed) that can be continually refined and optimized.

Some of the key principles of the First Way are:

  • Making work “visible”. Unlike manufacturing processes, which are easily observable on a plant floor, the flow of software through its development lifecycle is not easily seen. Using methods such as Kanban boards can surface the activities going on behind the scenes, by showing the left-to-right movement of a user story through the development phases.
  • Limiting work-in-progress (WIP). Keeping work-in-progress to a minimum has also been shown to accelerate work flow, because it minimizes multi-tasking and context-switching.
  • Reducing batch sizes. “Chunking” work into smaller pieces like a two-week sprint can also help deliver features (albeit smaller ones) and bug fixes to the customer faster. Issues are often caught earlier when those updates and additions are released sooner.
  • Reducing hand-offs between teams. The risk of “dropping the baton” increases as the hand-offs do. Although hand-offs can’t be completely minimized, the key is to keep the teams in tight communication with one another so that the hand-off itself is almost a non-event rather than a large ordeal with the potential for communication missteps along the way.
  • Identifying and removing constraints and waste. Constraints might be bottlenecks in the process, such as environments, test setup, and overly tight architecture, while waste includes things like manual work, heroics, and context-switching.

The Second Way: Principles of Feedback

The Second Way works to enable fast and constant feedback cycles throughout all stages of a development cycle.

Some of the key principles of the Second Way are:

  • Swarming and solving problems to build new knowledge. This principle fits into the “fail fast” mentality, so that teams can find issues with an implementation as soon as possible and address them early and often as iterations continue.
  • Pushing quality closer to source. This principle is at the core of the DevSecOps movement, which is concerned with addressing security concerns during the development cycle, instead of at the end, when rework to remediate is more difficult and costly.
  • Optimizing for downstream work centers. This principle works against the “throw it over the wall” mentality, by underscoring that development should be just as invested in their application being deployable, working with operations to bridge that gap (and vice versa).

The Third Way: Principles of Continuous Learning

The Third Way seeks to create a culture of continual learning and experimentation within the development organization.

Some of the key principles of the Third Way are:

  • Enabling organizational learning and a safety culture. Leaders must help “set the tone” for the organization, making it okay to learn, make mistakes, and try again.
  • Institutionalizing the improvement of daily work. Improving what you do and how you accomplish it should be part of everyone’s daily thinking and call to action.
  • Transforming local discoveries to global improvements. Surfacing and sharing improvements at all levels will help enable a “bubble up” culture of continuous improvement.
  • Injecting resilience patterns into daily work. Some examples might include rehearsing failures, and working toward improving key metrics for deployment.
  • Leaders enforcing a learning culture. Organization-wide learning is unlikely to take hold and become pervasive unless it is sanctioned and exemplified by its leaders. So being intentional about communicating the value of learning and problem-solving is crucial to building that culture.

Framework #2: Accelerate’s Technical and Management Practices of High-Performing DevOps Teams

In Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations, authors Nicole Forsgren, Jez Humble, and Gene Kim present research that surfaces the key technical and management practices that high-performing DevOps teams have adopted and continue to refine.

Technical Practices

According to Accelerate authors, the technical practices used by high-performing DevOps teams are focused in three key areas:

  • Continuous Delivery
  • Architecture
  • Product and Process

Continuous Delivery

The Accelerate authors chose to combine several different practices, each important on its own as a discipline, under the umbrella of continuous delivery (CD). Although CD itself is its own principle, keep in mind that high-performing DevOps teams are doing all of these things in concert with one another to achieve a truly exemplary continuous delivery model:

  • Version Control
  • Deployment automation
  • Continuous integration (CI)
  • Trunk-based development
  • Test automation
  • Test data management
  • Shift left on security (DevSecOps)
  • Continuous delivery (CD)

Architecture

When it comes to DevOps architecture considerations, the characteristics that most high-performing teams were likely to agree with were the following:

  • “We can do most of our testing without requiring an integrated environment.”
  • “We can and do deploy or release our application independently of other applications/services it depends on.”

The Accelerate authors are quick to mention that the type of technology used, no matter how popular, is no guarantee of high performance:

“…employing the latest whizzy microservices architecture deployed on containers is no guarantee of higher performance if you ignore these characteristics.”

In fact, the characteristics described above—testability and deployability—are achieved by implementing the following two architectural technical principles of high-performing DevOps teams, as Accelerate authors explain:

  • Loosely coupled architecture. “The goal is for your architecture to support the ability of teams to get their work done—from design through to deployment—without requiring high-bandwidth communication between teams.”
  • Empowered teams. “When teams can decide what tools they use, it contributes to software delivery performance, and, in turn, organizational performance.”

Product and Process

High-performing DevOps teams have also embedded the following key principles into their product management processes:

  • Customer feedback. Product teams seek out customer feedback early and often throughout the development process.
  • Value stream. Product teams understand that continual optimization of their process delivers value to customers faster.
  • Working in small batches. Smaller batches of work allows delivering of MVPs, features, and bug fixes sooner, which also helps enable the customer feedback loop above.
  • Team experimentation. Teams perform better when they’re allowed to test out new ideas and theories without outside approval.

Management practices

According to Accelerate authors, the management practices used by high-performing DevOps teams are focused in two key areas:

  • Lean Management and Monitoring
  • Cultural

Lean Management and Monitoring

In our Why DevOps? article, we discussed how Lean grew out of the need to deliver value to customers faster by identifying and removing waste from a manufacturing process, or (put another way) by reducing lead times through optimization of the value stream.

Some of the key management practices of Lean management and monitoring include:

  • Lightweight change approval processes. “Teams that reported no approval process or used peer review achieved higher software delivery performance…teams that required approval by an external body achieved lower performance.” (Accelerate)
  • Monitoring. Use data from monitoring tools used for applications and infrastructure to inform daily decision-making.
  • Work in Progress (WIP) limits. Keep teams efficient and focused, increasing throughput, by minimizing the different projects in process for a particular team at a given time.
  • Visualizing work. Make use of a Kanban board or similar method to show work moving through development stages.

Cultural

Accelerate authors discuss Westrum’s generative (performance-based) organizational culture as one where “information flows unimpeded and drives better software delivery performance and, ultimately, organizational performance.”

Some of the key management practices around culture include:

  • Supporting learning. Leaders should “set the tone” for a learning culture in the organization and build it into the day-to-day tasks.
  • Collaboration among teams. Encouraging cross-functional teams and collaborative projects and methods will help foster that “all in this together” cultural shift.
  • Job satisfaction. Providing your teams with the right tools and resources to do their work is key to job satisfaction among employees, as well as minimizing the inevitable burnout from complicated deployments and manual tasks.
  • Transformational leadership. Having a vision, communicating in a motivating and inspiring way, enabling intellectual stimulation, embodying supportive leadership, and providing personal recognition are all characteristics of a transformational leader.

Framework #3: CALMS (Culture, Automation, Lean, Measurement, Sharing)

Created by Jez Humble, co-author of The DevOps Handbook and Accelerate, the CALMS framework is used as a means of assessing whether an organization is ready to adopt DevOps processes, or how an organization is progressing in their DevOps transformation. It is based on the following five pillars:

  • Culture. Before silos can be torn down, there needs to be a culture of shared responsibility, or at least a group of people devoted to establishing that culture in a grassroots type of way, with management approval and support.
  • Automation. Similar to the technical practices centered around continuous delivery mentioned above, teams undertaking a DevOps transformation should be devoted to automating as many manual tasks as possible, especially with respect to continuous integration and test automation.
  • Lean. Development teams are making use of lean principles to eliminate waste and optimize the value stream, such as minimizing WIP, making work visible, and reducing hand-off complexity and wait times.
  • Measurement. The organization is devoted to collecting data on their processes, deployments, etc., in order to understand their current capabilities and where improvements could be achieved.
  • Sharing. A culture of openness and sharing within and between teams (and enabled with the proper tools) keeps everyone working toward the same goals and eases friction with hand-offs when issues arise.

So, What Do These Frameworks Have in Common?

As you looked at the different frameworks in this article, I’m sure you noticed several recurring themes and common elements among them. In this section, I want to distill three key practices that help promote a virtuous cycle—positive outcomes that continue to be reinforced and strengthened as they are iterated on—as you mature DevOps in your organization.

These three high-level concepts encompass so many of the principles discussed in more detail above, and when they’re tackled in order, a continuous, virtuous cycle can gradually gain momentum.

They are the 3Cs: - Culture - Collaboration - Continuous Improvement

A Virtuous DevOps Cycle

Culture

Once again, culture is a broad term that can mean different things to different people. But as we discussed in our What is DevOps? article, it is one of the foundational aspects of DevOps that other technical and management practices must be built upon to succeed.

As Accelerate authors discuss, Ron Westrum’s culture typologies for organizations can range from pathological (power-oriented) to bureaucratic (rule-oriented) to generative (performance-oriented). A generative culture is one in which bridging between teams is encouraged, risks are shared, and failure leads to inquiry, rather than finger-pointing. Working toward these cultural paradigm shifts—and giving them time to take hold through practice—is an important first step.

Collaboration

Once the groundwork of a generative culture is laid—one in which everyone feels safe enough to “put themselves out there,” experiment, admit failures, and try again without fear of punishment or shame—greater collaboration can begin to be unlocked within and among teams.

Empowered employees are more open to sharing and receiving feedback, and the more those actions are witnessed, the more others on the team will begin to emulate similar behaviors. Overall performance and shared goals begin to be prioritized over protecting oneself or one’s silo.

Continuous Improvement

Finally, once the teams are collaborating well, with everyone taking personal responsibility for performance, the continuous improvement piece begins to take care of itself. Here is where leadership can continue to reinforce a learning culture as well, one where taking time outside of the usual job responsibilities to work on personal and organizational improvements is key to keeping this virtuous cycle’s momentum going.

Every organization will implement these concepts differently. For example, some may choose Kanban over Scrum as means of collaboration; some may prefer Travis over Jenkins for CI/CD. What’s important to realize is that, regardless of implementation details, decisions should be guided by and also reinforce the concepts of Culture, Collaboration, and Continuous Improvement.

Sources:

Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations by Nicole Forsgren PhD, Jez Humble, and Gene Kim

CALMS definition

The DevOps Handbook: How to Create World-Class Agility, Reliability, & Security in Technology Organizations by Gene Kim, Jez Humble, Patrick Debois, & John Willis

The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win by Gene Kim, Kevin Behr, & George Spafford

The Three Ways: The Principles Underpinning DevOps by Gene Kim

Using CALMS to Assess DevOps Organizations by B. Cameron Gain