Occasionally I invite outstanding thought leaders to write a commentary on dominant IT business issues. Gene Kim, CTO of Tripwire and Julia Allen, Senior Member of the Software Engineering Institute at Carnegie Mellon University, discuss their research on high performing IT operations and explore three critical questions in the context of solving the most common IT challenges.
IT is being challenged on many fronts, from cost containment, business alignment, compliance, competitive pressures in managing outsourced IT services, and security. Many experienced IT practitioners confronted by this potentially staggering array of challenges will point out that the solution to virtually all these issues is more repeatable IT processes and effective controls. However, merely understanding this does not necessarily equate to an effective plan to solve the problems, and may create more questions than answers. To simplify the problem, Dr. Eliyahu Goldratt, creator of the "Theory of Constraints", articulates three simple questions that must have credible answers: what do I need to change, what should I change to, and how do I cause the change?
Finding answers to those three questions has been an area of passion for Gene since 1999. He has been researching high-performing IT operations and security organizations, attempting to understand what makes them so different than typical IT organizations, as well as studying how organizations have accomplished the transformations that take them from being merely average to best in class. Along this journey, Gene started working with other organizations that are also interested in these issues, such as SANS, the IT Process Institute (ITPI), the Institute of Internal Auditors (IIA), and most recently, the Software Engineering Institute (SEI) with Julia Allen. In particular, the collaboration between the ITPI and SEI has yielded some extremely promising results, both in characterizing high-and low-performing IT organizations, the key differences in their belief systems, and the necessary components to achieve an organizational transformation from low- to high-performer.
In this article, we will discuss two areas of research that we believe are foundational for answering the question of what IT organizations typically need to change and what they need to change it to. We will present a working definition of what characterizes a high-performing IT organization, and then discuss the key differences in the belief systems between them and more typical IT organizations in three areas of pain: patch management, proliferation of IT management scorecards, and managing outsourced IT services.
Lastly, to help answer the question of how to cause the change, we will describe the publicly available Visible Ops methodology, which captures how IT organizations have transformed into high-performers in a way that is can be accomplished in four steps, each which is a finite project and returns more value back than was invested. We will also describe the ITPI Community of Practice Listserv, and the upcoming VEESC benchmarking study. We conclude the article with a call to action and an active solicitation for feedback in participation in creating this community of practice for high-performing IT organizations.
Key Characteristics of High-Performing IT Organizations
Since 1999, after studying the IT processes of hundreds of organizations, it started becoming clear to Gene that a handful of them stood out as somehow different from the others in some notable way. He started keeping a list of these organizations, at that time informally called “Gene’s list of people with amazing kung fu.” In 2000, Gene started working with Kevin Behr, CTO of one of these unusual organizations, and they started a more systematic analysis of what was common to these organizations, called the “best in class IT operations and security organizations.” In 2003, Julia Allen from the SEI actively joined this effort, which resulted in a remarkable event in October 2003 at Carnegie Mellon University called the Best In Class Security and Operations Roundtable (BIC-SORT).
Among the stated goals were to “begin to build an executive-level community of practice for IT (information technology) operations and security, with a common sense of purpose and a desire to influence other relevant and connected communities of practice; and to better capture and articulate the relevant bodies of knowledge that enable and accelerate IT operational and security process improvement.” Since then, we have been actively processing and synthesizing the data we collected.
Based on our analysis, we have created the following working definition of high-performing IT organizations: They are effective and efficient and they succeed in applying resources to accomplish their stated business objectives with little to no wasted effort. These organizations have evolved a system of process improvement as a natural consequence of their business demands. They regularly implement formal, repeatable and secure operational processes.
Results of informal benchmarking indicate that in these best-in-class IT organizations, IT operations and security work together to create higher service levels (e.g., as measured by mean time to repair, mean time between failure); higher percentage of planned, scheduled work relative to unplanned work; unusually efficient cost structures (e.g., as measured by server to system administrator ratios); productive working relationships with management and peers; and smoother audits. Furthermore, they have more timely identification and resolution of security incidents, the earliest integration of information security requirements in the service delivery lifecycle, and the ability to quickly return to a reliable and trusted operational state. And perhaps most admirably, these organizations devote increasingly more time and resources to strategic issues, having mastered tactical concerns.
The high-performing organizations desire to detect production variances early so they can fix problems in a planned manner and where the repair costs are lowest and have the least impact. They value repeatable and verifiable processes and use controls to improve efficiency and effectiveness. And because these organizations use controls to improve their own operation, life is much easier for auditors who evaluate operational risk based on the presence of effective and verifiable preventive, detective, and corrective controls. In other words, the controls aren’t there just because auditors asked for them, but because they are used to improve daily operations! As a result, high-performing organizations require considerably less effort to meet management and audit expectations.
To achieve these characteristics, several key performance metrics are essential to this level of performance: they have the highest change success rate (typically over 98%), highest effective rate of change (sometimes making over 1000+ successful changes per week), highest level of mastery of production infrastructure (achieved by low configuration counts and low configuration variance), and highest ratio of staff dedicated to pre-production activities (achieved by release management processes, pre-production testing, etc.).
Surprisingly, we found that all of the high-performing IT organizations had independently developed virtually the exact same processes to achieve these results. They shared similarities in three key process areas, which we will describe in the parlance of ITIL (IT Infrastructure Library): they had a “culture of causality” that ensured all production problems ruled out change as early as possible in the repair cycle (resolution processes), they had a “culture of change management” embedded in the way all work is done (control processes), and they moved as many production changes through a pre-production process that orchestrated changes with the production environment (release processes).
The high-performing organizations all implemented virtually the same procedures in these three ITIL process areas, which form the minimal closed-loop that generates metrics that allow continual process improvement. These procedures and processes are described in the Visible Ops methodology in detail, published by the ITPI.
Belief System Differences Between High- And Low- Performing IT Organizations
Given the fact that high-performing IT organizations exist, what prevents low-performing organizations from becoming high-performers, given the promise of a better way? Understanding why this was so became one of the main areas of activity after the BIC-SORT event. Julia Allen, Kevin Behr, and Gene Kim from the ITPI and SEI have been synthesizing the captured list of key areas of pain and promise from the participating organizations during BIC-SORT. Our goal was to create a taxonomy of pains, find any cause-effect relationships and root causes and understand what belief systems that preserved the status quo for the low-performers.
In the BIC-SORT, we captured almost one hundred specific areas of pain, such as the challenges of keeping up with security patches, the massive efforts required to do effective audits of business peers, and so forth. Of these, we chose to analyze three of the most acute of the listed pains: keeping up with patching, dealing with the proliferation of management scorecards, and management of outsourced IT services.
