Saturday, February 21, 2009

Continuous integration is a powerful iterative process of incremental delivery in which an entire business solution is constructed, validated, and accepted package by package in a series of iterations, where each iteration delivers an operational and functional subset of the total evolving solution. When the last package has been integrated and delivered in this fashion the application is considered complete.

But, why is continuous integration and incremental delivery so powerful?

It turns out that unmanaged risk is the source of all project problems. And, since risk is a measure of uncertainty, our focus must be on reducing this uncertainty.

After reuse, nothing reduces uncertainty as simply and as effectively as customer usage.

Everything else is merely a weak approximation.

As a result, the use of high frequency iterations in the manner illustrated above has the following benefits:
  •  Allows progress to be defined in terms of actual customer capabilities that can be touched and experienced, rather than by often abstract technical or project oriented stages, and thus serves as a practical progress management tool
  • Creates a momentum of success since new tangible value is available (typically) every few weeks
  • Ensures an operational subset of the solution is always available, i.e., the most recent failure free iteration
  • Allows the proposed benefit stream to be realized as early as is feasible
  • Facilitates the transfer of ownership from the developers to the customer—a critical step for success
  • Closes expectation gaps—which may have nothing to do with the stated requirements but only surface when customers actually experience something
  • Identifies defects very early in the project life cycle which substantially reduces project costs while improving customer satisfaction
  • Increases scheduling flexibility by permitting the delivery sequence to be more easily altered as conditions warrant
  • Isolates problems to the current package being integrated, thus dramatically reducing correction and reintegration costs and schedule delays
  • Promotes a high degree of parallelism in scheduling that can dramatically reduce overall project calendar time
But, how do we define these iterations?

This process starts with the packaging plan which captures the size, scope, dependencies, and complexity of the packaging structure and from which the resulting iterations can be dynamically assembled. (For more, see the discussion of Glue and its information model.)

Packages should be small, functionally independent business-centric feature sets. In general, it is important to define packages and the packaging plan so as to maximize:
  • Tangible business value to the enterprise
  • Momentum of success, especially early in the project
  • Early problem detection and especially the validation of architecturally significant requirements and solution components
Finally, it is important to organize and sequence the packages so that they maximize the total return across all benefit streams for the enterprise.

One observation on the size and scope of a package: Size is important, because ideally a project manager will almost always prefer a greater number of smaller packages, rather than only a few large packages. This is for three reasons:
  • Allows packages to move more rapidly through the implementation process (typically, every two to four weeks), which creates a momentum of delivery and success
  • Presents a smaller conceptual “footprint” to the customer so that it’s content is easier to understand, embrace, and accept
  • Increases delivery sequence flexibility by permitting more granularity for rearranging packages to better accommodate changes in project and user priorities, unplanned scheduling contingencies, delays in other packages, etc.
While there are many benefits to this incremental approach to solution delivery, one simple and very powerful benefit is that by chunking the solution delivery one can dramatically reduce sizing (or scoping) risk. This risk is particularly prevalent in situations where there is a high degree of solution complexity, inexperienced teams, unknown or unproven technologies, etc.

The key principle is that by increasing the customer delivery points we increase the effective resolution of the solution delivery process, which quickly and routinely flushes out illogical, missing, or incorrect requirements and expectation gaps, so that rework, delay, and overruns are substantially reduced, while significantly increasing the ultimate quality of the total delivered solution.

Finally, this approach of continuous integration and incremental delivery implies two distinct project management skills:
  1. How to manage each cycle, i.e., the requirements specification, solution construction, and validation of the feature set or functional chunk associated with that cycle
  2. How to manage across cycles, i.e., the synchronization, coordination, and integration of each new chunk into an evolving and conceptually whole solution

Unmanaged risk is the source of all software engineering problems. Risk is a measure of uncertainty in the project’s outcome. The greater this uncertainty, the greater is the risk.

Consequently, an IT organization’s quality efforts must be focused on best practices that reduce this uncertainty.

If we look at history we can make a few obvious observations. First, we can say that this project uncertainty can vary significantly over the life cycle of the effort. In general, it is at its highest when the project begins, since this is where we least understand the project and its dynamics.

As the project advances our degree of uncertainty about the project’s outcomes (e.g., delivery date, cost, quality, adoption rate within the enterprise, ROI, etc.) generally decreases—although in fits and starts and with setbacks and leaps forward—until we reach the end of the project, when our true understanding is at its highest, and our uncertainty is close to zero.

This leads us to instantly draw several conclusions about how to reduce uncertainty and risk. For example, if risk is at its minimum at the end of any project, and if we happen to come across another customer with exactly the same needs as that first customer, then it should follow that the project to deliver the first solution, again, to the new customer can be carried out with very little risk.

Thus, our first axiom: Reuse reduces uncertainty.

Wherever possible, solutions should be delivered by assembling pre-built and pre-validated components, rather than by constructing them. The fastest, cheapest way to build anything is to not build it all. This is the primary learning that arose out of the many quality efforts during the last fifty years in engineering and manufacturing. Reuse is so powerful that it absolutely demands emphasis. Very few actions that an organization can take will have this level of ROI.

Reuse derives its power because after we build that first solution we have two critical pieces of knowledge that we didn’t have before: (1) A customer together with the requirements, needs, constraints, and expectations of a successful solution, and (2) an actual instance of such a successful solution.

So, whereas at the project’s beginning, we were uncertain about these two points, we now, at the end, have much greater clarity, understanding, and certainty.

This leads us to our second axiom: Customer usage reduces uncertainty.

This is the primary learning from reuse. Everything else (better requirements, inspections, reviews, testing, more effective tools, etc.) are all merely weak approximations to simply getting the solution in front of a customer. And, then let them experience it in some organized way.

Remember, the quality principles require that any effective process (in this case, the SE process) must listen to the voice of its customer and then match that voice with the voice of the process (the solution). Further, we know from experience that in SE projects, the voice of the customer cannot be completely captured in the documented requirements, but only fully surfaces and reveals itself when customers actually experience something. These missing requirements are often called expectations, and must be identified, captured, and met for the solution to be considered successful. Moreover, the best way to flush out these missing or misunderstood requirements and expectations is to give customers something to touch and use and interact with.

Consequently, if after exploiting reuse as best we can, we find that we must still build all or portions of the solution, then we must exploit the second axiom, customer usage.

We do this by finding ways to structure and package the total solution into a sequence of component sub-solutions, and then incrementally deliver these sub-solutions to the customer throughout the project’s lifecycle, not just once at the end. Each one of these incremental sub-solutions is simply an opportunity to directly engage the customer with a chunk of the evolving total solution and to get feedback on how well that chunk meets the customer’s needs and expectations. But, because these chunks (customer interaction opportunities) are spread out over the duration of the project, we quickly learn and can reduce uncertainty rapidly.

So, whereas, the first axiom, reuse, is based on delivering the same solution to a different customer, this axiom, is based on delivering different solutions to the same customer. The first reduces uncertainty by leveraging the certainty in the solution, the second by leveraging the certainty in the customer.

An important conclusion to all this is that the higher the delivery frequency (the more chunks the business solution is composed of), the greater is the reduction in uncertainty. However, since each delivery incurs a necessary cost in energy utilization (labor and resources are consumed to deliver each chunk to the customer), the question that matters, is “How many delivery cycles are necessary to reduce risk commensurate with the proposed energy expenditure?” In other words, how do we balance the dramatic reduction in risk associated with each subsequent customer delivered chunk, with the incremental additional expenditure of resources?

This is, in many ways, the key trade-off for IT leaders and their project managers to more fully understand.


Thursday, February 19, 2009

In our practice, it is common that an organization at some point embarks on the visioning journey. Depending upon the company and its history, this journey can take a variety of paths. But in almost all cases, it involves the creation of one or more artifacts: Mission, vision, principles, values, strategy, and the like.

The world has learned several important points about these efforts:
  • It is not about documents, but about the thinking. In particular, the need for pervasive strategic thinking. These documents are of course necessary. It is simply that the artifacts themselves are not the goal, or where the true value lies.
  • All these artifacts should collectively tell essentially the same story, just from a different perspective and with a different focus. They must all reinforce the same central themes and describe the same entity. Otherwise, what gets communicated is confusion and irrelevance, regardless of the actual content. In general, the fewer artifacts the better. The simpler the message the better. The more focused the better.
  • The essential value and power of all these artifacts lies in the degree to which the leadership actually lives and breathes these principles and continually reinforces their essential ideas through frequent and direct interaction with all constituencies—customers, employees, partners, and shareholders. These interactions represent opportunities for the leadership to highlight practical examples where these ideas have worked or where gaps are found. It is through this pervasive personal dialogue (not a speech, or presentation), and only this—the documents (or, posters, web pages, etc.) themselves will always be weak vessels—can the essence of the ideas come alive and mean anything. Consequently, in addition to the creation of any new visioning documents, the organization must include how it should be communicated and used as a tool for increasing dialogue and actively and continually promoting its key messages.
  • Finally, when the leadership thinks about this type of new vision or strategy effort, they need to answer a few questions: What is our goal? Why are we changing what we have now? What exactly does success look like? How will we know whether the new "vision" made any difference?
We use the phrase that "everyone has a piece of the truth".

If the goal is simply a new document (or, paragraph) that does a better job of describing who the organization is now and why they exist and then gets published somewhere, then it is a fairly straightforward PR or marketing piece. In other words, the skills needed are good writing skills to achieve this document re-write goal.

But, if the goal is change, then that needs to start with some alignment on what is not working now, as well as what the future-state should look like. This requires intensive, inclusive (and safe) dialogue among all constituencies, where the leadership can see who they are (their "truth") through their constituents' varied lenses.

For this goal, the writing is the easy part.