Excerpted from Systems Architecting, Creating and Building Complex Systems, Eberhardt Rechtin, Prentice Hall, 1991.

MANAGERIAL RESPONSE IV: INDEPENDENT REVIEWS

Without question, there are no people more interested in system success and quality than the system's architects, engineers, builders, and sponsoring clients.

If there are difficulties with the system, they want to find and correct them first, before anyone else. Their commitment to the system is, and must be, complete. As a matter of personal and group pride, they want the system to work magnificently, as error- and failure-free as it is possible to make it. Their internal progress reviews are as thorough and self-critical as they can make them/

They are not, however, the only people deeply concerned with the and the process by which it is built. Two other groups, the end users and upper management, have a considerable interest in how things are going. These groups feel the need for independent reviews, not to critique the work of the people, but for needs of their own gained from experience over the years.

End users found that systems were being presented to them, often after a decade of development and production, that did not fit their present needs. Much of the problem was the long development time. Conditions, needs and people change. Part of the problem was lack of user contacts during conceptual design—or contacts with wrong or nonusers. Whatever the problem, organizationally independent operational test and evaluation (OT&E) was instituted toward the end of development, prior to full-scale production, to make sure than the end product would be useful and cost effective. Although the most formal version of OT&E is governmental, private industry does something similar during its product development—testing the market from time to time using representative potential customers.

Management found itself making critical resource-allocation decisions for which all sources of information were inherently biased. At the same time, management wanted to be sure that enough resources were made available at the right time and place to keep things on track. To do that required knowing—and judging for itself—the levels of risk being incurred in the project.

Independent reviews by independent expert teams, by surfacing the major problem areas and prior decisions (most of them resolved and validated, respectively), give top management a feeling for the risk level of the decisions they are called upon to make. They are a safeguard against the normal human tendency to solve all problems at as low a level as possible without "bothering" the higher levels—why report "solved problems?’’ But many problems are not solved; they are resolved at some risk level, for example, "we decided the risks were low and the costs were too high, so we decided not to take corrective action on this one." Fair enough, but how many such "low-risk problems were there? And does the problem appear as "low risk" to others? Top management, by hearing a review of what has gone on before, from an independent group's perspective, is given a good opportunity to evaluate total total risk, even if all prior actions are affirmed.

Putting it mildly, when independent reviews were initiated, they were barely tolerated by the systems people as a necessary evil imposed from above. They did take valuable time of key people. But with well-understood procedures, participants, and purposes, these reviews have unquestionably helped maintain discipline, encouraged good documentation, ensured accountability for decisions, and reduced operational failures. They have become an essential part of ultraquality processes.

Part of the benefit is direct. New ideas for improvement are indeed brought forward. Most of the benefit is indirect. The process of getting ready for an independent review, in the minds of many systems people, has been much more valuable than the review itself. Decisions are rethought. Problems are discovered that are easily and quickly fixed. Internal criticism is looked on favorably - what if the same criticism first surfaced in the independent review?!

At the same time, the user and management groups gain an appreciation of what are the problems and risks, what might be done to fix them, and how to deal with them if they do occur in operation. Independent reviews aid the system acceptance process yet to come, relieving to some degree the client/user's trauma in accepting uncertifiable ultraquality systems. They keep raising the explicit question: Do we, collectively, accept the risks - small, we hope - of the system not meeting its ultraquality objectives?

The question focuses on the consequences of system performance - good and bad - instead of on the details of system implementation. In so focusing, client/users look for ways to minimize their maximum risks, technical, financial and social. Objectives can be changed. The definition of success can be refined to make it better fit reality. The public can be better informed. Additional resources can be provided. Mitigation measures can be instituted to minimize potential damage. Backups can be acquired. If worst comes to worst, the system can be terminated or scaled back pending further technological developments.

In management terms, independent reviews allow the client/users to make provisional acceptance decisions well before the formal acceptance procedure. Actions can be taken that will yield a better system, better understood, at less cost and risk.