An Evaluation of Risk Factors in Large Systems Engineering Projects
System project failures are a well-known part of systems development; however. all the potential risks of planning and executing a project effort are not. This article offers heuristic guidelines to help Information Systems managers assess these inherent risk factors before initiating and while executing a project. Case examples illustrate how a particular risk factor can result in a failed systems development effort. Most Information Systems managers who have been responsible for any type of systems development effort are familiar with project failure. Although publicity is rare, a few project failures have received coverage in the trade press. Some project failures have been labeled runaways, which is “a system that is millions [of dollars] over budget, years behind schedule, and—if ever completed—less effective than promised.
Several years ago, another runaway project disaster received notoriety because the failure affected five million New Jersey state drivers.
The fourth-generation language used to develop the system was not equipped to handle the daily volume of car registrations and license renewals. According to one accounting firm’s study, the problem is widespread: runaway projects accounted for some 35% of the total projects under development in its 600 largest client organizations. Federal government officials have become wary of a similar phenomenon known as grand design approaches to systems development.
A grand design, based on traditional systems development principles, dictates specification of all requirements up front. Problems surface in the federal government bureaucracies because projects are usually large in size and complexity. A large project scope means that costs escalate, time schedules lag, and user, project, and vendor personnel change frequently. Furthermore, Congress balks at the estimated costs, forcing compromise solutions that are prone to implementation failure.
This article attempts to determine the patterns that exist within systems projects that begin well and, for whatever reasons, finish less successfully. To focus the initial work, each project selected used a traditional systems development methodology. Each project failed to meet user expectations. The following observations tie in well with the authors’ experiences with systems development projects. More importantly, these observations may be useful in helping project managers assess the impact of changes to their projects relative to their future success or failure. Furthermore, senior managers and systems sponsors may find help with their decisions regarding the continuation or cancellation of troubled systems projects.
Framework to Identify System Failure Factors
The factors are organized along two dimensions: planning versus executing and technical versus human. The placement of the factors defies pigeonholing; as the framework suggests, few factors are purely classified in one dimension. Broken lines in the exhibit indicate dimension zones. For the purposes of this article, a clockwise approach explains the framework. Beginning with those factors that are most planning related, continuing clockwise through planning-human, executing-human, executing, executing-technical, and planning-technical.
Planning Factors
Although most project failures surface in late execution, the problems often originate during the planning and can occur irrespective of the planning approach being used. Such problems involve mistakes made in time and budget estimates as well as in compression (i.e., project scheduling and management activities) Estimating. Independent of specific tools or techniques, project managers or project sponsors generally use one of five estimating approaches—available budget, level of effort, bottom-up, comparable case, and executive mandate. Note that the approach taken appears to correlate with prospects for project success.
Available Budget
Information Systems managers accustomed to operating their department in a stable environment usually believe that projects will continue within the allotted budget, plus or minus a small percentage. The threat to their projects success is they fail to recognize that taking on a project adds many tasks and virtually none of these tasks can be shortchanged without jeopardizing the entire project.
Level of Effort
The level-of-effort estimate relies on a judgment call that a certain number of employees should be able to complete each task in a certain amount of time. For example, task one should take three people two months to complete. This approach at least recognizes that there is a relationship between the effort and the tasks initiated. However, the approach fails because there is no inherent feedback loop. Because the estimate is judgmental and no two tasks are the same, the next project is always slightly different. Thus, this type of project estimating does not improve with project manager experience.
Bottom-Up Approach
With this approach, planners ascertain individual tasks (e.g., number of pages of procedures to be written, number of employees to be trained, number of screens to be designed). Then, they assign resources and time blocks to accomplish the tasks. Because these estimating parameters are reasonably constant, they provide a common denominator from project to project. Through project time control systems, Information Systems managers know whether or not their estimates are correct. Therefore, the bottom-up method has a self-correcting mechanism as the project proceeds and a learning component to use for each project.
Comparable Case
Very often, especially in long-term planning, estimates must be made for projects prior to the knowledge of detail required for bottom-up estimating. Usually, there are a handful of parameters that determine the size of the project and compare it to similar projects in similar-size organizations in the same industry. For example, although accounts receivable projects may be very different between a construction company and a hospital, they should generally be comparable between hospitals that are approximately the same size. This method is particularly beneficial for project prioritization and rough-cut decisions regarding the necessary worker hours. However, statistics from similar projects may be difficult to obtain. Once the project begins, planners may convert to the bottom-up method.
Executive Mandate
At times, an organization’s senior executives dictate a project completion date on the basis of political, environmental, and market conditions. For example, one organization’s president pushed up a target completion date from three years to one year for a $25 million project involving major technological change and data conversion. Estimates crunched data conversion to two weeks and allowed only one week for a systems test. The reason: that president wanted to implement the system before his retirement.
Studies have shown that in many cases, assigning more staff to a project will not serve to compress the project completion timetable.
As a general rule and regardless of the estimating method used, the more detail included in estimating project time and budget, the more accurate the estimation. In practice, planners do not include enough detail, and a primary source of systems failures is a predilection toward gross underestimation. Gross underestimation does not mean 20% to 30% underestimation, but rather 100% to 1,000% (based on field observations). Furthermore, an acceptable estimation with sufficient detail does not guarantee that the estimate will be allocated the appropriate resources. The following example is a case in point.
Example 1
A client of one of the author’s was, for reasons outside his control, required to replace all of the organization’s information systems. He had a staff of eight and a 1 ‘/2-year timetable to complete the task. His own level of effort estimate suggested that this time was sufficient. An outside group performed a comparable case estimate indicating that the implementation time would require approximately 25,000 workdays or almost 20 times what he had available. Senior management was shocked, agreed in principle to the external estimation, but did nothing other than transfer responsibility for the project to another department.
The project, still staffed at eight but gradually growing, was allowed to drift. Two years later, there were more than 100 employees on the project with a considerable amount of work yet to be done. Had the organization acted on the estimates at the time, the project probably would have essentially been complete at the end of the two-year period and certainly the team members would have felt much better about their involvement.
Compression
This is the act of speeding up tasks. There are two types of compression: planned (i.e., the fast-track approach) and unplanned (i.e., the catch-up approach). Fast-tracked compression management is an art requiring the ability to begin tasks in parallel that are usually conducted sequentially. Managers must rely on their own judgment to predict enough of the outcomes of the tasks usually done first to successfully mesh succeeding tasks and compress the schedule for the final outcome.
Catch-up compression management is probably more common than fast-track management in systems development projects. The problem is that certain tasks do not compress well. Studies have shown that in many cases assigning more staff to a project will not serve to compress the project completion timetable. Instead, more staff usually delays further the completion of the project.
Planning-Human Factors
The planning-human factors include planning aspects that deal predominantly with human communication and scheduling. These factors are bid strategy, staffing, and scheduling. Bid Strategy. Almost all contemporary systems projects use outside vendors either for hardware, software, or other services.
These products and services are often put out to bid. The bidding strategy, whether it be for the entire project or for subcontracting portions of the project, has a major impact on the project’s success. Unfortunately, there is no single best bid-ding methodology. However, most project managers or organizations have a favorite approach. Examples include a fixed-price strategy (i.e., bidding to a set price) and always subcontracting software development to someone other than its designer to prevent conflict of interest. Among the most popular bidding strategies is the lowest bidder strategy.
The inherent risk in selecting the lowest bidder is the magnitude of differences in productivity between programmers and designers, often a factor of 10:1. The impact of this productivity differential is even greater for the individuals who have a major role in the project (e.g., systems architects and project managers). With that great individual variability on the quality and the productivity, selecting a low bid vendor would seem to be an almost certain prescription for failure, and it usually is. Government agencies in particular are forced through regulations to accept the lowest bidder.
As a result, they go to great lengths to disqualify vendors that private businesses would have eliminated on the basis of subjective evaluation. Instead, they redefine their requirements to be strict enough to attempt to eliminate the unqualified on the basis of submitting nonresponsive bids. Staffing. There are two facets to the staffing problem: incorrect staffing and staff turnover.
Incorrect Staffing
The most serious aspect of the incorrect staffing problem stems directly from the estimating problem. That is, inadequate total staff is due to a shortsighted estimate. Other incorrect staffing problems are retaining project members who are not results oriented, or who lack the ability to cooperate as team members, or who do not have the skills called for in the work plan (e.g., failure to include staff with end-user knowledge, or with sufficient technical knowledge of the architecture, or with systems development experience). The following example illustrates project difficulties when there is a lack of end-user knowledge.
Example 2
An agency was processing several million dollars worth of revenue checks per month. The processing procedure was very complex and required some extensive reconciliation. Many of the checks were for large amounts of money. In the current procedure, the check accompanied the reconciling paperwork until the reconciliation was complete. At that point the check was forwarded for deposit. As long as the reconciliation was done on the same day this was not a major problem.
However, the reconciliation process had become very involved, and as a result, checks were being delayed or misplaced. The proposed and designed solution was to create a system that would log the movement of the check so that at any given time anyone could tell the exact location of a check.
One change made to a program module or other design feature often creates a domino effect.
When it was suggested that a more straightforward solution would be to put the chicks directly into the bank and do the reconciliation later, the new system was strongly resisted. A significant problem in this example was that there was no one on the system project team familiar with current practices in revenue and collection systems.
Staff Turnover
This is a two-pronged problem. One prong is ensuring sufficient continuity throughout the life of the project; the other prong is recognizing that there will be major changes in the composition of the team as the project progresses. It is difficult to ensure absolute continuity of staff to a project team because employees are always free to quit, or they may get reassigned. However, it is possible to avoid planned changes in the staff team. The classic waterfall life cycle methodology provides the opportunity, and in some cases the requirement, to reconsider and rebid every project at the end of every systems development phase.
Besides the effect of losing the skills and knowledge of the outgoing members of the team, there is a much more important and subtle factor at work. That is, the new team may feel no compulsion to the design or the solution as proposed in the previous phase. Particularly tenuous are those decisions related to project scope and expense. As a result, there is very often a subtle rescoping of the project every time the members of the project team change.
Scheduling
Still in the realm of planning-human, although now at a more tactical level, are those factors related to scheduling (including the sequencing of activities). The sequence and scheduling of project activities will vary by project type. For example, transaction processing systems usually follow the traditional systems development life cycle (SDLC) approach whereas decision support systems may use applications prototyping. Whatever methodology is used, the sequence and scheduling of project activities should follow a logical, additive progression toward defined goals.
Executing-Human Factors
Project execution, from the human (or personal) side, may be stalled by the lack of feedback, user involvement, and motivation. Each factor inhibits smooth execution of the project.
Feedback
The lack of unbiased feedback surfaces when project managers and systems personnel believe that they can move forward and meet impossible deadlines. Part of the problem is that progress in software or systems development is mostly invisible. Programmers may deceive both themselves and their managers about the extent of their progress. For example, a programmer believed to be 90% complete on a module may actually have 50% of the work yet to do.
User Involvement
This factor includes both the importance of user participation in the design and adequate user training to use the system. The time allotted to procedure development and user training is often too short. User management usually regards systems training as an addition to the regular job requirements rather than giving users sufficient time away from their job responsibilities to learn the system. One of the keys to a system’s project success is to establish ownership among systems users. An effective method of accomplishing user ownership is to let users change requirements during the development process or tailor requirements afterwards. However, this is a two- edged sword if change is not controlled. The importance of user ownership is illustrated in the following example.
Example 3
Because of regulatory changes, a company needed to change both its cost accounting system and allocation methods. Users initiated an internal project that defined some unnecessary complex algorithms and required a major mainframe development effort. A senior vice-president, realizing that the project would not be completed by the established deadline, authorized a second, external team to create a second backup system. Within three weeks, the external team completed the second system using a fully functional prototype at about one-tenth the development costs and one-tenth the operating costs of the internal project.
By this point, the users had such a strong personal involvement with the system they were developing that they rejected this new system despite its advantages. From the external team’s perspective, the project was a failure in spite of the overwhelming cost, schedule, and functional advantages. This failure occurred be- cause the users were not involved in the development of the system.
Motivation
Motivation is a universal personnel problem. With respect to systems personnel, standard time allotments for a given unit of work are not always valid because the motivation of systems personnel varies greatly; that is, the level of motivation of systems staff members will determine how quickly (or slowly) it takes them to complete the work. Two aspects of motivating systems personnel are project-reward structures and the project manager’s success in motivating the team members.
The reward structure can have a significant effect on a project’s outcome. For example, one project team was staffed by employees who performed this systems implementation in addition to their full-time jobs. They were told that when the system converted, they would have the new positions that this new system implied.
As a result, team members worked a lot of overtime to complete the project. Another project effort involved the same arrangement except for the reward structure. Team members had to develop the system in their spare time without the incentive of building future jobs. As a result, the project failed.
Executing Factors
Change management and subsequent workarounds and catch-ups are execution problems stemming directly from the lack of unbiased feedback. When people set impossible deadlines, they act frantically to get the work done rather than admit that they are behind schedule.
Change Management
Once a design specification is finalized, the order goes out to freeze the spec. Although no specifications are ever absolutely frozen, too many changes after finalization may create havoc. One change made to a program module or other design feature often creates a domino effect. Actual changes are often on a larger scale than originally intended.
The extent and impact of subsequent changes to design specifications are products of design quality, project team-user relations, and the project team’s attitude toward change. Analysts who want to please everyone may bring about a project’s demise. Every suggestion brought forward by a user becomes a change. Changes beget other changes until the project dies of accumulated entropy, as in the following example.
Example 4
One project exploded with accumulated changes (see Exhibit 2). Suggested changes (e.g., an inventory aging report) were documented on functional investigation memos (FIMs). A FIM is a one-page narrative describing the change from a user’s perspective. The team analyzed each FIM to determine such affected systems components as data base layout changes or program changes. The change to each affected component was documented on what was called a systems investigation memo (SIM).
Once, authorized, SIMs became technical change requests (TCRs), which consist of module-level detail from which programmers could code. The idea was to accumulate all changes and then open each module only once. This philosophy missed an important point. That is, it is a better strategy to focus on the impact of a single change throughout the entire system than to focus on testing individual modules. In any case, requested changes kept coming in and, as the exhibit suggests, complexity multiplied. After nearly a year of implementing changes, the change backlog was larger than the original change backlog.
After a change in management, the team eliminated all of the unnecessary changes, backed out of some of the coding changes that had already been made, and established a completed version of the system within five months.
Workarounds
Some project managers will continue to find more ingenious ways to live with a problem rather than solve it. For example, one organization implemented an MRP II system (Manufacturing Resource Planning) that was inconsistent with its previous business practices, which were already identified on PC software and procedures. Rather than reconciling the inconsistencies and opting in favor of one of the two systems, employees looked for ways to work around the inconsistencies.
For example, the data element due-date on the informal PC system was the most recent estimate as to when the materials would arrive. In contrast, due-date on the formal MRP II system was based on calculated need dates from the current engineering design. Instead of recognizing the problem that they were trying to pursue two ways to conduct business, the employees attempted to reconcile the two due dates. Eventually, the workarounds became intolerable, and within nine months, the PC system and old procedures were abandoned.
Executing-Technical Factors
Although there are human aspects to vendor, control, and performance factors, the authors view these factors as predominantly technical issues that can compromise project execution.
Vendor
As outsiders, vendors usually get the blame for many project problems, and often, they deserve it. Depending on the particular role they play, vendors may be part of other failure factors discussed in this article. Here, the concentration is on vendor problems in conjunction with the request for proposal (RFP). Ironically, the RFP process may result in the very problems that the process was designed to avoid. That is, the RFP may lead to the implementation of inappropriate products or solutions. Through RFPs, vendors are often requested to respond to user requirements through a checklist evaluation. The evaluation will be weighted by the customer to determine the vendor with the highest score.
This method, however, has several flaws. First, vendors are trusted to make the evaluation for themselves and considerable license is usually used with their answers. Customers often try to address this by making the vendor’s response an addendum to the eventual contract. Experience suggests that this approach is not very effective. If not solved during design or conversion, performance problems may never be resolved.
Second, implicit in the checklist evaluation approach is the assumption that a series of very minute and discrete requirements will actually result in a comprehensive high-quality system. It is often surprising how vendors will continue to add features into their system with very little regard to the impact on the integrated whole, as in the following example.
Example 5
A medium-sized construction company performed a very detailed RFP with checklists.
A relatively unknown software vendor came in first on the functional requirements checklist evaluation even though its solution was written in an exotic operating system and programming language. Management ignored a recommendation to weigh the reliability of a more established vendor’s credibility and in stalled base. Problems occurred immediately.
The package promised a multiplicity of features (e.g., the A/R system supported either an open item or balance forward processing variable by customer). However, these features constantly interfered with one another; the total number of options far exceeded any possibility of testing all combinations and permutations. The implementation team knew that the vendor had performed little integration testing. As a result, they had to do the vendor’s debugging as well as a considerable amount of design. The only way the team could make the system workable was to remove large portions of code to reduce testing requirements to a manageable level.
Control
Controls are a vital part of information systems and in particular those that handle financial transactions. Control code in these systems may be greater than the entire application proper, and adding controls to a system after the fact may cost as much as the original system itself. One organization experienced these costs with a general ledger accounting system, as shown in this example.
Example 6
The literature promoting a system advertised its ability to post transactions into any period: past, present, or future. “What if,” stated the literature, “you find an invoice in the bottom of the drawer that is more than a year old? It does happen.”
However, the controls for this complexity were not built into the, system. Transactions could be posted in previously closed months without a corresponding transaction to change the current month’s balance. Miskeying a date could post it into the distant future, omitted from all financial reports until that date arrived. The solution was straightforward but not easy. The application needed a new control record to indicate which months and years were open for posting. Unfortunately, nearly 100 transactions in nine applications had to be changed to check for an open accounting period prior to creating a transaction, and all programs had to be reviewed to check for the possibility of creating unbalanced general ledger transactions.
Performance
Almost every meaningful systems development project has performance problems. If not solved during design or conversion, performance problems may never be resolved. Some examples include a minicomputer-based system with an interface file with seven records that took more than 12 hours to process, and an online shop floor control system with a 45-minute response time for some transactions.
Planning-Technical Factors
This last section targets two factors that may lead to systems failure. They are experimenting with new technology and technical architecture (i.e., designing the system independent of technical considerations).
Experimenting with New Technology
Experimenting with new technologies is not a problem unless managers take a mainstream business systems development project and jeopardize it by experimenting with unproven technology. The following example illustrates this problem with a data base system.
Example 7
In the midst of the design of the large online materials management system, a hardware vendor suggested to a client that an advanced data base system it was working on would solve the client’s data base tuning and performance problems.
This data base system relied on transferring part of the data base search logic to the disk read heads, which would allow it to search an entire disk for unstructured information very rapidly without draining the CPU resources. One of the authors pointed out that it would be useful for unstructured queries, but the application being designed was for designated transactions that knew which data base records they required. The vendor persisted and sent senior designers of the product to the client to argue the case.
Fortunately for the client, the vendor’s own statistics provided evidence that this product would not help the performance of the application and indeed could hinder it significantly. It seemed as more users got on this system and began cueing up unstructured queries, the system degraded exponentially. Although this particular client was spared the expense and distraction of this technical experimentation, another client (in the same city) purchased the system to use it for transaction processing and accessing unstructured queries. These unstructured queries so degraded the transaction processing that a separate machine had to be set up to provide queries on a non- integrated, standalone basis.
Technical Architecture
Not too long ago, it was popular to design systems independent of their technical architecture. The intention was to prevent knowledge of technical details from biasing the best functional solution. However, this does not work well, as shown in the following example.
Example 8
Analysts familiar only with minicomputer architectures were trying to develop an application for a mainframe environment. In this case, a minicomputer architecture would not work because in a mainframe online architecture, information is keyed into the terminal where it is buffered and stored until the user completes a full screen of information. Then, the user presses a send key, and the entire screen’s worth of information is sent as one block to the mainframe.
This initiates the application program long enough to process the incoming message, perform any necessary data base updates, and format and send another screen of information to the user. The application program then effectively terminates. In a minicomputer online architecture, however, the application program is constantly active when the user is at the workstation. Every time the user interacts with the screen, the application program responds.
In one of its late design reviews of this project, management noted that the user design implied a minicomputer architecture; that is, interaction with the CPU after entry of every field. Not only was this design used to create error messages, but also to actually change the layout of the screen. At this point, the analysts refused to change their design and convinced users to purchase a standalone minicomputer. This meant a need less expense to the users and gave them a nonintegrated, standalone system.
Conclusion
Structural and civil engineering made some of their great strides when they systematically studied failed structures and then incorporated lessons from those investigations into their methods, standards, and approaches. The information systems industry is still in the early stages of a similar evolution.
It is as if it is just beginning to investigate collapsing bridges and broadly categorize the failures (e.g., structural problems, weakness in materials, and unanticipated environmental forces such as flooding). Systems failures are commonplace. Heuristic advice about how to prevent systems failure once a project is underway is less common. The 15 project-risk factors, identified on the systems failure risk framework, and the case examples illustrating how each of these factors can con tribute to project failure, are designed to help Information Systems managers understand and control their own systems development projects.
Using this framework as a guide, Information Systems managers can broaden their perspective on the sources of potential problems, and in so doing, prevent some of the unnecessary project failures they currently face.
David McComb is president of First Principles Inc, a consulting firm that specializes in the application of object-oriented technology to business systems. Previous to founding the company, McComb was a senior manager for Andersen Consulting, where for 12 years he managed major systems development projects for clients on six continents. He received a BS degree and an MBA from Portland State University. Jill Y. Smith is an assistant professor of MIS in the College of Business Administration at the University of Denver. She obtained a PhD in business computer information systems from the University of North Texas. She is also a research associate in UNT’s Information Systems Research Center.
Notes
1. J. Rothfeder, “It’s late, costly, incompetent—but try firing a computer system,” Business Week (November 7, 1988), pp 164—165.
2. D. Kull, “Anatomy of a 4GL Disaster,” Computer Decisions (January 11, 1986), pp 58—65.
3. Rothfeder.
4. An Evaluation of the Grand Design Approach to Developing Computer Based Application Systems, Information Resources Management Service, US General Services Administration (September 1988).
great deal of time in elegant designs and even plans. But until and unless the architecture gets implemented it isn’t anything at all; it’s just a picture. What does get implemented has an architecture. It may not be very good architecture. It may not lend itself to easy modification or upgrade or any of the many virtues that we desire in our “architected” solutions. But it is architecture.
sides of the same coin. But in the final examination, we find that they are intimately intertwined but still separate and potentially independent. The motivation for this paper was an observation that much of our work deals with system planning of some variety. And yet, there is virtually nothing on a web site on this topic.
(I’m indebted to another author for this insight, but unfortunately I cannot credit him or her because I’ve lost the reference and can’t find it either in my pile of papers I call an office, nor on the Internet. So, if anyone is aware whose insight this was, please let me know so I can acknowledge them.)