Complexity in Information Systems
by Dave McComb
- What is complexity?
- How much do I need?
- How much can I stand?
These are the key questions we need to be asking ourselves as we build increasingly complex information systems.
Defining complexity
There are many definitions of complexity. In Roger Sessions’s book, Simple Architectures for Complex Enterprises, he shows how complexity increases with the number, variety and interaction of the various parts of a system. For example, a phone book with a million names and numbers is far less complex than a computer system with a million lines of code. This is because with computer code the behavior of the entire system may be affected by changes to any of the lines of code, whereas changing someone’s phone number does not affect anyone else’s phone number. Furthermore, even though the phone book is basically a paper and ink repository of data, it is far less complex than a digital database containing the same information because the data base organizes the information in different tables and uses different schemas. So the database is made more complex by the variety of ways it can store and provide access to the data.
As complexity goes up, costs go up and flexibility goes down.
And not just a little bit…
Every estimating methodology and every study we’ve looked at suggests that costs go up exponentially as complexity increases. This is why large systems projects are so expensive.
Furthermore, flexibility, e.g., the ability to change the implemented system, goes down just as rapidly as the complexity increases.
And when costs go up and flexibility goes down…
Frustration escalates. Often this is the frustration of the owner or stakeholder of the systems, who are paying more and more for what they perceive to be less and less.
This isn’t good.
So what can we do about this?
First, we can’t become modern day luddites and reject technology and complexity completely. Some of the complexity is essential to solving the complex business problems the systems support.
However, some is the key word to consider. In the majority of the thousands of systems we’ve analyzed, the essential complexity – the variations that are essential to solving the problem at hand – is dwarfed by the accidental complexity – the variation introduced by the solution that is not essential to solving the problem.
This problem is especially acute in federated applications, systems of systems. For example, one system might distinguish employees from contractors, a reasonable distinction in many systems. Another system might distinguish workers from vendors. This would result in four distinctions where most likely only two are needed. Multiply this example by several thousand and the problems caused by accidental complexity become very clear.
The problem of complexity is about to get far worse. Up to recently, IT managers have created these kinds of complexity with systems over which they had control. Increasingly, organizational IT managers are being asked to open their systems up to a broader world, where there are even more distinctions created by anyone accessing the information. If we don’t have a strategy to manage complexity and start implementing it now, we will soon be overcome by a tsunami of complexity.
A strategic process to reduce complexity
A strategy to deal with complexity has four main pillars:
- Containment
- Alignment
- Retirement
- Enhancement
Containment
In much the same way that ‘information hiding’ works when you are programming individual programs, containment provides huge benefits when programming systems of systems. By drastically reducing the number of distinctions any given application foists on the rest of the enterprise, we can reduce the overall complexity of the enterprise systems. This is one of the main advantages of Service Oriented Architecture as long as it is implemented consistently.
Alignment
Alignment means anticipating which attributes and classes have different names for similar or identical concepts. Once we know that ‘employee’ and ‘worker’ are synonymous, we can set up systems to allow us to use the terms interchangeably. This helps us reduce the complexity of the resulting systems.
Even knowing that ‘exempt’ is a special kind of “employee” helps a lot. In many use cases we can ignore the difference and only deal with it when it makes a difference to the use case.
Retirement
As you progress through the containment and alignment disciplines, the systems can be strategically retired become more obvious. We know that eventually all our legacy systems will become obsolete and must be retired. However it is often difficult to figure out the right moment to retire them because they are so internally complex and interconnected that the prospect of replacing them is daunting.
Once you implement an aligned containment strategy, each legacy system or each main portion of a legacy system becomes a replaceable component in a larger architecture. The prospect of migrating your way out of the legacy quagmire becomes feasible, although not necessarily easy.
Enhancement
You don’t have to wait until you’ve mothballed all your old systems to begin the journey to the next generation of systems. There are plenty of opportunities that are being underserved because of the burden of maintaining existing systems. Consider new applications such as integrating structured and unstructured data, mining social media, augmenting systems with data from external sources, ramping up mobile initiatives and participating in the semantic web. Once you get a handle on your internally created complexity, and begin a systematic program to reduce it, you will free up resources.