What is Software Architecture?

Originally published as What is Software Architecture on August 1, 2010

As Howard Roark pointed out in “The Fountainhead” the difference between an artist and an architect, is that an architect needs a client.

Software Architecture is the design of the major components of complex information systems, and how they interact to form a coherent whole. The identification of what constitutes the “major” components is primarily a matter of scale and scope, but generally starts at the enterprise level and works down through subsidiary levels.

The term “architecture” has been overused in the software industry to the extent that it is in danger of becoming meaningless. This is unfortunate, for it comes at a time when companies are in greatest need of some architectural direction. Architecture deals primarily with the specific configuration of technologies and applications in place and those desired to be in place in a particular institution.  While we often speak of a “client/server” architecture  or a “thin client” architecture, what we are really referring to is an architectural style, in much the same way that we would refer to “gothic” as a style of physical architecture, but the architecture itself only exists in particular buildings.

It isn’t architecture until it’s built

As Howard Roark pointed out in “The Fountainhead” the difference between an artist and an architect, is that an architect needs a client. Architecture, in the built world as well as the software world, generally only comes into play when the scale of the endeavor is such that an individual cannot execute it by themselves.

Generally, architecture is needed because of the scale of the problem to be solved. The phrase “it isn’t architecture until it’s built” refers to the difference between architects who draw drawings that may be interesting or attractive, but don’t result in structures being built have only participated in artwork, and not architecture.

Dealing with the “as-is”

Another area of confusion for the subject is the relationship between the “architecture” of a procured item, and the architecture of the environment in which it is implemented. We often speak of software with a “J2EE architecture,” and while it is true the framework has an architecture, the real architecture is the combination of the framework of the procured item with the host of components that exist in the implementation environment.

In the built world we may procure an elevator system, and this may be instrumental in the height and use of the building we design and build, and while the elevator system itself no doubt has architecture, we wouldn’t say that the building has an “elevator architecture.” This confusion of the procured parts with the architecture is what often leads people to short shrift their existing architecture. Sponsors may sense that their current architecture is inadequate and desire to replace it with something new.

However, unless they are in a position to completely eliminate the existing systems, they will be dealing with the architecture of the procured item as well as the incumbent one. All information systems have an architecture.  Many are accidental, but there is an organization of components and some way they operate together to address the goals of the organization.  Much as a remodeler will often employ an architect to document the “as built” and “as maintained” architecture before removing a bearing wall, remodelers of information systems would do well to do the same.

Architecture’s many layers

Architecture occurs in many layers or levels, but each is given by the context of the higher-level architecture. So, we can think of the architecture of the plumbing of a house. It has major components (pipes and traps and vents) and their interrelationship, but the architecture of the plumbing only makes sense in the context of the architecture of the dwelling.

It is the same in information systems. We can consider the error handling architecture, but only in the context of a broader architecture, such as Service Oriented or Client/Server. The real difference between and intentional and an accidental architecture is whether the layering was planned and executed top down, or whether the higher-level architectures just emerged from a bottom up process.

Beyond Building Materials

The software industry seems to equate building materials with architecture. We might talk about an architecture being “C++” or “Oracle” or “Object Oriented” (We’ve heard all these as answers to “what is your architecture?”).

But this confusion between what we build things out of and how we assemble the main pieces would never happen in the built world. No architect would say a building architecture was “brick” or “dry wall” or even “post and lintel,” even though they may use these items or techniques in their architecture.

Conclusion

No doubt there will continue to be confusion about architecture and what it means in the software world, but with a bit of discipline we may be able to revive the term and make it meaningful.

Who Needs Software Architecture?

Originally published as Who Needs Software Architecture on August 5, 2010

Most firms don’t need a new architecture.

They have an architecture and it works fine. If you live in a Tudor house, most of the time you don’t think about architecture, only when you want to make some pretty major changes. You might expect, given that we do software architecture and this is on our web site, that we would eventually try to construct this theme to say, well, nearly everyone sooner or later needs a software architect.

But that’s just not true. Most companies don’t need software architects and even those that do don’t need them most of the time. Let’s take a look at some of the situations where companies don’t need software architects.

Size

Small companies generally don’t need software architects. By small we mean companies of typically fewer than 100 people, however, this can vary quite a bit depending on the complexity of the information they need to process. If they are in any standard industry and if there exists packaged software which addresses their business needs, most small-business people would be far better off to adopt that package or the package of their choice and simply live with the architecture that comes with it.

For instance, in the restaurant industry now, there is a company called Squirrel that has by far the largest market share of the restaurant management applications. You can take orders on Squirrel, print out the receipts, take the credit cards, manage your inventory, schedule your wait people, cooks, busboys, and the like. For the most part, restaurant owners should not care what architecture Squirrel uses. It has an architecture but it’s not an important concern at that scale.

Stability

Larger companies most of the time will find themselves in a relatively stable state. They have a set of applications sitting on a set of platforms using a set of database management systems communicating via some networking protocol and communicating with some set of desktop or portable devices.

No matter how various it is, that is their current architecture and to the extent that it is stable and they are able to maintain it, make extensions to it and satisfy their needs, that is exactly what they should do and they should live within the architecture they’ve created, no matter how accidental the architectural creation process was.

It is really only where there are relatively complex systems, where the complexity is interfering with productivity or the ability to change and respond, or where major changes to the infrastructure are being contemplated, that companies should really consider undertaking architectural projects.

What Does a Software Architect Do?

Originally published as What Does a Software Architect Do? on August 11, 2010

The Software Architect’s primary job is to help a client understand deeply the architecture they have built, understand clearly what they desire from their information systems, and help them construct a plan to get there.

The simple answer of course is that the software architect or architectural firm creates the architecture.

The more involved question is, what goes into that and what process is typically followed to get to that result? The architecture, or as we sometimes refer to it, the “to-be” or target architecture, is an architecture that does not yet exist; and in that sense it is prescriptive. However, in order to define, articulate, draw, and envision a future architecture, we must start from where the client’s architecture currently is and work forward from there.

Divining the “As-is” Architecture

The client’s current architecture is a combination of a descriptive and visual representation of all the key components in the information infrastructure that currently exist. We have found time and time again that the mere inventorying, ordering, arranging, and presenting of this information has yielded tremendous benefit and insights to many clients.

Typically, the process involves reviewing whatever written documentation is available for the existing systems. Sometimes this is catalogued information such as a listing or repository of existing applications/technologies. Sometimes it’s research into the licensing of pieces of software. Sometimes it’s a review of diagrams: network diagrams, hardware schematics, etc.

The architect then interviews many of the key personnel: primarily technical personnel but also knowledgeable end-user personnel who interact with the systems and in many cases understand where other shadow IS systems live.

The end product of these interviews is a set of diagrams and summary documentation that show not only the major components but how they are interrelated. For instance, in some cases we have found it important to document the technical dependency relationships, which include the relationship of an application to the technologies in which it was created and therefore on which it is dependent. (See our article on technical dependencies for more detail in this area.)

Listening to the Stakeholders

The second set of inputs will come primarily from the business or user side of the organization. This will include interviews to establish not only what exists, and especially what exists that doesn’t work well, but also what is envisioned; what it is that the organization wishes to live into and is potentially hampered by in their existing information systems.

The real art comes in how we get from where we are now to where we want to be.

This is a very intense active listening activity in that we do not expect end users to be able to articulate architectural themes, needs, requirements, or anything of that nature. However, they do have the key raw material that is needed to construct the target architecture, which can be drawn out in conversation. The end product of this activity combined with what’s known from the current architecture is the first draft of what is called the target architecture or the to-be architecture. At this point the major themes, or styles if you will, are described and decided upon. It’s very much as if at this point the client is choosing between new urbanism or neo-modern styles of architecture.

Again, unless you know going in the style of architecture that will be required, it is best to work with architects who have a range of styles and capabilities. As the architects conceive of the overall architecture, they shift into a collaborative and consensus building mode with senior management and any other stakeholders that are essentially the owners of the long-term architecture. This process is not merely laying out the blueprints and describing them but is a longer ongoing process of describing themes, trade-offs, economics, manageability, and the like; trying out ideas and gathering feedback from the team. Again, active listening is employed to ensure that all concerns are heard and that, in the end, all participants are in agreement as to the overall direction.

Migration Plan

The real art comes in how we get from where we are now to where we want to be. First, the architects and management team need to discuss urgency, overall levels of effort, and related questions. Getting from an existing architecture to a future architecture very often resembles construction on a major arterial highway.

We all know it would be simpler and far more economical to shut down the highway for six months or a year and do all the improvements. But the fact is that most urban arterial roads are in very heavy use and shutting them down for efficient construction is not feasible, so the actual road construction project becomes a very clever series of detours, lane expansion, and, unfortunately, very often reworking the same piece of pavement multiple times. And so it is in the software industry.

Ten years ago, it was fashionable to “bulldoze the slums,” in other words, to launch massive projects that would essentially start from a green field and build all new systems that the owners could move into. There have been multiple problems with this approach over the years, the first being that the sheer size of these projects has had a dramatically negative impact on their success.

The second problem is that we are all, if you will, living in those slums; we are running our businesses with the existing systems and it is very often not feasible to tear them down in a wholesale fashion. So, one of the duties of the architects is to construct a series of incremental projects, each of which will move the architecture forward. At the same time, many, if not all, should be designed to provide some business benefit for the project itself.

This is easier said than done, but very often there is a backlog of projects that needs to be completed. These projects have ROI (return-on-investment) that has been documented and it is a matter of, perhaps, re-scoping, retargeting, or rearranging the project in a way that not only achieves its independent business function and return-on-investment but also advances the architecture.

Balancing Short Term and Long-Term Goals

This is an area that in the past has been sorely neglected. Each project has come along focused very narrowly on its short-term payoff. The net result has been a large series of projects that not only neglects the overall architecture but also continues to make it worse and worse, such that each subsequent project faces higher and higher hurdles of development productivity that it must overcome in order to achieve its payback.

When an overall plan and sequencing of projects has been agreed upon, which by the way often takes quite a significant amount of time, the plan is ready to be converted into what is more normally thought of as a long-range information system plan, where we begin to put high-level estimates on projects, define some of the resources, and the like.

That, in a nutshell, is what the software architect does. At the completion of this process, the client knows with a great deal of certainty where he’s headed to architecturally, why his destination architecture is superior to the architecture he currently has, and the benefits that will accrue once he is in that architecture.

And finally, he has a road map and timeline for getting from his current state to the desired state.

How to Select a Software Architect

Originally published as How to Select a Software Architect on August 31, 2010

Selecting a Software Architect is an important decision, as the resulting architecture will impact your information systems for a long time.

We present a few thoughts for you to keep in mind as you consider your decision. Assuming you have come to the conclusion that you can use the services of a software architect, the next question becomes, how do you select one? We’re going to suggest three major areas as the focus of your attention:

• Experience
• Prejudice
• Chemistry

Experience

By experience we are not referring to the number of years of specific experience with a given technology. For instance, assuming that you did “know” somehow that your new architecture was going to be Java J2EE-based (though by the way, a decision like that would normally be part of the architectural planning process and it would often be detrimental to “know” this information going in). Even if you did know this information, it would not necessarily be beneficial to base your selection of an architect on it.

This would be akin to selecting your building architect based on the number of years of dry walling experience or landscaping experience that they had had. At the same time, you certainly do not want inexperienced architects. The architectural decisions are going to have wide-ranging implication for your systems for years to come, and you want to look for professionals that have a great depth of knowledge, and breadth of experience, of different companies and even of different industries that they can draw upon to form the conclusions that will be the basis for your architecture.

Prejudice

By prejudice we mean literally prejudgment. You would like to find an architect as free as possible from pre-determined opinions about the direction and composition of your architecture. There are many ways that prejudice creeps into architecture, some subtle and some not so subtle. For starters, hardware vendors and major software platform vendors have architects on staff who would be glad to help you with your architectural decisions. Keep in mind that most of them either overtly or perhaps more subtly are prejudiced to create a target architecture that prominently features their products, whether or not that is the best solution for your needs.

Other more subtle forms of prejudice come from firms with considerable depth of experience in a particular architecture. You may find firms with a great deal of experience with Enterprise JavaBeans or Microsoft Foundation Classes, and in each case, it would be quite unusual to find them designing an architecture that excluded the very things that they are familiar with. The final source of prejudice is with firms who use architecture as a loss leader to define and then subsequently bid on development projects. You do not really want your architecture defined by a firm whose primary motive is to use the architecture to define a series of future development projects.

Chemistry

The last criterion, chemistry, is perhaps the most elusive. We’re considering chemistry here because a great deal of what the architect must do to be successful is to elicit from the client, the client’s employees, potentially their customers and suppliers, and from existing systems their needs, aspirations, and constraints, and to hear that in full fidelity. For this to work well there must be a melding of the cultures or at least an ability to communicate forthrightly and frankly about these issues and really the only way to make this sort of determination is through interview and reference. The selection of the software architect is an important decision for most companies, as the creation of the architecture is likely to be the single most important decision that will affect future productivity as well as the ability to add and change functionality within a system.

We do enterprise architectures, service-oriented architectures, and semantics. I suppose it was just a matter of time until we put them together. This essay is a first look at what a semantic enterprise architecture might look like.

What problem are we trying to solve?

There are several problems we would like to address with semantic architecture. Some of them are existing problems that are just not solved well enough yet. Even though many organizations intellectually understand what a service-oriented architecture is and what it might do for them, the vast majority remain unmotivated to invest in the journey to migrate in that direction.

Certainly we want the semantic enterprise architecture to address the same sorts of situations that the service-oriented architecture can handle, such as the ability to rapidly adapt new applications; to swap out technologies and applications at will; and to do so with commodity priced services and technologies. But let’s also focus on what additional problems a semantic enterprise architecture could address.

Dark matter and dark energy

As strange as it may seem there is mounting evidence that the knowable and visible universe, that is, the earth, the sun, the planets, the stars, the galaxies, and all the interplanetary and intergalactic dust represent some four to five percent of  the total amount of “stuff” that there is in the universe. We live in the froth of a giant wave whose presence we can only infer.

What makes up the rest of the universe is what physicists now call dark matter and dark energy. Dark matter makes up some 25% of the mass and energy deficit of the universe and is the primary force holding galaxies together, as the gravitational attraction of the masses of many stars and black holes is insufficient to do the job on its own.

Rather than the universe’s expansion slowing down since the big bang as the amount of mass and dark matter would suggest, apparently the universe is moving apart at an even faster rate. The propelling force has been dubbed “dark energy” and it comprises the remaining 70% of all the “stuff” in the universe. And so it is with our corporate information systems. We fret continuously over our SAP databases or the corporate data warehouse or the integrated database system that we run our company with, but this is very much like the five percent of the universe that we can perceive. It’s comfortable to believe that’s all there is.

But it just doesn’t square with the facts. Rogue applications, such as those built in Microsoft Access, Excel or FileMaker are the Dark Matter of an information system.  Like the cosmic Dark Matter, in some fashion they are holding an enterprise together, even though most of the time we can’t see them. Messages and documents are our Dark Energy equivalent: they are the expansive force in the enterprise. And like Dark Energy in the Universe, they are undetected by the casual observer.

What Does Enterprise Architecture Have to do with Semantics?

We ignore our information dark energy and information dark matter largely because at a corporate level we literally do not understand them. In our corporate databases we’ve invested decades of effort and typically millions and usually tens and hundreds of millions of dollars of implementation, standardization, training, and documentation in an attempt to arrive at a shared meaning for the corporate information systems. As we’ll discuss in other white papers, this has still left a great deal of room for improvement. Indeed, most of the meaning is shared only within an individual application.

Occasionally, corporations invest mightily in ad hoc semantic sharing between applications, under the guise of Systems Integration. But what we’re going to talk about today is the information dark matter and information dark energy and bringing them into the light. With the rogue systems, what we need to know is: what is the correspondence, if any, (and, by the way, it is usually considerable) between the rogue systems and the approved systems. More often than not, a rogue system is populated from either extracts or manual creation of data from approved systems. The rogue system is often created in order to make some additional distinctions or provide additional behavior or extensions that were not possible in the approved system. But this does not mean they don’t have a shared root and some shared meaning.

Occasionally, rogue systems are created to deal with some aspect of the corporation that, at least initially, appeared to be completely outside the scope of any existing application. You may have a videotape renting application or a phone number change request application or any of a number of small special-purpose systems. However, if they become successful and if they grow, inevitably they begin to touch aspects of the company that are covered by officially sanctioned systems.

What we want to do with semantics is to find and define where the commonalities lie in such a way that we may be able to take advantage of them in the future. For the unstructured data we have an even bigger challenge. With the rogue system, once we deduce what a column in the Access database means, we have a reasonable prediction of what we are going to find in each change record. This is because the Access database, while it may not be as rigorous as the corporate database, provides structure and validation.

Not so with the unstructured data. With the unstructured data, we need to find ways to find and organize meaning where every instance may be different. Every email, every memo, every reference document contains different information. The semantic challenge is to find, wherever possible, references in this unstructured data to information that is known at a corporate level. The approach here is almost exactly the opposite: in documents, people rarely refer to what we think of as meta-data or categories or classes or columns or entities — or anything like that. In documents, people refer to specific instances. They may refer to their order number in an email; they may refer to a set of specific codes in a reference manual; they may refer to a particular procedure in procedure manual. Our semantic challenge in this case is to find these items, index them, and associate them to the meta-data and even the instances that exist at a corporate level.

Semantic Enterprise Architecture

So what’s in the architecture? The Sematic Enterprise Architecture is still primarily based on Service Oriented Architecture concepts. We want to be able to communicate between largely independent applications and services using well-defined and corporate standard messages. These messages should be produced and consumed in such a way that allows at least some local change in their structure and syntax without breaking the rest of the architecture.

But we need to go considerably beyond that. We will need a meta-data repository that links the enterprise’s shared schema with a more generalized and, at the same time, more precise description of what these things mean. This meta-data repository will be populated by a combination of machine and human inferences from the description of the meta-data that exists in the many dictionaries and documentation bits as well as from the product of data profiling. Data profiling in the corporate systems will tell us not what we intended the data in our corporate systems to mean but in practice, from how we have been using the system, what it has come to mean.

This expression of the enterprise meta-data in a rigorous format is just the beginning or the gateway for incorporating our dark energy and dark matter. The rogue systems need to have their meta-data catalogued in a compatible fashion to the enterprise meta-data repository. This will allow us at least to know when the corporate systems may want to refer to the rogue system for additional details. Conversely, it creates at least some hope that the rogue system may have a defined interface to the corporate system and may be informed if things change.

The unstructured data will be incorporated using technologies that already exist, including text interpretation, first to find any specific nouns or events that are called out from the unstructured data. Using this information, the unstructured data can be cross-referenced to instances and by extension into the entire enterprise data network.

How to Get Started

This all sounds a bit incredible. And the endgame is likely a ways off. But we don’t have to go to the endgame for this to be useful. As Jim Hendler says, “A little bit of semantics goes a long way.” Even if we only pick a few topics to index in our meta-data repository, and even if we choose a few well-known rogue applications to cross-reference, and even if we only grab the low hanging fruit from our unstructured data, as many companies are already doing, we will still see considerable benefit.

Many content management and some knowledge management projects are aimed at using humans to perform this style of indexing on the reference documents that we often use in our organizations. But with a little extension this can go considerably farther. As it is, it’s generally an island unto itself. But as we’re proposing here the island can be extended and incorporated into the broader enterprise landscape.

Concluding Thoughts

We’ve barely scratched the surface here. However, many of the technologies needed to make this work already exist and have been proven in isolated settings. What are needed are companies willing to invest in the research and infrastructure in order to profitably include the other 95% of their information infrastructure into their enterprise architecture.