Most companies of any size have created an internal Tower of Babel that frustrates all attempts to integrate their disparate systems.
White Paper: Categories and Classes
Getting the categories and classes distinction right is one of the key drivers of the cost of traditional systems.
We’ve been working with two clients lately, both of whom are using an ontology as a basis for their SOA messages as well as the design of their future systems. As we’ve been building an ontology for this purpose we became aware of a distinction that we think is quite important, we wanted to formalize it and share it here. In an ontology there is no real distinction that I know of between a class and a category. That is: classes are used for categorizing, and you categorize things into classes. If you wanted to make a distinction, it might be that category is used more in the verb form as something you do, and the class is the noun form.
Categories and Classes in Traditional Apps
But back in the world of traditional applications there is a quite significant difference (although again I don’t believe this difference has ever been elaborated). In a traditional (relational or object oriented) application if you just wanted to categorize something, (say by gender: male and female) you would create a gender attribute and depending on how much control you wanted to put on its content you
would either create an enum, a lookup table or just allow anything. On the other hand if you wanted behavioral or structural differences between the categories (let’s say you wanted to distinguish sub
contractors from employees) you would set up separate classes or tables for them, potentially with different attributes and relationships. We’ve been studying lately what drives the cost of traditional systems,
and getting this category/class distinction right is one of the key drivers. Here’s why: in a traditional system, every time you add a new class you have increased the cost and complexity of the system. If you
reverse engineer the function point methodology, you’ll see that the introduction of a new “entity” (class) is the single biggest cost driver for an estimate. So every distinction that might have been a class, that
gets converted to a category, provides a big economic payoff. It’s possible to overdo this. If you make something a category that should have been a class, you end up pushing behavior into the application code, which generally is even less tractable than the schema. So we were interested in coming up with some guidelines for when to make a distinction a category and when to make it a class.
Categories and Classes in gist
As it turns out, we had foreshadowed this distinction, although not for this reason, in gist, our upper ontology. Gist has a class called “category” whose intent is to carry categorical distinctions (from one lower level ontology to another) without necessarily carrying their definitions. For instance when we worked with a State Department of Transportation, we had a class in their enterprise ontology called “Roadside Feature.” A Roadside Feature has properties such as location and when by what process it was recorded. Several of their applications had specific roadside features, for instance “fire hydrants.” In the application fire hydrant is a class, and therefore is one in the application ontology. But in the enterprise ontology “fire hydrant” is an instance of the category class. Instances of fire hydrant are members of the roadside feature class at the enterprise ontology level, and associated with the category “fire hydrant” via a property “categorizedBy.” A fire hydrant can therefore be created in an application and communicated to another application that doesn’t know the definition of fire hydrant, with almost no loss of information. The only thing that is lost on the receiving end is the definition of fire hydrant, not any of the properties that had been acquired by this fire hydrant.
Building Ontologies Visually Using OWL
Faced with the challenges of UML and other modeling notations, we developed our own Visio-based ontology authoring tool. We’ve been building large enterprise ontologies for our clients using the W3C web ontology language OWL. If you’re not familiar with OWL, think of it as a data modeling language on
steroids. It also has the fascinating property of being machine interpretable.
You can read the model with the aid of an inference engine, which not only tells you if all the assertions you have made are consistent, but also can infer additional parts of the model which logically follow from those assertions. So far the available tools for developing OWL ontologies, like Top Braid Composer and Protégé, look and feel like programming development environments. Some visualization tools are available but there is no graphical authoring tool, like Erwin or ER Studio, that data modelers have become used to.
The larger your ontology gets, the harder it is to understand and navigate using the current tools, and enterprise models push the boundaries of size and complexity for OWL ontologies. Another problem is that, because of OWL’s expressiveness, the UML diagramming conventions can result in a very complex-looking diagram even for simple ontologies.
This makes it hard to review models with subject matter experts (SMEs), who typically have about 15 minutes’ worth of tolerance for training in complex modeling notations. Faced with these problems, we developed our own Visio-based ontology authoring tool. It uses a more compact, more intuitive diagramming convention that is easier for SMEs to grasp.
From the diagram you can generate compliant OWL in XML format that can be read by any of the standard editors or processed by an OWL inference engine. The tool, which we call e6tOWL, is built as an add-in to Visio 2007 which provides the diagramming platform. In addition to the Visio template containing the diagramming shapes and the OWL generation function, the tool provides a lot of layout and management functionality to help deal with the challenges of maintaining large and complex diagrams.
So far the results have been good. We find it faster, easier and more fun to create ontologies graphically, and SMEs seem able to understand the diagramming conventions and provide meaningful feedback on the models. We are still using Composer and Protégé for running inference and debugging, but all our authoring is now done in Visio. We currently maintain the tool for our own use and provide it to our clients for free to help with the ongoing development and maintenance of ontologies.
The Return on Investment (ROI) for Architecture
In many organizations, ROI is a euphemism for quick fix or short term payout projects.
I’ve come across a couple of articles recently on either the difficulty or the impossibility of constructing a Return on Investment (ROI) analysis on enterprise architecture projects, which, of course,
we would take to also include service oriented architecture projects. While I agree with many of the authors’ points and frustrations, particularly in organizations with a policy of ROI-based project approval, I don’t completely agree with the assessment that ROI is not possible. The difficulties in constructing an ROI for architectural projects lie in two major areas: timeframe and metrics.
Timeframe Issues
In many organizations, ROI is a euphemism for quick fix or short term payout projects. It’s not so much that an architectural project does not have a return on investment but typically the investment is considerably longer than befits a quick fix project. Organizations that overemphasize ROI are typically addicted to short term solutions. While a great deal of good may be done under a shorter time horizon, there is also often a dark side to them. That dark side is that the quick fix is often implemented at the expense of the architecture. What remains after a long period of time is often an architecture of many arbitrary pieces. As a result, each succeeding project becomes harder to accomplish and maintenance costs go up and up. However, these weren’t factors in the ROI for the project and as a result the project approval scooted on through. Indeed, if there were no return on investment for an architectural project one would argue that we shouldn’t do it at all; that if we were going to spend more than we would ever save, that we should just save ourselves the headache and not do it. However, I and most people who give it serious thought recognize that there is a great deal of payoff in rationalizing one’s architecture, and that the payout occurs over a long period of time in increased adaptability, reduced maintenance, additional reuse, etc.
The ROI Denominator
Another piece of the ROI equation is the denominator. In other words, return on investment is what benefit or what return you got divided by the investment. One of the difficulties in justifying some architectural projects is that the denominator is just too large. Some architectural projects will overwhelm and outweigh any conceivable benefit. However, these projects do not have to be large and costly to be effective. Indeed, we find that the best combination is a fairly modest architectural planning project, which then uses monies that would have been spent anyway, supplemented with small bits of seed money, to grow the architecture in a desired direction incrementally over a large period of time. Not only does this reduce the denominator, more importantly, it reduces the risk because with any large infrastructure-driven architectural project there’s not only a large investment to recoup but there’s always the risk that the architecture might not work at all.
Getting the Right Metrics
The final problem, even after 50 years of software development, is that we’re still not routinely collecting the metrics we need to make a rational decision in this area. Sure we have a few very gross metrics, such as percent of IT spending to sales or proportion of new development versus maintenance; and we have some low-level metrics such as cost per line of code or cost per function point. But neither of these are much help at all in shining light on the kinds of changes that will make a great economic difference. In order to do this, we now need to look at the numerator of the ROI equation. The numerator will consist primarily of benefits expressed as cost savings relative to the current baseline. Within that numerator we will divide the activity into the following categories:
- Operational costs
- User costs
- Maintenance costs
- Opportunity costs
Operational Costs
Operational costs are the total costs required to keep the information systems running. This includes hardware costs: leases and amortization of purchased costs; software licensing costs: onetime costs and especially annual support costs. This also includes all the administrative costs that must be borne to support the systems, such as backup, indexing, and the like. In addition, we should include the costs required for direct support such as help desk support, which is required just so that users can continue to use the software systems as intended. This category of costs is relatively easy to get in the aggregate. In other words, from your financial statements you probably have a pretty good idea of these costs in total. In many organizations it’s more difficult to break these costs into smaller units, where the investment analysis really occurs. This includes breaking them down into application level costs as well as cost per unit of useful work, which we will talk about more toward the end of this article. An enterprise architecture can very often have dramatic impacts on operational costs. Some of the more obvious include shifting production from inappropriate hardware and operating system platforms. For example, in many cases mainframe data centers are cost disadvantaged relative to newer technology. Also very often the architecture can help with the process of rationalizing vendor licenses by consolidating database management systems or application vendors. There can be considerable savings in that area. An architecture project may be targeted at changing an unfavorable outsourcing arrangement or conversely, introducing outsourcing if it’s economically sensible. The longer run goals of service oriented architecture are to make as many as possible of the provided services into commodities with the specific intention of driving down the cost per unit of work. Without the architecture in place your switching costs are high enough that, for most shops, it’s very difficult to work a process of ratcheting down from higher to lower cost suppliers.
User Costs
By user costs we mean the amount of time spent by users of the system over and above the absolute bare minimum that would be required to complete a job function. So if a job requires hopping from system to system, navigating, copying data from one system to another or from a report into a system, transcribing, studying, etc., all of this activity is excess cost created by the logistics of using the system. These costs are much harder to gather because they are spread throughout the organization and there is no routine collection of the amount of time spent on these activities versus other non system mediated activities. Typically, what’s needed in this area is some form of activity-based costing, where you can audit on a sampling basis how people spend their time and compare that against “should cost” analysis of the same tasks. Even when the task has been off loaded to the end user, what’s called “self-service,” it still may be worthwhile to review how much excess time is being used. In this case, it’s not so much a measure of the resources lost from the organization but it may be an indicator that competitors may be able to take advantage of an effort difference and steal customers. Many aspects of service oriented architecture are aimed exactly at this category of costs. Certainly, all the composite application, much of the system integration, and the like, is aimed at reducing non value-added time that the end users spend with their systems. Implementing service oriented architecture as well as workflow or business process management or even process orchestration can be aimed directly at these costs.
Maintenance Costs
These are the costs of keeping an information system in working order. It includes breakdown maintenance, which is fixing problems that occur in production; preventative maintenance, which is rarely done in the information industry but would include re-factoring; and reactive maintenance, which is maintenance that is required due to changes in the environment. This category includes changes to the technical environment, such as when an operating system is discontinued and we’re forced to maintain an application to keep it running, as well as changes in the regulatory environment where a law changes and we are forced to make our systems comply. I did not include proactive maintenance or maintenance that is improving the user or business experience in this category as I will include them in the opportunity cost category. Maintenance costs are typically a function, not so much of the change to be made, but of the complexity of the thing to which the change is being applied. Most maintenance to mature systems involves relatively small numbers of lines of code. Especially when we exclude changes that are meant to improve the system we find fewer and fewer lines of code for any given maintenance activity. That’s not to say that maintenance isn’t consuming a lot of time; it is. Maintenance very often involves a great deal of analysis to pinpoint either where the operational problem is or where changes need to be made to comply with environmental change. Once the change site is identified another large amount of analysis needs to be done to determine the impact the change is likely to have. Unfortunately, the trend in the nineties to larger integrated systems essentially meant larger scope to search the problem and larger scope for the change analysis. The other major difficulty with getting metrics on maintenance is that many architectural changes eliminate the cost so effectively that people no longer recognize that they are saving money. One architectural principle that we used in the mid-nineties was called “riding lightly on the operating environment.” We argued that a system could have the smallest possible footprint onto the operating system’s API. In many ways this is the opposite way many applications are built. Many application developers try to get the maximum use out of their operating environment, which makes sense in the short term development productivity equation, but as we discovered the fewer points of contact you have with the operating system the more immune you are to changes in the operating system. As a result, systems we’ve built in that architecture survived multiple operating system upgrades, in many cases without even a recompile and in other cases with only a simple recompile or a few lines of code changed. The well-designed enterprise architecture goes far beyond that in the realm of reducing long-term maintenance costs. In the first place, the emphasis on modularity, partitioning, and loose coupling means that there are fewer places to look for the source of a problem, there is less investigation to do for the side effect, and any extremely problematic area can just be replaced. In this area, we will likely have to calculate the total cost per environmental change event, such as the cost to upgrade a version of a server operating system or, as we recently have history with, the cost when the environment changed at Y2K and two digit years were no longer sufficient.
Opportunity Costs
In this category I’m putting both the current cost to make proactive changes – in other words, cost to improve the system to deliver better information with fewer keystrokes – as well as the cost of the opportunity lost for not making a change because it was too difficult. The architecture can drastically improve both of these measures. In the first case, it’s relatively straightforward to get the total cost of proactive changes to current systems. However, we likely need to go much beyond that and look at change by types. For instance, what is the cost of adding additional information into a use case? What is the cost of streamlining three steps in the use case into one? Perhaps someone will come up with a good taxonomy in this area that would give some comparable values across companies. We also have to include the non-IS costs that go into making these kinds of changes. Not only does this include the business analyst’s design time but if it’s currently possible for analysts to make a change on their own, we should count that as development time. In the longer run, we expect end user groups to make many of these kinds of changes on their own; and indeed, this is one of the major goals of some of the components of an enterprise architecture. The other side, the opportunity lost, is very difficult to measure and can only be done by interviewing and guesstimate. But it’s very true that many companies miss opportunities to enter new markets, to improve processes, etc., because their existing systems are so rigid, expensive, and hard to maintain that they simply don’t bother. Also in this category are the costs of delay. If a proposed change would have a benefit but people delay, for often times years, in order to get the budgetary approval to make the change that’s necessary, this puts off the benefit stream potentially by years. With a modern architecture, very often the change requires no capital investment and therefore could be implemented as soon as it’s been well articulated.
Putting It All Together
An enterprise architecture project can have a substantial return on investment. Indeed, it’s often hard to imagine things that could have a larger return on investment. The real question is whether the organization feels compelled to calculate the ROI. Most organizations that succeed with enterprise architecture initiatives, in my observation, have done so without rigorous ROI analyses. For those that need the comfort of an ROI, there is hope. But it comes at a cost. That cost is the need to get, in the manner we’ve described in this white paper, a rigorous baseline of the current costs and what causes them. Armed with this you can make very intelligent decisions about changes to your architecture that are directly targeted at changing these metrics. In particular, I think people will find that the change that has the biggest multiplier effect is literally changing the cost of change.
White Paper: The Seven Faces of Dr. “Class”
The Seven Faces of Dr. “Class”: Part 1
“Class” is a heavily overloaded term in computer science. Many technologies have implemented the concept slightly differently. In this paper we look at the sum total of concepts that might be implemented under the banner of “class” and then later we’ll look at how different technologies have implemented subsets.
The seven facets are:
- Template
- Set
- Query
- Type
- Constraint
- Inclusion
- Elaboration
Template
One aspect of a class is to act as a “template” or “cookie cutter” for creating new instances. This is also called a “frame” based system, where the template sets up the frame in which the slots (properties) are defined. In the simplest case, say in relational where we define a table with DDL (Data Definition Language) we are essentially saying ahead of time what attributes a new instance (tuple) of this class (table) can have. Object Oriented has this same concept, each instance of a class can have the attributes as defined in the class and its superclasses.
Set
A class can be seen as a collection of all the instances that belong to the set. Membership could be extensional (that is instances are just asserted to be members of the class) or intensional (see below under the discussion about the inclusional aspect). In the template aspect, it’s almost like a caste system, instances are born into their class and stay there for their lifetime. With set-like classes an instance can be simultaneously members of many sets. One of the things that is interesting is what we don’t say about class membership. With sets, we have the possibility that an instance is either provably in the set, provably not in the set, or satisfiably either.
Query
Classes create an implied query mechanism. When we create instances of the Person class, it is our expectation that we can later query this class and get a list of the currently know members of the class. In Cyc classes are called “collections” which reflect this idea that a class is, among other things, a collection of its members. A system would be pretty useless if we couldn’t query the members of a class. We separate the query facet out here to shine a light on the case where we want to execute the query without previously having defined the class. For instance if we tag photos in Flickr with a folksonmy, and someone later wants to find a photo that had a combination of tags, a class, in the traditional sense was not created, unless you consider that the act of writing the query is the act of creating the class, and in which case that is the type of class we’re talking about here. This is primarily the way concept like taxonomies such as SKOS operate: tags are proxies for future classes.
Type
Classes are often described as being types. But the concept of “type” despite being bandied about a lot is rarely well defined. The distinction we’re going to use here is one of behavior. That is, it is the type aspect that sets up the allowable behavior. This is a little clearer in implementations that have type and little else, like xsd. It is the xsd type “date” that sets up the behavior for evaluating before or after or concurrent. And it is the polymorphism of types in object oriented that sets up the various behaviors (methods) that an object can respond to. It is the “typeness” of a geographicalRegion instance that allows us to calculate things like its centroid and where the overlap or boundary is with another geographicalRegion. We rarely refer to the class of all items that have xsd:date as if it were a collection, but we do expect them to all behave the same.
Constraint
Constraints are generally implemented as “guards” and prevent noncompliant instances from being persisted. There is no reason that the constraints need to be associated with the classes, they could easily be written separately and applied to instances, but many implementations do package constraints with the class definition, for two reasons: one the constraints are naturally written in and lexically tied to the cl ass definition and the other is just for packaging around the concept of cohesion. The constraint can be a separate language (as with OCL the Object Constraint Language) or may be an extension to the class definition (as ranges and foreign key constraints are in relational).
Inclusion
That is, inclusion criteria. This is for classes that support inference, and are the rules that determine whether an instance is a member of the class, or whether all members of a class are necessarily members of another class. It also includes exclusion criteria, as they are just inferred membership in the complement. While it is conceivable to think of the “open world” without inclusion criteria, it really comes to the fore when we consider inclusion criteria. Once we have rules of inclusion and exclusion from a set, we have set up the likelihood that we will have many instances that are neither provably members or provably not members, hence “satisfiability.”
Elaboration
Elaboration is what else can be known about an item once one knows its class membership. In Object Oriented you may know things about an instance because of the superclasses it is also a member of, but this is a very limited case: all of this elaboration was known at the time the instance was created. With more flexible systems, as an instance creates new membership, we know more about it. For instance, let’s say we use a passport number as evidence of inclusion in the class of Citizens, and therefore the class of People, we can know via elaboration that the passport holder has a birthday (without knowing what their birthday is). To the best of our knowledge, there is no well supported language and or environment that supports all these facets well. As a practical consequence designers select a language implement the aspects that are native and figure out other strategies for the remaining facets. In the next installment of this series, we will examine how popular environments satisfy these aspects, and what we need to do to shore up each.
FOIS Keynote: Ontology-Driven Information Systems
Orginally published in October 2008
The following paper was given as a keynote address at the 5th International Conference on Formal Ontology in Information Systems held in Saarbrücken, Germany in October 2008.
Title: Ontology-Driven Information Systems: Past, Present and Future Author: Michael Uschold Abstract : We trace the roots of ontology-drive information systems (ODIS) back to early work in artificial intelligence and software engineering. We examine the lofty goals of the Knowledge-Based Software Assistant project from the 80s, and pose some questions. Why didn’t it work? What do we have today instead? What is on the horizon? We examine two critical ideas in software engineering: raising the level of abstraction, and the use of formal methods. We examine several other key technologies and show how they paved the way for today’s ODIS. We identify two companies with surprising capabilities that are on the bleeding edge of today’s ODIS, and are pointing the way to a bright future. In that future, application development will be opened up to the masses, who will require no computer science background. People will create models in visual environments and the models will be the applications, self-documenting and executing as they are being built. Neither humans nor computers will be writing application code. Most functionality will be created by reusing and combining pre-coded functionality. All application software will be ontology-driven.
Interviews on Business Semantics
Links to two interviews with Dave regarding business semantics. Click here for an interview involving semantics, ontologies, and modeling. Click here for the transcript of an inteview, “Why is Business Semantics such a Hot Topic?”
Web 3.0
I’m weighing in in favor of Web 3.0 as an alias for the Semantic Web.
I’m weighing in in favor of Web 3.0 as an alias for the Semantic Web. I know there are a lot of people who will roll their eyes and initiate some anti hype exorcism, but let’s have a sober look at the pluses and minuses here. Web 3.0 is not without its problems. The first problem with Web 3.0 is that everyone is defining it to their own ends. As Montoya Herald summed it up at http://www.christianmontoya.com/2007/10/08/web-30-i-about-money/, Web 3.0 is essentially whatever each of the companies that used the term are working on next. The second problem is that it does pander to the hypemeisters. But the very people who decry hype the loudest are often those who benefit from it the most. (Who can argue that the hype of the Web and Web 2.0 didn’t advance the careers and opportunities of the very people who now think Web 3.0 is hype?) A lot of people seem to be comfortable with Web 2.0 now, despite the fact that it has no real unifying principle. Web 2.0 is blogs and wikis and Facebook and MySpace (user generated content) and AJAX and Rich Internet Applications for a richer user experience in a browser, but really there isn’t anything holding it together or giving it a defined shape. Maybe we don’t need to call the Semantic Web: Web 3.0. But if we don’t some other marginal improvement in an existing technology will claim the moniker. In other words, there will be a Web 3.0 and we will find ourselves explaining to people: “well yes, but that is just a part of the vision…” Isn’t the term “Semantic Web” good enough? It’s good for the population that is already “in the tent” but it suffers from being the next big thing for too long for many others. Many people have discounted what they believe the Semantic Web to be(often by making up things that it isn’t and then objecting to that straw man). Web services suffered from a similar fate, for a long time, as thought leaders confused it with services delivered over the web (Software as a Service for instance) which it has some things in common with, but the two aren’t the same. For some, calling the Semantic Web: Web 3.0 gives an opportunity to take another look. So, I’m coming down in favor of “Web 3.0 = Semantic Web.” What do you think?
Silos of Babel
Semantic technology can be a ‘lingua franca’ to connect disparate information silos.
Several speakers and writers have referred to the problem of systems integration as being one of getting systems written in different languages to communicate. Whenever I’ve heard or seen this, the speaker/writer goes on to
say that the difficulty is that getting COBOL to speak to Java is like translating Spanish to Chinese. (Each speaker puts different languages in there.) I like an analogy as much as the next person, but this one strains it a bit too much.
Semantically, COBOL and Java (or any two procedural languages) aren’t that different. If you look in their BNF (Backus Naur Form – a standard way of expressing the grammar of a programming language) you’ll find only a few dozen key words in each language. Yes, there are grammatical differences and some things are harder to express in some languages than others.
But this isn’t where the problems come from. More modern languages (C++, Java, C# and the like) have very large frameworks and libraries, each consisting of tens of thousands of methods. If you wanted to make a case that the semantic differences in languages made integration difficult, this would be the place to make it.
Certainly it is these libraries that make learning a new language environment difficult, and it is these libraries that provide most of the behavior of a system, and therefore most of its semantics. This is what makes systems conversion difficult, for in converting a system to an equivalent in another environment, all the behavior must be reimplemented in an equivalent way in the new environment, and depending on how much of the environment was used this can be a large task. But this still isn’t where the integration problem lies.
Most integration is performed at the data level, either through extract files or some sort of messages. What makes integration difficult is each system to be interfaced has developed its own language about what each data attribute “means.” This is obvious with custom systems: analysts interview users, get their requirements and turn them into data designs. Each table and attribute means something, and this meaning is conveyed partially in the name of the item and partially in its documentation. Developers develop the system, procedure writers develop flow and procedures, and end users develop conventions.
Eventually each item means something, often close to what was intended, but rarely exactly the same. The same thing happens more subtly with packaged software. First, most packages have far more complex schemas than would be developed in a comparable custom system (the package vendors have to accommodate all possible uses of their package). Second, package vendors have tended to make their field definitions and validation more generic, as this is the low road to flexibility.
The process of implementing a package is really one of finalizing the definition of many of the attributes, some of which is done through parameters and some of which is procedural and convention. The net result is that a large organization will have many applications (classically each in its own “silo” or vertical slice of the organization) each of which has its data encoded in its own schema, essentially its own language. I sometimes call these their own idiolects (a dialect for one).
In my opinion it is these “silos of Babel” that create most of the difficulty, cost and expense in integrating. Luckily we have a solution to this the application of semantic technology as an intermediary, a lingua franca if you will, not of computer languages or libraries, but of meta data.
Written by Dave McComb
Schools of Enterprise Application Architecture
An Enterprise Application Architecture is the coordinating paradigm through which an organization’s IT assets inter-relate to form the computing infrastructure and support the business goals.
The choice of architecture impacts a range of issues including the cost of maintenance, the cost of development, security, and the availability of timely information.
Introduction
Architecture in the physical world often conforms to a style. For the most part we know what to expect when we hear a building described as ‘Tudor’ or ‘Victorian.’ All this really means is that the builder and architect have
agreed to incorporate certain features into their design which are representative of a given school of thought for which we have a label. Similarly, there are schools of thought in Enterprise Architecture which when followed produce equally distinctive architectural results. This paper is an overview of the more prominent of these schools.
The Default Architecture
Imagine an enterprise – and for many of us this might be quite easy – in which no unifying discipline is applied to the application development and design process. Applications are created as needs are discovered, their implementation is directed by that part of the organization providing the budget, and their scope is constrained by the organizational unit within which they are conceived. The applications themselves perhaps go through a rigorous and well understood development process which ends, often as an afterthought, with an integration task in which the shiny new application is plugged into the rest of the enterprise.
Imagine, further, that we are now looking at this enterprise after this approach to application development has been practiced for years, and perhaps decades. What we will see is an evolutionary enterprise architecture in which the impact of non-technical issues, such as the personalities of the managers and the distribution of budgets between lines of business, is clearly visible in the legacy fossil record. The architecture will be a collection of seemingly arbitrarily defined and scoped applications tightly coupled to each other through hand crafted point-to-point interfaces. The problems with this architecture become evident as the enterprise grows. In theory, if every application has to share data with every other application, then the number of interfaces will approximate n(n-1)/2, where n is the number of applications.
This is an O(n2) growth in complexity, and is consequently a point of failure as the enterprise infrastructure grows, and n becomes large. In reality, of course, all of the applications do not share data with all the other applications, which suggests the increase in complexity is not necessarily exponential, but it is nonetheless a problem. John Zachman, creator of the Zachman framework, describes what occurs when we create applications in this manner as ‘post integration.’ The difficulty with post-integration, as he points out, is with semantic consistency. It becomes increasingly difficult to make sure that what we mean by a piece of data in one application is what we mean by that same piece of data in a different system which has received the data through one or more interfaces.
Controlling, and indeed just discovering, the semantics of the data is a difficult undertaking with this architecture, and without a clear definition of the semantics virtually nothing can be done with confidence. Poorly controlled semantics, of course, is exacerbated by another characteristic of the default architecture, which is uncontrolled data replication. Our multiple applications each have their own database, with their own copies of data, which they interpret in their own way. The replicated copies of data, in turn, rapidly become out of synch with each other, leading to an environment in which data meaning, currency and validity are all uncertain.
A fundamental problem with the default architecture is application coupling, by which we mean that a change to any application will have a scope of effect beyond that application. The enterprise applications are all tangled together, a bit like a ball of string. This means that changes that should logically be simple, localized and cheap end up as complex, broad ranging and expensive. The problems with the default architecture are manageable in small enterprises. It is only as the enterprise becomes larger that they become impossible to control. This paper is an overview of the successive approaches which enterprises have employed in response to this issue.
The Integrated Database Architecture
With the advent of the scaleable relational database in the mid 1980s enterprises saw a possible solution to the complexity created by the default architecture. The theory was quite compelling; the enterprise should have a single, large database which implements an enterprise wide conceptual data model. The semantics of the data would be centrally defined and under tight control. All of the application logic would operate off of the single data store, and its central definition.
Consequently there would be no growth in the number of interfaces, because there would be no need to have interfaces. There would be no data consistency or replication problems, because there would only be one copy of the data. Constraints in the database would require data to be collected completely by all parts of the application logic, and to be consistent. This was a seemingly perfect solution which most enterprises embraced enthusiastically.
The problems with the integrated database architecture did not appear immediately. The primary problem, in fact, is one of change over time. The integrated database is a great point in time solution. The problem is that once we have all of our applications based on this single database we have created, in programming terminology, a single giant ‘global variable,’ which, if changed in any way, has a potential effect everywhere. In other words the integrated database gives us tremendous data integration at the cost of extremely tight application coupling.
If, for example, we wish to change the logic of an application, perhaps to send our customers birthday cards, we are changing the same data structure – the database – which we are using to run our mission critical systems, and we are potentially also having to change those mission critical systems even though they do not care at all about birthday cards. So, at the end of the day, we are more likely than not to decide that we won’t change the mission critical systems, for reasons of risk and cost, and that we will rather forgo the birthday card function. The tightly coupled integrated database, then, has a flexibility problem.
Business processes change over time, and each change potentially impacts all of our applications. This means that making any single change is disproportionately expensive, and tends to be resisted, producing a non-responsive IT support infrastructure. When the business cannot change its core systems cost effectively, enterprising business users and IT managers will typically conclude that the obvious answer is to build ‘their own’ little system in parallel to the integrated database and then build an interface. This can often look like a constellation of Microsoft Access databases circling a mainframe, and performing both pre and post processing to support the business’ actual processes. In due course the peripheral applications become as important as the integrated database applications, and eventually the integrated database architecture begins to look like the default architecture, with one or more anomalously large applications. In the end, the integrated database architecture fails because it cannot inexpensively accommodate rapidly changing business processes, which are a hallmark of the modern enterprise.
The Distributed Object Architecture
The arrival of Object Orientation in the early 1990s heralded yet another approach to enterprise architecture. This approach said, in effect, that the problem with the integrated database architecture was one of programming. In order to create an application a developer would have to understand this large complex schema, and would then create logic to manipulate it. This logic, or behavior, defined for the data does, in effect, define the semantics of that data. Having developers re-define logic each time for core bits of functionality creates a ‘semantic drift,’ where the actually implemented behavior, from application to application, is inconsistent.
The distributed object architecture is a discipline which requires the enterprise to create an object representation of its core concepts, such as Customer, Order, and so on. When developers create an application they do so by invoking this predefined behavior, thus ensuring semantic equivalence between applications. The distributed object architecture is attractive in so far as the object analysis process extends logically from the Semantic Model, and leads to a centrally defined and controlled definition of data semantics and process. It is clearly an improvement over straight procedural logic sitting on top of a global database schema, as in the Integrated Database architecture, but it is only an incremental improvement.
The core limitations of the Integrated Database architecture, namely tight coupling and inflexibility, live on in the Distributed Object Architecture, which is not a surprise given that this approach is really nothing more than an object veneer over the integrated database. The distributed object architecture is implemented in a variety of technologies, including Enterprise Java Beans, Microsoft DCOM, and the Common Object Request Broker (CORBA).
The Message Bus Architecture
Flexibility has become one of the most important qualities of enterprise application architectures. ‘Flexibility’ is the capacity to change elements of the architecture at acceptable cost. The key to creating a flexible architecture is to decouple the independent pieces from one another, such that a change to one of the pieces does not unnecessarily require a change in any of the other independent pieces.
This capability is what has been missing from the prior architectures, and this is the primary contribution of the Message Bus Architecture. The message bus architecture returns us to an environment of independent applications maintaining their own databases. We add to this (typically) an ‘integration broker’, which is broadly responsible for communicating data between applications. The data communicated in this way is referred to as a message. By introducing the message broker as an intermediary, we are able to decouple applications from one another. Semantic consistency is enforced by representing the enterprise conceptual data model as a message model, or a centrally controlled message schema.
The n(n-1)/2 point to point to point interfaces are replaced by n interfaces – one from each application to the broker. We necessarily introduce a degree of data replication, but we control the replication through a change notification mechanism provided by the broker, typically in the form of publish/subscribe messages.
This controlled replication manages the data consistency issues, while at the same time creating a degree of ‘runtime decoupling,’ which allows the independent applications to operate even though other parts of the infrastructure may be unavailable. In this environment applications can be implemented in any technology and using whatever database schema they choose.
Their obligation to the enterprise is to generate a set of defined messages conforming to the message model, and to process incoming messages. They are free to change as and when they wish, as long as they continue to support their message contract. This is what is meant by decoupling, and this is where flexibility originates.
In a Message Bus architecture the message routing function, that is to say the logic which controls where a message is delivered, can be centralized in the broker without a loss of decoupling. When this is done, it becomes possible to see the message broker as a business process management (BPM) tool, and as a means of implementing enterprise wide workflow through the addition of rules. When fully supported by the applications, the Message Bus Architecture allows the implementation of the ‘real time enterprise,’ in which all business events, regardless of origin, appear on the message bus and can be consumed by any interested application.
This can become especially interesting when events generated by external business partners reach the internal message bus, and vice-versa. The Message Bus Architecture requires careful implementation to provide true decoupling and flexibility. It is quite possible to create a network of point-to-point logical interfaces over the single technology interface to the broker. This occurs when applications create ad-hoc messages for every integration case; the solution here is to proactively architect the message model and ensure that it is not circumvented at the application level. The Message Bus Architecture is not, however, a complete solution.
At the technology level it is usually implemented with proprietary technology for the message broker, which is expensive to buy, and requires scarce and equally expensive personnel to use. The distributed nature of the solution necessarily creates multiple points of failure – which can be mitigated through careful design to maximize runtime decoupling – and one central point of failure in the integration broker. Performance issues are also a potential problem; poor application partitioning can create excessively high volumes of messages, and some use cases can be impacted through high network latency. The Message Bus Architecture is a viable solution, but it is not a trivial implementation.
Service Oriented
The Service Oriented Architecture is a refinement on the Message Bus Architecture. The advance with this architecture is the realization that many large granularity functions are automated in the enterprise in multiple places.
Many of our applications will do reporting, most applications will implement a user interface, most will concern themselves with security, and most will implement some form of business logic, and so on. The Service Oriented Architecture posits that the applications should be refactored and these pieces of functionality should be removed from the applications and implemented as a single ‘service’ which can be invoked at runtime. So, for example, reporting becomes a responsibility of the Information Delivery service, which might be implemented through a data warehouse, the user interface might be delegated to a portal service, the security functions will be implemented by an authentication and authorization service, and business logic perhaps by a business rules service.
The likely candidates for service orientation tend to be business neutral, in part because these functions appear repeatedly across the application inventory. The trade off of creating a service is that we have potentially created runtime-coupling between the service and the invoking application, and consequently created a point of failure. The benefit is the reduction of redundant functionality, and its central control and unification.
Taken to a logical extreme Service Orientation will allow applications to divest themselves of the responsibility for security, business logic, workflow and presentation, leaving very little beyond data store and configuration. Service Orientation can be implemented in a messaging environment using a broker, however this is not a requirement. Much of the current literature confuses the implementation technology with the concept, especially where the implementation technology is Web services.
The Orchestrated Web-Service Architecture
The latest technology trend is Web services. Web services are positioned to become the open standards based implementation of the Message Bus Architecture. Where applications currently communicate with the Message Bus using a vendor proprietary adaptor we will have a standard Web service interface instead.
Where the Message Bus Architecture performs message routing using a proprietary extensional routing – or orchestration – tool, or using intentional publish/subscribe logic, the web-service architecture will use the corresponding web service standard, at present BPEL4WS. Where the Message Bus Architecture implements guaranteed delivery through proprietary queuing mechanisms such as IBM-MQSeries and others, the Web service architecture will use upcoming standards such as HTTP-R or Web services reliable messaging.
The Web services standards are currently incomplete, and don’t fully overlap the proprietary products offerings, however the promise is clearly that in the near future Web services will offer an open standards alternative. Web services are by nature point-to point connections. Used naively this will create a technically state of the art implementation of the Default Architecture, with applications tightly bound to each other through many uncontrolled interfaces. The Orchestrated Web service architecture, consequently, introduces a broker to which all Web service calls are made, and which is responsible for forwarding those requests to the applications providing the service.
This centralized orchestration is what allows the Web service approach to remain decoupled. Similarly, by implementation asynchronous request/reply logic – which is to say the requestor does not block waiting for the reply – and by supplementing the standard Web service call over HTTP with guaranteed delivery, the broker is able to create an environment which is similar to that of the Message Bus Architecture. The Web service architecture is practical today, supplemented with various proprietary technologies. It represents an improvement over the Message Bus Architecture by being based on open standards and consequently reducing vendor lock-in.
The Just-in-time Integration Architecture
One of the interesting capabilities which the Web service technologies introduced is the concept of runtime discovery. The UDDI Web service specification allows an application to find a service at runtime, to bind to it, and invoke it. The client application searches for the service based on service categorization and conformance to an interface specification – in this case a WSDL document.
This capability allows us to conceive of an architecture in which Applications and Services create web-service interfaces and place their WSDL descriptions in the enterprise UDDI repository. When an application wishes to invoke a service it looks it up in the repository and invokes it. The key benefit of this approach is that the inter-application binding is entirely dynamic and consequently decoupled; we can replace the service provider at any time simply by changing its entry in the UDDI repository. With this approach there is no broker, and consequently there are no centrally provided management and control functions. However, in a decentralized internet based situation this maybe an appropriate architectural choice.
Conclusion
The choice of Enterprise Application architecture is critical to creating a successful IT infrastructure which is responsive to the business needs and which reinforces the qualities which are of value to the organization. All of the schools of architecture which are described here can be valid choices, just as building a Victorian style house is as legitimate a decision as building in a Tudor style. It is the responsibility of the architect, however, to ensure that the chosen architecture is appropriate for its environment. Although these architectural schools are evolving, and new ones are being created, most enterprises are clearly in a position to benefit from the adoption of a defined enterprise application architecture.
Written by Dave McComb
