In this column, I am making the case for Data Centric architectures for enterprises.  There is a huge economic advantage to converting to the data-centric approach, but curiously few companies are making the transition. One reason may be the confusion of Data Centric with Data Driven, and the belief that you are already on the road to data centric nirvana, when in fact you are nowhere near it.

Data-Centric

Data-centric refers to an architecture where data is the primary and permanent asset, and applications come and go.  In the data-centric architecture, the data model precedes the implementation of any given application and will be around and valid long after it is gone.

Many people may think this is what happens now or what should happen.  But it very rarely happens this way.  Businesses want functionality, and they purchase or build application systems.  Each application system has its own data model, and its code is inextricably tied with this data model.  It is extremely difficult to change the data model of an implemented application system, as there may be millions of lines of code dependent on the existing model.

Of course, this application is only one of hundreds or thousands of such systems in an enterprise.  Each application on its own has hundreds to thousands of tables and tens of thousands of attributes. These applications are very partially and very unstably “interfaced” to one another through some middleware that periodically schleps data from one database to another.

The data centric approach turns all this on its head. There is a data model—a semantic data model (but more on that will be in a subsequent white paper)—and each bit of application functionality reads and writes through the shared model.  If there is application functionality that calculates suggested reorder quantities for widgets, it will make its suggestion, and add it to the shared database, using the common core terms.  Any other system can access the suggestions and know what they mean.  If the reordering functionality goes away tomorrow, the suggestions will still be there.

Click here to read more on TDAN.com

Model Driven Development

Model Driven seems to be enjoying a bit of an upsurge lately.  Gartner has recently been hyping (is it fair to accuse the inventors of the hype curve of hyping something? or is it redundant?) what they call “low code/ no code” environments.

Perhaps they are picking up on and reporting a trend, or perhaps they are creating one.

Model Driven Development has been around for a long time. To back fill what this is and where it came from, I’m going to recount my experience with Model Driven, as I think it provides a first person narrative for most of what was happening in the field at the time.

I first encountered what would later be called Model Driven in the early 80’s when CAD (Computer Aided Design—of buildings and manufactured parts) was making software developers jealous.  Why didn’t we have workbenches where we could generate systems from designs?  Early experiments coalesced into CASE (Computer Aided Software Engineering).  I was running a custom ERP development project in the early 80’s (on an ICL Mainframe!) and we ended up building our own CASE platform.  The interesting thing about that platform was that we built the designs on recently acquired 8-bit microcomputers, which we then pushed to a compatible framework on the mainframe.  We were able to iterate our designs on the PCs, work out the logistical issues, and get a working prototype UI to review with the users before we committed to the build.

The framework built a scaffold of code based on the prototype and indicated where the custom code needed to go.  This forever changed my perspective on how systems could and should be built.

What we built was also being built at the same time by commercial vendors (we did this project in Papua, New Guinea and were pretty out of the loop as to what was happening in mainstream circles).  When we came up for air, we discovered what we had built was being called “I-CASE” (Integrated Computer Aided Software Engineering), which referred to the integration of design with development (seemed like that was the idea all along).  I assume Gartner would call this approach “low code” as there still was application code to be written for the non-boiler-plate functionality.

Next stop on my journey through model driven was another ERP custom build.  By the late 80’s a few new trends had emerged.  One was CAD was being invaded by parametric modeling.  Parametric modeling recognizes that many designs of physical products did not need to be redesigned by a human every time a small change was made to the input factors.  A motor mount could be designed in such a way that a change to the weight, position, and torque would drive a new design optimized for those new factors. The design of the trusses for a basketball court could be automatically redesigned if the span, weight, or snow load changed and the design of big box retail outlets could be derived from, among other things: wind shear, maximum rainfall, and seismic potential.

The other trend was AI (remember AI?  Oh yeah, of course you remember AI, which you forgot about from the early 90’s until Watson and Google’s renaissance of AI).

Being privy to these two trends, we decided to build a parametric model of applications and have the code generation be driven by AI.  Our goal was to be able to design a use case on a post-it note. We didn’t quite achieve our goal.  Most of our designs were up to a page long.  But this was a big improvement over what was on offer at the time.  We managed to generate 97% of the code in this very sophisticated ERP system.  While it was not a very big company, I have yet to see more complex requirements in any system I have seen (lot based inventory, multi-modal outbound logistics, a full ISO 9000 compliant laboratory information management system, in-line QA, complex real time product disposition based on physical and chemical characteristics of each lot).

In the mid 90’s we were working on systems for ambulatory health care.  We were building semantic models for our domain.  Instead of parametric modeling we defined all application behavior in a scripting language called tcl. One day we drew on a white board where all the tcl scripts fit in the architecture (they defined the UI, the constraint logic, the schema, etc.) It occurred to us that with the right architecture, the tcl code, and therefore the behavior of the application, could be reduced to data.  The architecture would interpret the data, and create the equivalent of application behavior.

We received what I believe to be the original patents on fully model driven application development (patent number 6,324,682).  We eventually built an architecture that would interpret the data models and build user interfaces, constraints, transactions, and even schemas.  We built several healthcare applications in this architecture and were rolling out many more when our need for capital and the collapse of the .com bubble ended this company.

I offer this up as a personal history of the “low code / no code” movement.  It is not only real, as far as we are concerned, but its value is underrepresented in the hype.

Data-Centric Architecture

More recently we have become attracted to the opportunity that lies in helping companies become data-centric.  This data-centric focus has mostly come from our work with semantics and enterprise ontology development.

What we discovered is that when an enterprise embraces the elegant core model that drives their business, all their problems become tractable.  Integration becomes a matter or conforming to the core.  New system development becomes building to a much, much simpler core model.

Most of these benefits come without embracing model driven.  There is amazing economy in reducing the size of your enterprise data model by two orders of magnitude.

Click here to read more on TDAN.com

In the previous installment (The Data Centric Revolution: The Evolution of the Data Centric Revolution Part One), we looked at some of the early trends in application development that foreshadowed the data centric revolution, including punched cards, magnetic tape, indexed files, databases, ERP, Data Warehouses and Operational Data Stores.

In this installment, we pick up the narrative with some of the more recent developments that are paving the way for a data centric future.

Master Data Management

Somewhere along the line, someone noticed (perhaps they harkened back to the reel-to-reel days) that there were two kinds of data that are, by now, mixed together in the databased application: transactional data and master data.  The master data was data about entities, such as Customers, Vendors, Equipment, Fixed Assets, or Products.  This master data was often replicated widely. For instance, every order entry system has to have yet another Customer table because of integrity constraints, if nothing else.

If you could just get all the master data in one place, you’d have made some headway.  In practice, it rarely happened. Why? In the first place, it’s pretty hard.  Most of the MDM packages are still using older, brittle technology, which makes it difficult to keep up with the many and various end-points to be connected.  Secondly, it only partially solved the problem, as each system still had to maintain a copy of the data, if for nothing else, for their data integrity constraints.  Finally, it only gave a partial solution to the use cases that justified it. For example, the 360^o view of the customer was a classic justification, but people didn’t want a 360 ^o view of the master data; they wanted to see the transaction data.  Our observation is that most companies that had the intention to implement several MDMs  gave up after about 1 ½ years when they found out they weren’t getting the payout they expected.

Canonical Message Model

Service Oriented Architecture (SOA) was created to address the dis-economy in the system integration space.  Instead of point-to-point interfacing, you could send transactional updates onto a bus (the Enterprise Service Bus), and allow rules on the bus to distribute the updates to where they are needed.

The plumbing of SOA works great.  It’s mostly about managing messages and queues and making sure messages don’t get lost, even if part of the architecture goes down. But most companies stalled out on their SOA implementations because they had not fully addressed their data issues.  Most companies took the APIs that each of their applications “published” and then put them on the bus as messages.  This essentially required all the other end-points to understand each other.  This was point-to-point interfacing over a bus.  To be sure, it is an improvement, but not as much as was expected.

Enter the Canonical Message Model.  This is a little-known approach that generally works well, where we’ve seen it applied.  The basic concept is to create an elegant [1] model of the data that is to be shared.  The trick is in the elegance.  If you can build a simple model that captures the distinctions that need to be communicated, there are tools that will help you build shared messages that are derived from the simple model.  Having a truly shared message is what gets one out of the point-to-point trap. Each application “talks” through messages to the shared model (which is only instantiated “in motion,” so the ODS problem versioning is much easier to solve), which in turn “talks” to the receiving application.

Click here to continue reading on TDAN.com