We have been portraying the move to a Data-Centric paradigm as a “Revolution” because of the major mental and cultural shifts that are prerequisites to making this shift. In another sense, the shift is the result of a long, gradual process; one which would have to be characterized as “evolutionary.”
This column is going to review some of the key missing links in the evolutionary history of the movement.
(For more on the Data Centric Revolution, see The Data Centric Revolution. In the likelihood that you’re not already data centric, see The Seven Warning Signs of Appliosclerosis)
Applications as Decks of Cards
In the 50’s and 60’s, many computer applications made very little distinction between data and programs. A program was often punched out on thin cardboard “computer cards.” The data was punched out on the same kind of cards. The two decks of cards were put in the hopper together, and voila, output came out the other end. Payroll was a classic example of applications in this era. There was a card for each employee with their Social Security Number, rate of pay, current regular hours, overtime hours, and a few other essential bits of data. The program referred to data by the “column” numbers on the card where the data was found. Often people didn’t think of the data as separate from the program, as the two were intimately connected.
Magnetic Tape
In the magnetic tape era, we replaced cards with reel-to-reel magnetic tape. This gave more independence to the data. Initially, what we did was take the transactional part of the time card data (the current hours reported) and put it on a reel of tape. Being freed from sharing an 80-column card with a bit of master data allowed people to add things like charge codes, projects, and more categories of pay, which they did.
The other part of the card represented data that did not change on a week-by-week basis, “master data.” This master data went on another reel of tape and was called, remarkably, the master data file. When you wanted to run an application program, you first had to get the correct master file (versioning was a big deal) and the correct transaction file. Then you mounted those files and another blank tape on three tape drives, and started the program. The prevailing paradigm was to sort the master file and the transaction file into the same order and process them sequentially, creating the output (paychecks) as well as writing an updated master file onto the blank tape.
Most programs still relied on an offset of a number of characters from the beginning of each record to understand where the data was and what it meant. Because knowledge of this data was in the computer program itself, most of the knowledge of the data was there as well.
The existence of this master file was the beginning of the idea of shared data. An employee master file might be used by the payroll department as well as the HR department. Might have been. Usually wasn’t. There were mechanisms that gave names to the offsets in the records. Mostly, these were “copy books” which were usually chunks of the front part of the Cobol program. In principle, these could be shared between departments who wished to use shared data. Usually this did not happen in practice.
Indexed Files
With the invention of the disk drive [1], we could have master data that was permanently “mounted.” These files had indexes (ISAM and VSAM were popular) that allowed direct access to any record. There was a bit of data sharing, but not as much as you would think. The inventory system kept the inventory master file, the payroll system kept the employee master file, etc.
Databases, Metadata, & Schemas
Databases created a more public and shared view of the data being managed. This was true with the hierarchical and codasyl databases, which were mostly proprietary. This more shared view increased with the advent of relational databases, and especially SQL. It was an open and shared grammar, playing the role of a lingua franca for database access. Not only could many programs share data tables, they did not even have to be written in the same programming language. Database access was now mediated through SQL.
The first generation of “databased” systems were re-creations of the systems that had preceded them. Old habits die hard. If you had a tape-based inventory system, there was a good chance you replaced it with a VSAM inventory system, which you eventually replaced with a relational database inventory system. It was going to take a while before “materials management” came into the lexicon.
The other thing that databases brought with them was the idea of metadata and schemas. Metadata [2] for a database is primarily the information about the tables and columns: what are they called? How are they related? What can you put in each field? This existed in a primitive form prior to the database, but came to its fore in the database era.
Integrated Systems
The database systems were creating silos. For the most part, each application had its own database. This was causing a huge amount of “interfacing” (writing code to move data from one database to another). In the 80’s, people started seeing the potential for integrating more functions through shared data. Inventory and Purchasing could share product data. Accounts Payable and Purchasing could share vendor data.
Initially companies added functionality (application programs) to existing systems, and (if they were lucky) achieved integration incrementally. This was possible to do if you had written both the applications and databases, you had documented knowledge of the systems, and you had skilled programmers who could make the changes in a predictable amount of time.
Eventually the software vendors got into this business. They discovered that the more functions they could bundle onto one database, the more RFPs they could win.
ERP System
The logical extension of this thought process was the ERP (Enterprise Resource Planning) system. “If we could just get all our enterprise data into one integrated system, it would be the holy grail. We would eliminate all interfacing.” This sounded so good.
The problem was, with the technology of the time, covering all functionality for all types of industries required a huge amount of complexity. The average SAP (the premier ERP vendor) installation has 90,000 tables. There are over 1 million attributes (columns) on all those tables.
Despite the complexity, some companies did get most of their functionality on a single instance and achieved many of the touted benefits. Most did not; and the bigger the company, the less likely the success. Not only did many companies have many existing systems which they could not and did not port to the ERP platform, in a bizarre twist, a majority of large enterprises ended up with multiple ERP systems, most of which were not integrated with each other. Let that sink in.
The necessity of keeping track of millions of attributes was the mother of the invention of the metadata repository. A metadata repository is a database of (hopefully) all the attribute and field names in your various systems.
The Data Warehouse
Somewhere along the way, companies recognized that neither their database applications, nor even their ERP applications, were delivering an integrated set of data for their entire enterprise. No amount of interfacing would address this, because any one system only had a subset of the total data.
Thus, the data warehouse was born. The essential premise was: rather than integrating the operational systems (those that actually run the day-to-day business), let’s integrate the reporting databases. So people made copies of the data in all their operational databases, and put them in a data warehouse.
Well it wasn’t quite that easy. Each of those databased or integrated or ERP systems had its own schema. They were arbitrarily designed, generally as the most expedient way to solve the immediate problem at hand. Generally, the language used was that of the sponsoring department. If you bought a commercial off-the-shelf package, the schema often had no relation to your particular business, and certainly had no relationship to any system you’d implemented.
The real work of the data warehouse was in what would eventually become the ETL industry (“Extract, Transform, and Load,” referring to the three processing steps needed to “on board” a new application to the data warehouse). In the extract step you wrote some code that queried the application database and retrieved the data you were interested in having in the data warehouse. Later, tools were used to do essentially the same thing. The transform stage was how you converted the local language of the application to the shared language of the data warehouse. This occurred at two levels, metadata and data. At the metadata level you had to recognize that the “pid” column in one table actually corresponds to what the data warehouse has called the “patientID.” At the data level you had to realize that “01” in a row in the employee table means “M” in the data warehouse (which really means the gender is “Male”). And finally, in the load stage you had to efficiently load the transformed data into the data warehouse.
The data warehouse was not a transactional system. It was not designed to be updated in place; it was uploaded in batch via the “load” process. It was tuned for fast analytic queries, typically aggregates (rollup all the sales by region, product, promotion, month).
At last we had all of our data in one place—well sort of. In fact, it was just that small subset that we deemed to be of interest to the analysis. Generally, data warehouses were successful, and for quite a long time. Eventually, though, as we will see, the cost of the ETL process eventually caused them to lose favor.
The Operational Data Store (ODS)
The success of the data warehouse, and the general frustration with interfacing the growing number of application systems (which by now had also become the multi-billion soon to be multi-hundred-billion dollar “systems integration” industry), led to the creation of something often called the “Operational Data Store”. The idea was to create a data warehouse that would be used for the purpose of making the integration process easier, by making it into a hub-and-spoke model.
Hub-and-spoke said: put any data that might be shared between systems in a shared model, and then let anyone else access or subscribe to changes. Sounded great. We know of many cases where it didn’t work and no cases where it did work. The main cause of it not working was because the model was too complex—almost as complex as the union of all the schemas it was trying to integrate. It became a bottleneck for change. This bottleneck was exacerbated, because, inevitably, if one system changed it forced a change on the operational data store, which in turn forced all its subscribers to change.
Master Data Management
Somewhere along the line, someone noticed (perhaps they harkened back to the reel-to-reel days) that there were two kinds of data that are, by now, mixed together in the databased application: transactional data and master data. The master data was data about entities, such as Customers, Vendors, Equipment, Fixed Assets, or Products. This master data was often replicated widely. For instance, every order entry system has to have yet another Customer table because of integrity constraints, if nothing else.
If you could just get all the master data in one place, you’d have made some headway. In practice, it rarely happened. Why? In the first place, it’s pretty hard. Most of the MDM packages are still using older, brittle technology, which makes it difficult to keep up with the many and various end-points to be connected. Secondly, it only partially solved the problem, as each system still had to maintain a copy of the data, if for nothing else, for their data integrity constraints. Finally, it only gave a partial solution to the use cases that justified it. For example, the 360° view of the customer was a classic justification, but people didn’t want a 360° view of the master data; they wanted to see the transaction data. Our observation is that most companies that had the intention to implement several MDMs gave up after about 1 ½ years when they found out they weren’t getting the payout they expected.
Canonical Message Model
Service Oriented Architecture (SOA) was created to address the dis-economy in the system integration space. Instead of point-to-point interfacing, you could send transactional updates onto a bus (the Enterprise Service Bus), and allow rules on the bus to distribute the updates to where they are needed.
The plumbing of SOA works great. It’s mostly about managing messages and queues and making sure messages don’t get lost, even if part of the architecture goes down. But most companies stalled out on their SOA implementations because they had not fully addressed their data issues. Most companies took the APIs that each of their applications “published” and then put them on the bus as messages. This essentially required all the other end-points to understand each other. This was point-to-point interfacing over a bus. To be sure, it is an improvement, but not as much as was expected.
Enter the Canonical Message Model. This is a little-known approach that generally works well, where we’ve seen it applied. The basic concept is to create an elegant [3] model of the data that is to be shared. The trick is in the elegance. If you can build a simple model that captures the distinctions that need to be communicated, there are tools that will help you build shared messages that are derived from the simple model. Having a truly shared message is what gets one out of the point-to-point trap. Each application “talks” through messages to the shared model (which is only instantiated “in motion,” so the ODS problem versioning is much easier to solve), which in turn “talks” to the receiving application.
RESTful Endpoints
Roy Fielding’s PhD dissertation [4] reverse engineered the principles underlying the incredible success of the World Wide Web. We take it for granted that we now have access to a network that has billions of attached nodes and handles a zettabyte [5] of information that can be retrieved by any of them that has the access rights. One of the key principles he described is REpresentation State Transfer (REST), which is a style of interaction that focuses on the “Resources”, rather than APIs. Most APIs are very procedural (e.g., “getCustomerAddress”, “postBalanceToAccount”), whereas RESTful interfaces are data centric (“/Customers” will give you a list of customers and “/Customer/<custid>” will get you everything we know about that customer).
This is a great architectural development, and all that’s really needed to make a RESTful architecture data centric is to focus on the definition and modeling of the resources.
Big Data
Many people think of a very large (say multi-petabyte) data warehouse as “Big Data.” It is a lot of data, but it isn’t what Big Data practitioners mean when they say “Big Data.”
Big Data is more about an approach to dealing with data. The approach generally co-evolved with Map Reduce and Hadoop, and generally involves massively replicating small code emissaries, sending them off to distributed data stores, and then coordinating the results on their way back.
Companies have been able to handle incredible scale with the Big Data approach. But in my opinion, it’s the stylistic approach that makes it Big Data, not the amount of data.
The Big Data approach is easiest with huge amounts of similarly named and structured data. Examples of homogenous data include clickstream analysis, Twitter feeds, and query logs.
Data Scientists [6] have taken to the Big Data approach. And because of the nature of their tasks, they are often dealing with many heterogeneous data sets rather than the homogenous set on which Big Data originally cut its teeth. This is what led Gartner to coin the three Vs of Big Data: volume (the amount of data), velocity (really latency, as many Big Data algorithms will not give you real time information), and variety (it’s not really homogenous most of the time).
The variety piece is catching up to us. Several studies [7] have suggested that data scientists are spending 50–80% of their time hand-crafting the collecting, understanding, and organizing of the data they are dealing with.
The Data Lake
One of the current data fads is the “Data Lake.” It is generally motivated by either the cost and hand work involved in onboarding new data sources (in most mature environments it takes 3–6 months to bring a new data set into the data warehouse), or the latency (most data warehouses still work on a batch cycle).
The basic premise of the data lake is to skip most of the hand work in the ETL process and just lay the source data down in the “lake” pretty close to its native format. This also has the added benefit of not introducing errors in the extract and transform processes.
From there, data scientists (or business analysts who would like to believe they are data scientists) can dip into the lake and get “insights.” In other words, they can spend 50–80% of their time trying to interpret what has been laid down in the lake in hopes of finding a few nuggets worthy of justifying their time investment.
The good news about the Data Lake fad is that soon everyone will have a data lake and it won’t seem like a foreign concept that has to be sold to management. The bad news is that, in many organizations, it will get a bad name.
However, it offers a platform that, with a bit of work, has a lot of promise for the data-centric approach. Just a minimal bit of alignment on the way in will save incredible amounts of ad-hoc schema creation. Having a prebuilt starting point will also save a great deal of time. And for the time that does get invested in “research” (plumbing the depths of the data lake), we can create a data lake repository so that others following in the footsteps need not retrace all the steps.
The Evolution of the Data-Centric Revolution
I rehashed the history of the computer industry in order to point out that, almost since the beginning, there has been a latent desire to organize our systems around the data. The history of any industry is a series of advances, lurches, side steps, and retracing as we work through things that would have worked but were ahead of their time, and things that did work but ran into essential limits.
Many people over a long period of time have believed that an approach that puts data front-and-center will ultimately lead to much more satisfying results. We’re going to start with all the lessons learned, and the technology that is now available, and explore in this column what is now possible and what needs to happen to open up this possibility.
If you’re interested in this, by all means, sign the Data-Centric Manifesto at datacentricmanifesto.org.
Notes
[1] It’s hard to imagine a computer industry without disk drives, but it existed that way for quite some time.
[2] The prevailing definition of metadata is “data about other data,” which is hugely unsatisfying. See TechTarget’s definition.
[3] I’m using the 6th definition of elegant from Dictionary.com: “gracefully concise and simple; admirably succinct.”
[4] As PhD dissertations go, this is very readable: Roy Fielding’s dissertation.
[5] Cisco estimated we would hit a zettabyte in 2016: The Telegraph.
[6] Data science on Wikipedia.
[7] For Big-Data Scientists, “Janitor Work” Is Key Hurdle to Insights, The New York Times.
First published on TDAN.com in two parts, in March and June 2016.