Documents, Events and Actions
We have recently been reexamining the weird relationship of “documents” to “events” in enterprise information systems and have surfaced some new insights that are worth sharing.
Documents and Events
Just to make sure we are all seeing things clearly, the documents we’re referring to are those that give rise to financial change in an enterprise. This includes invoices, purchase orders, receiving reports and sales contracts. We’re not including other documents like memos, reports, news articles and emails – nor are we focusing on document structures such as JSON or XML.
In this context, the “events” represent the recording of something happening that has a high probability of affecting the finances of the firm. Many people call these “transactions” or “financial transactions.” The deeper we investigated, the more we found a need to distinguish the “event” (which is occurring in the real world) from the “transaction” (which is its reflection in the database). But I’m getting ahead of myself and will just stick with documents and events for this article.
Documents and Events, Historically
For most of recorded history, the document was the event, or at least it was the only tangibly recorded interpretation of the event. That piece of actual paper was both the document and the representation of the event. When you wrote up a purchase order (and had it signed by the other party) you had an event.
In the 1950’s we began computerizing these documents, turning them into a skeuomorph (a design that imitates a real-world object to make it more familiar). The user interfaces looked like paper forms. There were boxes on the top for “ship to” and “bill to” and small boxes in the middle for things like “payment terms,” and “free on board.” This was accompanied by a line item of the components that made up the bill, invoice, purchase order, timecard, etc.
For the longest time, the paper was also the “source document” which would be entered into the computer at the home office. Somewhere along the way some clever person realized you could start by entering the data into the computer for things you originated and then print out the paper. That paper was then sent to the other party for them to key it into their system.
Now, most of these “events” are not produced by humans, but by some other computer program. These ‘bills of materials’ processors can generate purchase orders much faster than a room full of procurement specialists. Many industries now consider these “events” to be primary. The documents (if they exist at all) are part of the audit trail. Industries like healthcare have long ago replaced the “superbill” (a document on a clipboard with 3 dozen check boxes to represent what the physician did to you on that visit) with 80 specific types of HL7 messages that ricochet back and forth from provider to payer.
And yet, even in the 21st century, we still find ourselves often excerpting facts from unstructured documents and entering them into our computer systems. Here at Semantic Arts, we take the contracts we’ve signed with our clients and scan them for the tidbits that we need to put into our systems (such as the budgets, time frame, staffing and billing rates) and conveniently leave the other 95% of the document in a file somewhere.
Documents and Events, what is the difference?
So for hundreds of years, documents and events were more or less the same thing. Now they have drifted apart. In today’s environment, the real questions are not “what’s the difference” but rather “which one is the truth.” In other words, if there is a difference which one do we use? There is not a one-size-fits-all answer to that dilemma. It varies from industry to industry.
But I think it’s fairly safe to say the current difference is that an “event” is a structured data representation of the business activity, while a “document” is the unstructured data representation. Either one could have come first. Each is meant to be the reflection of the other.
The Event and the Transaction
The event has a very active sense to it because it occurs at a specific point in time. And therefore, we record it in our computer system and create a transaction, which updates our database at the posting date and as the effective accounting date.
The transaction and the event often appear to be the same thing, partly because so many events terminate in the accounting department. But, in reality, the transaction is adding information to the event that allows it to be posted. The main information that is being added is the valuation, the classification and the effective dates. Most people enter these at the same time they capture the event, but they are distinct. The distinction is more obvious when you consider events such as “issuing material” to a production order. The issuer doesn’t know what account number should be charged, nor do they know the valuation (this is buried in an accounting policy that determines whether to cost this widget based on the most recent cost, the oldest cost or the average cost of widgets on hand.) So the “transaction” is different from the “event” even if they occur at the same time.
Until fairly recently, administrators wouldn’t sit at their computer and enter invoices until they were prepared for them to be issued. Most people wait until they ship the widget or complete the milestone before they key in the invoice data and email it to their customer. In this circumstance, the event and the transaction are cotemporaneous – they happen at the same time. And the document being sent to the customer follows shortly thereafter.
One More Disconnect
We are implementing data-centric accounting at Semantic Arts and have disconnected the “event” that is the structured data representation of the event, from its classification as an event. We realized that as soon as we had signed a contract, we knew at least one of the two aspects of our future invoices, and in many cases, we knew both. For fixed price projects, we knew the amount of the future invoices. The only thing we didn’t know was when we could invoice them – because that was based on the date of some given milestone. For time and material contracts we know the dates of our future invoices (end of the month often) but don’t know the amount. And for our best efforts contracts we know the dates and the amounts and adjust the scope to fit.
But knowing these things and capturing them in our accounting system creates a problem. They weren’t actually real yet (or at least they weren’t real enough to be invoices). The sad thing was they looked just like invoices. They had all the data, and it was all valid. They could be rendered to pdfs, and even printed, but we knew we couldn’t send all the invoices to our client all at once. So we now had some invoices in our system that weren’t really invoices, and didn’t have a good way to make the distinction.
As we puzzled over this, we came across a university that was dealing with the same challenge. In their case they were implementing “commitment accounting,” which is trying to keep track of the commitments (purchase orders mostly) that are outstanding as a way to prevent overrunning budgets. As people entered their purchase orders (structured records as we’ve been describing them) the system captured them as events. These events were captured and tallied by the system. In order to get the system to work, people entered purchase orders long before they were approved. In fact, you have to enter them to get an event (or a document) that can be approved and agreed to by your vendor.
The problem was many of these purchase order events never were approved. The apparent commitments vastly exceeded the budgets, and the whole system was shut down.
Actions
We discovered that it isn’t the document, and it isn’t even the event (if we think of the event as the structured data record of the business event) that makes the financial effect real. It is something we are now calling the “action,” or really a special type of “action.”
There is a magic moment when an event, or perhaps more accurately a proto-event becomes real. On a website, it is the “buy” button. In the enterprise ,it is often the “approval” button.
As we worked on this, we discovered it is just one of the steps in a workflow. The workflow for a purchase order might start with sourcing, getting quotes, negotiating, etc. The special step that makes the purchase order “real” isn’t even the last step. After the purchase order is accepted by the vendor, we still need to exchange more documents to get shipping notifications, deal with warranties, etc. It is one of those steps that makes the commitment. We are now calling this the “green button.” There is one step, one button in the workflow progression that makes the event real. In our internal systems we’re going to make that one green, so that employees know when they are committing the firm.
Once you have this idea in your head, you’ll be surprised how often it is missed. I go on my bank’s website and work through the process of transferring money. I get a number of red buttons, and with each one, I wonder, “is this the green one.” Nope, one more step before we’re committed. Same with booking a flight. There are lots of purple buttons, but you have to pay a lot of attention before you notice which one of those purple buttons is really the green one.
Promotion
And what does the green button in our internal systems do? Well, it varies a bit, workflow to workflow, but in many cases it just “promotes” a draft item to a committed one.
In a traditional system you would likely have draft items in one table and then copy them over to the approved table. Or you might have a status and just be careful to exclude the unapproved ones from most queries.
But we’ve discovered that many of these events can be thought of as subtypes of their draft versions. When the green button gets pressed in an invoicing workflow, the draft invoice gains another triple, which makes it also an approved or a submitted invoice – in addition to its being a draft invoice.
Summary
We in the enterprise software industry have had a long history of conflating documents and events. Usually we get away with it, but occasionally it bites us.
What we’re discovering now with the looming advent of data-centric accounting is the need not only to distinguish the document from the event but also distinguish the event (as a structure) from the action that enlivens it. We see this as an important step in the further automation of direct financial reporting.
