Is NoSQL the future for plant historians?

​My colleague Bob Mick recently started an email thread here at ARC:  "I've been wondering if I could use MongoDB to build a plant historian?"  Bob's created a few historians in his time - most likely, Bob knows infinitely more about historians than I'll ever know.  So I urged him to come up with an answer to his rhetorical question for two reasons.  First, it's a great question.  Second, I could score an easy blog by building on his wisdom….

So at the heart of the question is this:  What are some of the essential qualities of a plant historian?

• High write throughput.  Historians need to be able to continuously record multiple data samples in real-time from a manufacturing operation. 
• Variety of inputs and data types.  Typically, these would include, flow rate, pressure, temperature, and feedback from values and pumps.  Inputs may be analogue, digital, or text.
• Reliable long-term storage.  In addition to helping operators improve process performance, a company's historian's may be required to help address legal questions.  Reliably storing data captured over many years or decades - and in regulated environments - is necessary.
• Support reporting and analysis operations.  Often, simple reports may be all that is required.  For example, "What was the temperature of a particular vat during a specific time range?"  Other analysis may be more complicated.  For example, understanding why the temperature of a vat was too high may require more detailed analysis.

On the face of it, there's nothing in that list that couldn't be achieved with a relational database (RDBMS).    And yet, process historians - which really started to become prominent in the 1980's - didn't use them. 

Performance was borderline…

Truth is, back in those days relational databases would have struggled to keep up with the input rates cost effectively.  The Transaction Processing Council TPC-B benchmark is probably the closest industry standard database benchmark to the type of workload that a plant historian typically had to manage - although not a particularly close match.  The TPC-B benchmark test consists of a single transaction type which is intended to mimic a bank credit/debit transaction.  A simple, repetitive workload.  (Oddly, it's hard to find any published benchmark results, even though there was a time when a new press release raising the bar appeared almost weekly.  It's almost as though TPC-B has been banished because it's no longer in favor…. ). 

However, according to the TPC, the first TPC-B benchmark result published in mid-1991 achieved 102.94 transactions per second (tps), with a cost of $4,167 per tps.  To be fair, the database structure and schema would have been different to implement a plant historian.  But, a throughput of one hundred transactions per second is inadequate to capture real-time data from a manufacturing plant of any significance.  In this respect, RDBMS performance is also hampered by the need to be ACID compliant.  While important for the role of databases at that time (e.g. in banking applications), such strict rules on database consistency often restricted performance because of the locking mechanism required to implement it.

So performance was iffy, what else…?

Adding to the RDBMS woes, historians are built to capture time-series data.  This brings a number of unique challenges that don't necessarily sit well with something as general purpose and flexible as a relational database:

• The workload is almost all writes, with few reads, or updates.  Most historians (but not all) are big circular buffers.  Once a limit is reached, the oldest data is deleted when new data is added.
• Writes usually arrive in time-order, so write operations are straight-forward appends. 
• The reads that do occur are large and typically time-bounded and ordered.

Finally, historians also had such a large volume of data to capture and store that compression techniques were used.  Many analytic RDBMS's offer data compression now, but it wasn't generally an option thirty years ago.  Oracle, for example, made compression an option with Oracle9i in 2001.  Even then, applying compression to a database table actually reduced write performance - not desirable for a historian.  Plant historians avoided this to a large degree by compressing data using unique data compression algorithms.

So why NoSQL…?

So why might NoSQL databases be suitable as the foundation for a plant historian when relational databases were not?  There are a few reasons - but higher performance is definitely one of them. 

MongoDB, for example, has been designed from the ground up as a distributed database.  Most relational databases were designed in the day when enterprise IT revolved around large monolithic servers.  Consequently, there were inherent limits to scalability - and "solutions" that allowed those databases to work in a distributed computing environment never scaled anywhere close to linearly for most applications.  The database architecture just wasn't designed with that in mind.  Since NoSQL databases have largely been designed for distributed environments, performance can scale well across several nodes for many applications.   Consequently, much higher transaction rates can be handled.

Something that helps make the distributed approach scale so well is that NoSQL databases take a less-than-strict approach to ACID compliance.  ACID compliance is necessary for some applications.  For example, it's not acceptable for a financial transaction ever to be in-doubt, inconclusive, or somehow leave the database in an inconsistent or unknown state.  But, as noted earlier, there is a performance price to be paid for such robustness.  For some applications, however, strict ACID compliance isn't necessary - many NoSQL databases implement "eventual consistency".  Essentially, at some point in time all nodes will have a consistent view of the data.  For many NoSQL applications - capturing and analyzing web traffic, for example - eventually consistency is fine.  It's a trade-off worth making for the high performance (and low cost) that can be achieved.  Maybe that's the case for plant historians too. 

Many NoSQL vendors - such as DataStax and Basho - are talking up their time-series capabilities.  For its part, MongoDB uses its own database technology to collect data on its own cloud database service.  As the company noted, "The system processes over 5B events per day, and over 75,000 writes per second, all on less than 10 physical servers for the MongoDB tier."  Does that make it performant enough to be the core of a plant historian?  Probably, but there are other aspects to consider too - such as monitoring and reporting - that we'll touch on in a future blog.  But, for now, I'd love to hear your thoughts on this:  Are NoSQL databases going to be at the core of next generation plant historians?