Dark data and the Pareto principle

Formally, it's the Pareto principle.  That 80/20 rule-of-thumb that we glibly apply to many areas of our professional lives.  20% of our customers generate 80% of the revenue, 20% of healthcare patients account for 80% of the cost, 20% of fixed assets account for 80% of capex etc.  Microsoft found it applied to bugs too.  We've all heard it, many of us have said it. 

Lately though, I've started to hear something similar applied to corporate data as well.  That is, the typical organization only uses 20% of its data.  The other 80% - dubbed "dark" data - doesn't get used for decision making at all.  That might well be true for many organizations.  It feels right, and I don't have either data or intuition to suggest otherwise.  Some analytics vendors have seen this as a marketing opportunity.  Many solutions - Hadoop among them - have been proposed as a way to shine light onto that 80% of dark data. 

There might be value in that, but I'm not convinced.  If we truly believe in the 80/20 rule - which does have some science behind it by the way - then you can't ignore the other half of the rule.  Corporations might only use 20% of their data, but isn't that because that's where 80% of the value is…?  Maybe that other 80% of data has stayed dark for a reason. 

For some, there might be value in tapping that dark data.    Certainly, the device data that some enterprises are starting to generate from the industrial internet of things (IIoT) can have an immediate impact on asset performance management, for example.  But, IIoT use cases aside, I'm not sure it's wise for every company to try and extract value from dark data.  For many companies, the better analytics investment will be in providing more timely, more widespread access to the 20% of data they already use.  Please let me know if you disagree, but I'd bet that’s true for oh, roughly 80% of companies…