Two things really bug me about the big data discussion at the moment. First, many people seem to equate the "big" in big data with volume. In other words, you have to have big volume to have big data. Not so.
Doug Laney defined 3 V's, not 1 V, for a reason. Sure, people struggle to manage accelerating growth of data volume. But, they also struggle to cope with data that is coming at them faster than ever (velocity), or from more sources than before (variety). Big data isn't a thing, it's a journey. And if you're finding it tough to cope with any one of the V's, you're somewhere on that journey. Maybe not very far along in your journey, but on it none the less...
Second, the view that Hadoop is the be all and end all or big data. Again, not so. Must you use Hadoop to deal with higher velocity data..? Not at all. It may be part of a solution, but maybe you just need to enhance your existing RDBMS solution, or look at complex event processing/event streaming solutions instead. Or, if you do have a large data volume to manage, why not consider a NoSQL database instead of Hadoop (see popular choices at DB Engines by clicking on the chart below).
(Note, Hadoop isn't covered by the DB Engines rankings, because it isn't actually a database. Shouldn't the very idea of trying to manage large data volumes without a database at least give pause for thought...?)
And so in the last couple of weeks, it seems Hadoop has reached a bit of a crossroads.
Gartner recently poured a fair amount of cold water on the value currently being realized by Hadoop users. On the other hand,
InformationWeek speculated that 2015 may actually be the year of Hadoop. As a research analyst and marketer, my days of coding for a living are long gone. So I've never laid hands on Hadoop - nor do I really care to. But, aside from its technical merits and/or challenges, Hadoop has a serious problem. It's a marketing problem.
Only the seriously geeky understood Hadoop three years ago, and I would bet money that hasn't changed very much today. Consider the marketecture diagrams from a couple of the commercial independent Hadoop distributions.
This is what MapR's looks like (click to enlarge):
And here's Hortonworks (click to enlarge):
Clear as mud right...? No problem figuring out what you get, how you use it, and what business problem it solves..?
I've read that
Hadoop is on the verge of mainstream adoption. I don't believe that's possible anytime soon, simply because the marketing positioning and messaging hasn't matured yet. The level of technical marketing shown above isn't going to get Hadoop across
Geoffrey Moore's infamous chasm. It's more likely to get Hadoop a faceplant at the bottom. The current marketing is for a small minority of visionary's, it's not going to get you to the "pragmatics" (the larger market). Pragmatics need well formed use cases, proven solutions and a reasonable stab at a future ROI. Right now, Hadoop looks a bit too much like an Ikea project - only without the instructions - for it to have truly mainstream appeal.