There seems to be a divide between Big Data sources when it comes to trust.  Big Data efforts based on internal sources (e.g. log data, transactions, etc.) are trusted.  But apparently Big Data efforts based on external sources in the form of social media etc. is not.  See this IBM study for more details.

But this may be due to one of two things.  The first is simple experience with the external data. Yes it comes from the wild and wholly world of the internet but like the internal data that does not mean there isn't gold in there. 

Which leads to the second point - technique.  There is a wild array of techniques you can use to analyze your Big Data.  Finding the right size, shape and analytical technique in order to find that gold is not easy.  It takes trial and error and great skill in interpreting the results.  

Like all marketing efforts, Big Data is now suffering from "the gloss over effect".  We are given the impression that Big Data value is as simple as hooking a pipe of Big Data into a magic analytical box and out pops the gold you were told is hiding there.  Sorry - not that easy.  

Which is where the Predicate Free expression comes in.  Big Data is sometimes associated with the idea that you can learn a lot from a system simply by examining it output.  “If there are fumes coming from a tailpipe it must be a car”, or “if you smell food when you are near a factory it must be a bakery” kind of thinking.  The work is Predicate Free meaning you don’t need to be bothered by hypothesis, testing and proving.  A system observed over time will always be the same system observed. 

This level of trust can end up being misplaced.  Lots of exogenous shocks to systems change them forever.  Many times those exogenous shocks have never been seen before and so are not accounted for in any Big Data model.  A re-estimate is in order at that point and being Predicate Free trips you up.      

So the realism that sets in after giving your Big Data this initial try might also contribute to the skepticism.  Big Data should be thought of more like a science project - embarking on a journey of discovery.  It takes many dead ends and blind alleys before you reach the promised land.  But it is there.    

Comment