What we love so much about the Big Data effort are two things.  One is that what is old is useful again.  Two that there are some twists and progress along the way.

So what is old?  Some of the techniques are old but they still rock.  Regression is the most powerful analytical technique ever invented.  It forms the basis for all other modeling techniques including machine learning.  Why?  Because of the mathematical foundation on which it is built.  Namely - calculus and the great insight that levels or scalers or measurements don't matter.  Only the way they change matters.  Its the first derivative - remember that?  It's what happens AT THE MARGIN that is really important.  

You hear this everywhere. - in Economics, Finance, Science, Engineering.   What happens to the bridge's structural integrity if you add 1 more pound on top of it?  What happens to the efficacy of a new drug if you one more ml of an ingredient.  And so on.     

And what is new?  Taking the effective cool old stuff and applying it in ways and at a scale no ever imagined.  Huge quantities of input data are just a start.  Then processing that data with the cool old techniques up on self-expanding cloud services to get a job done that less than a generation ago took most of NASA to accomplish.  Now you do it from your desktop and a couple of interfaces.  And the coolest of all?  Letting the machine learn and show that it can do so by finding patterns and making fits of data that would have taken a modeler months of work, if ever.  

So welcome to Big Data.  Maybe it's geeky but they outcomes are cool and getting better all the time.  

Comment