Big Data holds a lot of promise. Some is current. Some far off. At Big Data Lens we look at the middle distance.
We don't believe slapping the a label on small sets of data using traditional techniques makes it Big Data. I am reminded of this when thinking about a presentation I sat through at a Big Data conference this past summer listening to a big company finance guy crow about his wondrous automatic manipulation of expense spreadsheets. This he called Big Data. It is not. The presenter who followed, however, had it right. His definition stands. Big Data is about doing what could NOT be done before. That means the volume of data processed (in a reasonable amount of time) using techniques never applied before.
So the middle distance is not about processing expense spreadsheets that could have be done with a small python script on 10 minutes work. Nor is Big Data about curing cancer in the short term - though it certainly may play a part in this. in the long term.
The middle distance is about using the whole internet in new ways, to find products that protect soldiers from risk that have not been applied before as we do with the Department of Defense. Or finding technologies to reduce risk and improve planning as we do for the Army. Or even finding and analyzing disparate sources of data for compliance and optimization as we do for a healthcare company.
In the slightly longer middle distance Big Data is about using the internet to understand competitors patterns of behavior and predict their next moves. Or to understand market dynamics and customer groups in ways unknown before to streamline operations. Or to allow social media to inform product design in ways that obviate the need for expensive and costly surveys, research and so on.
The middle distance for Big Data is about being practical. The middle distance for Big Data is about getting the most for this new science without having to wait a long long time. The middle distance is the sweet spot for Big Data.