Michael I. Jordan, one of the most prominent statisticians of our time, warns that “The overeager adoption of big data is likely to result in catastrophes…“. This comment raises a question: If much of the enthusiasm for big data is indeed mere hype, is the big data movement likely to deliver results in the long–term?

I believe that the big data movement will deliver many significant results, but not because it is a particularly good new idea substantiated by real progress in statistics, machine learning, or useful data collection. I agree that we “… are no further along with computer vision than we were with physics when Isaac Newton sat under his apple tree.”. Much of big data nowadays seems to be about manipulating people for political and mercenary purposes based on simplistic statistical models.

I think that any hype is likely to “succeed” if it attracts enough attention and resources and if it is stated broadly/vaguely enough. Resources and prominence attract great people and they bring many ideas, most of which way beyond the scope and the imagination of the people who started the hype. Some of these ideas are likely to be good and result in significant progress. A decade from now, we will be able to look back and see that many significant methodological developments in statistics were enabled by the big data hype and most likely would not have happened without it. What we would not be able to see are all the other, perhaps much more significant and beneficial, developments that would have happened if the resources and the attention were allocated to other areas.

Yes, mere hype can succeed in the long-term. Attracting resources and attention is the key to the success of any huge initiative, and a hype is perhaps the easiest way to attract them. Huge initiatives attract at least some good people who are likely to come up with at least some good ideas that deliver something useful. If the hype was stated broadly/vaguely enough, these results seem to validate the vision of the people who started the movement. The hype is declared a success.