Monday, 11 February 2013

'Big data' is a big opportunity, not a big error

In an article on Wired entitled Beware the Big Errors of 'Big Data', Nassim Taleb is fairly disparaging of 'big data'.

His main point appears to be that you can do bad research with small amounts of data, and that you can still do bad research whilst sounding more convincing with large amounts of data. (However, he uses much more complicated words than that to make his point sound more profound.)

What he fails to mention is that you can do very good research with small amounts of data, and even better research with large amounts of data. (See, for example, More Data Usually Beats Better Algorithms.)

In doing so, he makes the same mistake that he so often criticises others for making - that is, he presents data which support his preconceptions whilst ignoring data which refute them.

Big data provides organisations with a potential resource. As with all resources, organisations can choose to use it, or leave it alone. If they choose to use it, they can either use it well, or use it poorly. Undoubtedly we will hear many stories in the future of organisations that used 'big data' poorly, and the naysayers like Mr Taleb will jump on these instances as some form of proof that 'big data' is somehow a bad thing. In the meantime, the organisations which have invested in understanding how to use 'big data' successfully will simply ignore them and carry on benefiting from the advantages they achieve.

I believe that big data offers tremendous potential and will give firms that get it right yet one more advantage over those that don't.