Friday, 20 January 2012

More data usually beats better algorithms

Every so often I read something which subtly changes my perspective in a  fundamental way. Anand Rajaraman's post More data usually beats better algorithms is one such piece.

It contains two simple ideas:

Firstly, the main thesis is that adding new data to an analysis often beats coming up with a more clever algorythm. He suggests, for example, that by including which pages link to which other pages as additional data, Google was able to beat previous web search algorithms which focused exclusively on the words on the pages themselves.

Secondly, the related 'Rule of Representation' which says that you should 'Fold knowledge into data, so program logic can be stupid and robust.' This second idea is actually presented by someone writing a comment to the original blog entry, but it is still quite pertinent. Although this particular rule comes from the computer programming field, I think it has more general application.

I read this as I was starting to formulate my conclusions on a review I was doing of a client's strategy. Perhaps it struck a chord because the crux of my review had been to use some additional data about the market which had not been considered by the original authors of the strategy I was reviewing, and which led me to some different insights which had not yet been considered. And perhaps it struck a chord because I had been unable, within the confines of the assignment, to find some additional data which I feel would have further significantly enhanced the review.

Of course, it matters greatly what data you add to the analysis. To add value, the additional data must provide a different perspective and/or be from a different source than the data already included. This suggests that perhaps a key skill for the strategist is having a keen sense of what market data is available and where to source it, rather than simply relying on the data that is immediately to hand. (Oftentimes the data to hand relates to the organisation's own operations and performance, and the data that is more crucially missing relates to the competitive environment.)

And finally, I believe that somewhere towards the end of the strategic thinking process you have to look beyond the data and apply a little creative intuition. That is driven by your experience - the sum of data you've collected along the way but which you cannot tabulate in a spreadsheet. In the final analysis, it is that data that may be the most important addition, creating the distinction between also-ran me-too strategies and analysis, and truly differentiated and competitive strategies and analysis.