Saturday, 26 August 2017

Everybody Lies: The evolution of market research


I've only just started reading "Everybody Lies" by Seth Stephens-Davidowitz (see right), but already I am hooked.

Stephens-Davidowitz documents and evidences in page-turning style a view I have held for some years now:
  1. we can now observe how people actually behave, especially when they don't think anyone is looking, in ways which were previously not possible, and
  2. what we observe it is often very different from 
    1. what they say they do or will do, and
    2. how they behave when they think someone is looking.
I would guess that Stephens-Davidowitz borrowed the title of his book, whether knowingly or not, from the byline of the TV series "House M.D.", whose lead character says "It's a basic truth of the human condition that everybody lies. The only variable is about what."

Reading the book has given me pause to reflect on the evolution of market research. Without having actually done a comprehensive historical study of the subject, my personal experience suggests at least 3 waves of development.

Market Research 1.0

The first wave of market research consists of asking people for the views, preferences, intentions, wants and needs, etc. This can be quantitative, for example, in the form of a survey, or qualitative, for example, in the form of a focus group, etc.

The obvious problem with this is, of course, that people have many reasons to lie, and few reasons not to. Reasons to lie can be very simple, for example, wanting to appear intelligent, or virtuous, wanting to be liked or admired by the questioner. Often, people won't even be aware that they are lying. As humans, we are excellent post-rationalisers. Cognitive Dissonance Theory suggests that when faced with a question we can't or don't want to answer, in a situation where we'd like to, our brains simply fill in the blanks, making up a story, possibly without even being consciously aware of it. (Note, we're using the word 'lie' here throughout, even when subjects are doing it unintentionally and unknowingly.)

Another problem is that people find it difficult to answer questions about subjects outside of their existing frames of reference. For this reason, market research 1.0 is even less helpful when developing novel ideas. As Henry Ford apocryphally said: "If I had asked people what they wanted, they would have said a faster horse."  

Market Research 2.0

Market Research 2.0 attempts to build on Market Research 1.0 by showing customers examples of what future products or services might look like. Often, more than one version is shown, with subjects asked to interact with them, compare them and indicate preferences.

This can go a long way to alleviate subjects inability to imagine a different future, and if all options are attractive and presented positively, this will also reduce some of their incentive to lie.

Market Research 2.0 requires more work than Market Research 1.0 as it usually means that you first need to develop some ideas to test. If you're innovative in the development of those ideas, that helps. But the innovation is likely coming from the development of the ideas, rather than from the market research.

However, a number of examples illustrate the difficulties still inherent in the approach:
  1. Subjects reportedly overwhelmingly rejected the idea of ever withdrawing cash from a hole in the wall, as opposed to from a bank teller, but today, 94% of UK adults use cash machines.
  2. In market research, 68% of US customers said they liked the taste of New Coke, but 6 months after launch it was removed from the shelves, and the old formula relaunched. (Reference)
  3. Research conducted between the announcement and launch of the iPhone found high demand in emerging economies like Mexico and India, but not in developed countries, stating: “There is no real need for a convergent product in the US, Germany and Japan”. (Reference)
As I write, I can think of at least two factors which might contribute to this problem, and I am sure there are countless more:
  1. The Hawthorne Effect (also known as the Observer Effect) is named after experiments conducted from 1924-32 in which it was shown that subjects behaviour is altered by virtue of the fact that they know they are being observed.
  2. Research subjects typically have no 'skin in the game'. For example, it is a lot easier to say you'd be happy to pay, say, £100 for an item than it is to forego the other enjoyments you'd have to give up in order to do so. This is probably exacerbated where they are positively incentivised to take part in the study.


Market Research 3.0

Market Research 3.0 observes how real prospects and customers behave with and use products and services in the normal course of their lives and when they don't think they are being watched.

As technology evolves it is increasingly possible to track how customers move through a store, what items they buy and how they engage with and dispose of those products.

This involves developing and launching a product before market testing it, but with increasing software content in products and services (think the Internet of Things) and advances in technologies such as 3D printing, it is become ever cheaper to develop and pilot smaller batches of products or to mass-customise products and services.

Sample Application

I use these techniques to great effect in the development of StratNavApp.com in three ways. It is important to stress, however, that all three techniques are based on machine statistical analysis and don't involve anyone ever looking at users' data or strategies, or attributing the results to any specific individuals or companies.

Here are some common ways we measure and
  1. Web-site analytics: using even a simple (and free) tool like Google Analytics, it is possible to understand how users find the service, which parts of it they visit most frequently and in what order, how long they engage, and from where they leave. Using this insight, we can prioritise our development efforts to those areas and features users find most valuable. So, for example, we know that our SWOT analysis tool has been 24% more popular than our Strategy Canvas tool and 33% more popular than our Business Model Canvas tool (confirming our views on the continuing popularity of the SWOT).  
  2. AB Testing: almost all new features are first introduced to a randomly selected subset of users (the "A" group") while the remaining users (the "B" group) continue to see the site unaltered. We can then measure whether the A group engages more positively (against our own defined Critical Success Factors) than the B group or not. If they do, then the feature is released to the remaining users, and if they do not, then the new feature is rolled back or adjusted and retested. Either way, the results are analysed to enhance our picture of how users use the service, and how we can further improve it. By way of a very simple example, StratNavApp.com's byline "Collaborative strategy development and execution" was the winner from among a number of AB Tested alternatives considered.
  3. Content Analysis: StratNavApp.com provides a unique insight into how users develop and execute strategies. By way of a very simple example, we know that the word 'Market' is used almost twice as frequently as the word 'Customer' when describing strategic insights. We may not know why that is, and we may not know if it is a good thing or not, but we can certainly use it to enhance our product. We can analyse word counts, numbers and lengths, etc. of all elements used in StratNavApp.com with a view to optimising users' experiences of the tool.

Privacy and Ethics

This is not intended to be a post on privacy and ethics. However, it goes without saying that privacy and ethics have always been a key consideration in market research and it is right that there is ongoing debate and development of this subject as it evolves.

Conclusion

We've always known that market research is both invaluable and limited. As new technologies evolve we are able to increase the value it adds whilst simultaneously reducing its limitations. Those organisations that explore and utilise these new approaches will be at a distinct advantage over those that do not.