Everybody Lies: The evolution of market research

"Everybody Lies" by Seth Stephens-Davidowitz, is one of the most fascinating books I have read in a long while.

Stephens-Davidowitz documents and evidences in page-turning style a view I have held for some years now:

We can now observe how people actually behave, especially when they don't think anyone is looking, in ways which were previously not possible.
What we observe is often very different from:

what they say they do or will do, and
how they behave when they think someone is looking.

I would guess that Stephens-Davidowitz borrowed the title of his book, whether knowingly or not, from the byline of the TV series "House M.D.". The lead character says "It's a basic truth of the human condition that everybody lies. The only variable is about what."

Reading the book has given me pause to reflect on the evolution of market research. My personal experience suggests at least 3 waves of development.

Market Research 1.0

The first wave of market research consists of asking people for the views, preferences, intentions, wants and needs, etc. This can be:

quantitative, for example, in the form of a survey, or
qualitative, for example, in the form of a focus group, etc.

The obvious problem with this is, of course, that people have many reasons to lie, and few reasons not to. Reasons to lie can be very simple. For example, we may want to appear intelligent, or virtuous, or be liked or admired by the questioner.

Often, people won't even be aware that they are lying. As humans, we are excellent post-rationalisers. Cognitive Dissonance Theory suggests that when faced with a question we can't or don't want to answer, our brains simply fill in the blanks. We make up a story. We may not even be consciously aware of it. (Note: I'm using the word 'lie' here throughout, even when subjects are doing it unintentionally and unknowingly.)

Another problem is that people find it difficult to answer questions about subjects outside of their existing frames of reference. For this reason, market research 1.0 is even less helpful when developing novel ideas. As Henry Ford apocryphally said: "If I had asked people what they wanted, they would have said a faster horse."

Market Research 2.0

Market Research 2.0 attempts to build on Market Research 1.0 by showing customers examples of what future products or services might look like. Often, more than one version is shown. Subjects may be asked to interact with them, compare them and indicate their preferences.

This can go a long way to alleviate subjects inability to imagine a different future. And if all options are attractive and presented positively, this will also reduce some of their incentive to lie.

Market Research 2.0 requires more work than Market Research 1.0. It usually means that you first need to develop some ideas to test. If you're innovative in the development of those ideas, that helps. But the innovation is likely coming from the development of the ideas, rather than from the market research.

However, a number of examples illustrate the difficulties still inherent in the approach:

Subjects reportedly overwhelmingly rejected the idea of ever withdrawing cash from a hole in the wall, as opposed to from a bank teller. But today, 94% of UK adults use cash machines.
In market research, 68% of US customers said they liked the taste of New Coke. But 6 months after launch it was removed from the shelves, and the old formula relaunched. (Reference)
Research conducted between the announcement and launch of the iPhone found high demand in emerging economies like Mexico and India, but not in developed countries. It concluded that: “There is no real need for a convergent product in the US, Germany and Japan”. (Reference)

As I write, I can think of at least two factors which might contribute to this problem. I am sure there are countless more:

The Hawthorne Effect (also known as the Observer Effect). This is named after experiments conducted from 1924-32 in which it was shown that subjects' behaviour is altered by virtue of the fact that they know they are being observed.
Research subjects typically have no 'skin in the game'. For example, it is a lot easier to say you'd be happy to pay, say, £100 for an item than it is to forego the other enjoyments you'd have to give up in order to do so. This is probably exacerbated where they are positively incentivised to take part in the study.

Market Research 3.0

Market Research 3.0 observes:

how real prospects and customers behave with and use products and services,
in the normal course of their lives, and
when they don't think they are being watched.

As technology evolves it is increasingly possible to track:

how customers move through a store,
what items they buy, and
how they engage with and dispose of those products.

This involves developing and launching a product before market testing it. But with increasing software content in products and services (think the Internet of Things) and advances in technologies such as 3D printing, it is become ever cheaper to develop and pilot smaller batches of products or to mass-customise products and services.

Sample Application

I use these techniques to great effect in the development of StratNavApp.com in three ways. It is important to stress, however, that all three techniques are based on statistical analysis. They don't involve anyone ever looking at users' data or strategies. And they don't attributing the results to any specific individuals or companies.

Here are some common ways we measure and

Website analytics: using even a simple (and free) tool like Google Analytics, it is possible to understand

how users find the service,
which parts of it they visit most frequently and in what order,
how long they engage, and
from where they leave.

Using this insight, we can prioritise our development efforts to those areas and features users find most valuable. So, for example, we know that our SWOT analysis tool has been 24% more popular than our Strategy Canvas tool and 33% more popular than our Business Model Canvas tool (confirming our views on the continuing popularity of the SWOT).

AB Testing: almost all new features are first introduced to a randomly selected subset of users (the "A" group"). At the same time, the remaining users (the "B" group) continue to see the site unaltered. We can then measure whether the A group engages more positively (against our own defined Critical Success Factors) than the B group or not. If they do, then the feature is released to the remaining users. And if they do not, then the new feature is rolled back or adjusted and retested. Either way, the results are analysed to enhance our picture of how users use the service, and how we can further improve it.

By way of a very simple example, StratNavApp.com's byline "Collaborative strategy development and execution" was the winner from among a number of AB Tested alternatives considered.
Content Analysis: StratNavApp.com provides a unique insight into how users develop and execute strategies. By way of a very simple example, we know that the word 'Market' is used almost twice as frequently as the word 'Customer' when describing strategic insights. We may not know why that is. And we may not know if it is a good thing or not. But we can certainly use it to enhance our product. We can analyse word counts, numbers and lengths, etc. of all elements used in StratNavApp.com with a view to optimising users' experiences of the tool.

Privacy and Ethics

This is not intended to be a post on privacy and ethics. However, it goes without saying that privacy and ethics have always been a key consideration in market research. And it is right that there is ongoing debate and development of this subject as it evolves.

Conclusion

We've always known that market research is both invaluable and limited. As new technologies evolve, we are able to increase the value it adds whilst simultaneously reducing its limitations. Those organisations that explore and utilise these new approaches will be at a distinct advantage over those that do not.