Rethinking Big Data and Personalisation

I recently came across the claim that Data has become the basis for competitive advantage. Like the author of that article, that claim got me wondering if it was true (spoiler: yes, I think it is) and if so, what we should do about it.

To understand the import of this claim, we have to cast our minds way back into history.

In the agrarian age, the basis for competition was land. Feudal lords warred with each other to expand their territories. They profited by taxing the local populations. They forced them into military service to gain control of yet greater territories. In the industrial age, the basis for competition was resources. Nations and industrialists fought to control resources like gold, oil, coal and steel. They competed in terms of engineering ingenuity and patent protection.

But now in the information age, the basis for competition is information. Nations and businesses compete by gathering and processing information. Physical warfare and theft are being replaced by cyber-warfare, misinformation campaigns, hacking and identity theft. Of course, the agrarian, and industrial models continue in parallel in the background. But they are of relatively less importance as time goes by.

But what does this mean for businesses and competition?

Big Data and Machine Learning

The first implication is for big data and machine learning. In the early days of the information age, data described industrial concepts like stock units, accounts and transactions. These were relatively simple data constructs, and the volume of data was relatively low. The explosion of new devices of greater variety (think of the Internet of Things) means we're now collecting much greater volumes of data describing a much greater variety of real-world phenomena.

This so-called 'big data' is not only more voluminous but also less structured. This is where machine learning comes in. Big data is too vast and complex for humans to analyse and understand. So we're turning to computer algorithms to do it for us. Knowledge and understanding are being externalised. As humans, we see the results of machine learning, but we don't necessarily understand the thinking behind it.

The combination of big data and machine learning allows us to develop rich pictures of people and other real-world phenomena.

(See also: More data usually beats better algorithms)

Personalisation

Personalisation, in the sense of user experience design, is being presented as something new. But in some senses, it is anything but that. In the 'good old days' local craftspeople and traders serving local communities knew their customers, often personally. We lost that personal connection in business, when, in pursuit of the benefits of scale, we centralised customer engagement into call centres. Customers typically speak to a different agent each time they call. And the information available to the agent is very transactional. Things got even worse when, in pursuit of even greater benefits of scale, we moved everything online. Customer interactions were further simplified down to menu choices and button clicks.

Big data and machine learning offer an opportunity to re-introduce personalisation at scale. I am not suggesting we'll have computer systems that remember to ask me how my children are. But personalisation can be used to ensure the content, promotions and user experience I am exposed to is tailored to my specific requirements and circumstances.

A good way to explain personalisation is to offer examples of what it is not:

1. Every time I log on to my bank I presented with the same information about ringfencing, security and identity theft. It does not matter how frequently I see it, or what I do about it, the information is always the same.
2. My investment manager sends me a fortnightly newsletter full of insights which have nothing to do with my approach to investments. I consistently ignore them, but still, they keep coming.

As a result, I have become 'blind' to messages from my bank and investment manager - I simply don't bother to look at them. It is annoying for me, and a wasted opportunity for them.

Skills

Big data and machine learning mean that organisations need employees and leaders with different skillsets. In addition to engineers and accountants, organisations need data scientists and machine learning experts.

In these new fields, standards of competence are not yet clearly defined. Demand for skilled resources outstrips supply.

Impacts on Trust

The recent Facebook / Cambridge Analytica scandal has highlighted a number of problems in this new world:

1. Facebook allowed Cambridge Analytica to gain access to users' private data as a result of 'backdoors' in their algorithms. Facebook supposedly knew about these backdoors. But it had not prioritised fixing them.
2. Cambridge Analytica then tried to use that data for nefarious purposes. (I have heard quite a lot of doubt expressed as to whether they actually had any impact on the US presidential elections or the UK Brexit referendum.)
3. Facebook was not open and honest about the breach. It took an investigative journalist and a whistleblower to bring it out. Under the European GDPR, which comes into effect on 25 May 2018, Facebook would have been obliged to let every impacted user know, or face substantial fines. As it was, Facebook broke customers' trust, but not the law.
4. Once people started to become aware of the issue, they started looking at exactly what data Facebook had stored about them. They were surprised by just how voluminous and detailed it was. People have subsequently had similar revelations with Google. This is causing people to think more carefully about the value exchange in which they trade their personal data for free services.

My personal view is that people will continue to be happy to trade their data for services. But only if:

1. The data is being used in a way that they perceive to be of benefit to them. For many people, personalisation of advertising does not yet seem like a benefit, but that could change. And, of course, there are many other more direct benefits that could be delivered.
   AND
2. They trust the companies to look after their personal data and keep it safe.

The rise of the Personal Information Managers

A whole new category of services is arising in response to the challenges outlined above: the Personal Information Manager (PIM). A PIM acts as a single source of truth for an individual's data. It gives them control over which other organisations can access their personal data, for what purposes, and over what time periods. They offer organisations GDPR compliance. And they offer customers greater control.

Port.IM is an example of a PIM.

PIM's are a relatively new and not yet completely established class of application, which could play an increasingly important role as people grapple with how they want to manage their personal information.

Conclusions

Many of the biggest and fastest growing businesses of recent time - Facebook, Google and Amazon, for example - succeed largely on the basis of their ability to collect personal data and extract commercial advantages from it. More traditional organisations have, up until now, been able to continue to rely on the strength of their products and services to survive.

However, as more and more traditional competitors start to adopt these practices, those that don't risk being left behind.

Organisations should:

1. Take stock of the information they do and don't (but should) collect about customers and distributors.
2. Look for opportunities to experiment with big data, machine learning and personalisation to extract commercial value from data.
3. Ensure that the security of personal data is of paramount importance.
4. Work to ensure that customers understand exactly what data organisations collect about them and why it is in their interest to allow them to do so.
5. Explore relationships with PIM providers, rather than trying to solve all of the problems themselves.

photo credit: Christoph Scholz IT-Sicherheit Schloss mit binary Hintergrund digital via photopin (license)