Rethinking Big Data and Personalisation

I recently came across the claim that Data has become the basis for competitive advantage. Like the author of that article, that claim got me wondering if it was true (spoiler: yes, I think it is) and if so, what we should do about it.

To understand the import of this claim, we have to cast our minds way back into history.

In the agrarian age, the basis for competition was land. Feudal lords warred with each other to expand their territories. They profited by taxing the local populations. They forced them into military service to gain control of yet greater territories. In the industrial age, the basis for competition was resources. Nations and industrialists fought to control resources like gold, oil, coal and steel. They competed in terms of engineering ingenuity and patent protection.

But now in the information age, the basis for competition is information. Nations and businesses compete by gathering and processing information. Physical warfare and theft are being replaced by cyber-warfare, misinformation campaigns, hacking and identity theft. Of course, the agrarian, and industrial models continue in parallel in the background. But they are of relatively less importance as time goes by.

But what does this mean for businesses and competition?

Big Data and Machine Learning

The first implication is for big data and machine learning. In the early days of the information age, data described industrial concepts like stock units, accounts and transactions. These were relatively simple data constructs, and the volume of data was relatively low. The explosion of new devices of greater variety (think of the Internet of Things) means we're now collecting much greater volumes of data describing a much greater variety of real-world phenomena.

This so-called 'big data' is not only more voluminous but also less structured. This is where machine learning comes in. Big data is too vast and complex for humans to analyse and understand. So we're turning to computer algorithms to do it for us. Knowledge and understanding are being externalised. As humans, we see the results of machine learning, but we don't necessarily understand the thinking behind it.

The combination of big data and machine learning allows us to develop rich pictures of people and other real-world phenomena.

(See also: More data usually beats better algorithms)

Personalisation

Personalisation, in the sense of user experience design, is being presented as something new. But in some senses, it is anything but that. In the 'good old days' local craftspeople and traders serving local communities knew their customers, often personally. We lost that personal connection in business, when, in pursuit of the benefits of scale, we centralised customer engagement into call centres. Customers typically speak to a different agent each time they call. And the information available to the agent is very transactional. Things got even worse when, in pursuit of even greater benefits of scale, we moved everything online. Customer interactions were further simplified down to menu choices and button clicks.

Big data and machine learning offer an opportunity to re-introduce personalisation at scale. I am not suggesting we'll have computer systems that remember to ask me how my children are. But personalisation can be used to ensure the content, promotions and user experience I am exposed to is tailored to my specific requirements and circumstances.

A good way to explain personalisation is to offer examples of what it is not:

1. Every time I log on to my bank I presented with the same information about ringfencing, security and identity theft. It does not matter how frequently I see it, or what I do about it, the information is always the same.
2. My investment manager sends me a fortnightly newsletter full of insights which have nothing to do with my approach to investments. I consistently ignore them, but still, they keep coming.

As a result, I have become 'blind' to messages from my bank and investment manager - I simply don't bother to look at them. It is annoying for me, and a wasted opportunity for them.

Skills

Big data and machine learning mean that organisations need employees and leaders with different skillsets. In addition to engineers and accountants, organisations need data scientists and machine learning experts.

In these new fields, standards of competence are not yet clearly defined. Demand for skilled resources outstrips supply.

Impacts on Trust

The recent Facebook / Cambridge Analytica scandal has highlighted a number of problems in this new world:

1. Facebook allowed Cambridge Analytica to gain access to users' private data as a result of 'backdoors' in their algorithms. Facebook supposedly knew about these backdoors. But it had not prioritised fixing them.
2. Cambridge Analytica then tried to use that data for nefarious purposes. (I have heard quite a lot of doubt expressed as to whether they actually had any impact on the US presidential elections or the UK Brexit referendum.)
3. Facebook was not open and honest about the breach. It took an investigative journalist and a whistleblower to bring it out. Under the European GDPR, which comes into effect on 25 May 2018, Facebook would have been obliged to let every impacted user know, or face substantial fines. As it was, Facebook broke customers' trust, but not the law.
4. Once people started to become aware of the issue, they started looking at exactly what data Facebook had stored about them. They were surprised by just how voluminous and detailed it was. People have subsequently had similar revelations with Google. This is causing people to think more carefully about the value exchange in which they trade their personal data for free services.

My personal view is that people will continue to be happy to trade their data for services. But only if:

1. The data is being used in a way that they perceive to be of benefit to them. For many people, personalisation of advertising does not yet seem like a benefit, but that could change. And, of course, there are many other more direct benefits that could be delivered.
   AND
2. They trust the companies to look after their personal data and keep it safe.

The rise of the Personal Information Managers

A whole new category of services is arising in response to the challenges outlined above: the Personal Information Manager (PIM). A PIM acts as a single source of truth for an individual's data. It gives them control over which other organisations can access their personal data, for what purposes, and over what time periods. They offer organisations GDPR compliance. And they offer customers greater control.

Port.IM is an example of a PIM.

PIM's are a relatively new and not yet completely established class of application, which could play an increasingly important role as people grapple with how they want to manage their personal information.

Conclusions

Many of the biggest and fastest growing businesses of recent time - Facebook, Google and Amazon, for example - succeed largely on the basis of their ability to collect personal data and extract commercial advantages from it. More traditional organisations have, up until now, been able to continue to rely on the strength of their products and services to survive.

However, as more and more traditional competitors start to adopt these practices, those that don't risk being left behind.

Organisations should:

1. Take stock of the information they do and don't (but should) collect about customers and distributors.
2. Look for opportunities to experiment with big data, machine learning and personalisation to extract commercial value from data.
3. Ensure that the security of personal data is of paramount importance.
4. Work to ensure that customers understand exactly what data organisations collect about them and why it is in their interest to allow them to do so.
5. Explore relationships with PIM providers, rather than trying to solve all of the problems themselves.

photo credit: Christoph Scholz IT-Sicherheit Schloss mit binary Hintergrund digital via photopin (license)

The digital spiral towards innovation at the core

The nature of what the digital revolution means is evolving as organisations adopt digital in more mature and fundamental ways and as AI becomes an increasing factor in organisation's digital strategies.

Digital started at the edge of the organisation, where the organisation touches its customers and has evolved ever deeper into the organisation to where it now enables completely new ways of creating and delivering value.

This evolution is a continuum of change. But, I find it helpful to divide it into four distinct phases.

1. E-commerce

The first phase allowed businesses to market and sell their (existing) products and services to more customers more easily over digital channels. It started with marketing. Using informational websites, email and social media to reach customers. It gradually grew to include selling and eventually servicing.

This first wave of e-commerce is often referred to as Web 1.0. A myriad of startups flooded the market selling everything from books to pet food, whilst established pre-e-commerce business struggled to keep up. The inevitable bust which followed the boom left a much smaller number of massive winners (think Amazon.com), a much larger number of failures (think Pets.com). Almost no businesses exist today without a website.

E-commerce had more fundamental impacts on business strategy, though. Most importantly, it exposed how a lack of integration within the organisation and its system made it difficult if not impossible to deliver coherent DIY and self-service solutions to the end customers. From this, the concept of customer-centricity was born. (Although this term has now taken on a life of its own and been widely misappropriated.)

2. Supply chain automation

Whilst the focus of this first phase of e-commerce was on end consumers, the second phase tackled the B2B relationship between corporate buyers and their suppliers. B2B e-commerce sites quickly evolved to provide more direct integration between buyer and supplier systems. Standards, such as XML, evolved to facilitate this, and more recently we've seen a drive towards API-first business models.

Supply chain automation makes existing exchanges of information more efficient. It also increases the flow of information. For example, RFID allows retailers, distributors and manufacturers to track stock levels and movement, automating stock management and better integrating with robotics.

Not only did this improve efficiencies, with smaller orders and faster delivery cycles, but it also allowed previously monolithic organisations to disaggregate into networks of smaller interdependent and more specialised suppliers.

In 2017 McKinsey & Co estimated that whilst 49% of companies invest in E-commerce, only 2% of companies invest in digitalising their supply chain. A more recent 2023 study by PWC, found that 83% of executives said their supply chain technology investments haven't fully delivered expected results and that few said their companies are using or panning to use technology to enhance the execution of their supply chain over the next 24 months. A 2024 study by KPMG, in contrast, found that 50% of supply chain organisations will invest in application that support AI and advanced analytics, and that 2/3rds have already adopted low-code int their supply chains.

3. Workforce automation

With both the customer and supplier ends of the value chain being digitised it was inevitable that digitisation would begin to encroach on the work done by employees in between.

Although we've had expert systems for many years, it is the development of digital customer and supplier interfaces which generate the data required to move them towards true artificial intelligence. For an explanation of this effect, see More data usually beats better algorithms.

The now general release of Generative AI has significantly accelerated this trend.

Information-based jobs will inevitably be hardest hit first. Think of insurance underwriters and claims managers, fund managers, accountants, etc. But inevitably most jobs with any element of repetition will be impacted. Taxi drivers are at risk from autonomous vehicles, for example, and there is already talk of robots performing some surgeries more accurately than skilled surgeons. There is already evidence that banks are hiring fewer people with finance backgrounds to do the work, and more people with technology skills to automate it.

I prefer to think of workforce automation in terms of augmentation, rather than replacement of people. The industrial revolution provides a well-established narrative of technology replacing labour and yet somehow creating new and different jobs in the process. Most of us are grateful that we don't have to do the heavy labour now replaced by machines. And future generations will be grateful not to have to do some of the repetitive and uncreative jobs many people do today. (The emerging millennial workforce is already rejecting much of this kind of work.)

Ultimately, however, we should expect that computers and machines will eventually become better than humans at all tasks.

It strikes me as anachronistic that organisations are slow to adapt and to provide their employees with the same level of tools to use in their jobs as other companies provide them to use as their customers. Ultimately, this flows through into the customer experience when call centre operators are unable to provide quick and definitive solutions, leaving you with the impression they are still switching between multiple disconnected systems to get answers and process requests.

4. Digitisation

The resulting end-to-end digitisation of the value chain opens up possibilities. Not just for delivering existing products and services more effectively. But also for creating entirely new products and services.

Again, this started with information-based products. Think of the development of subscription-based streaming media services, compared to purchased physical media or broadcast services. But it is now moving increasingly into physical products as well.

This is enabled by the Internet of Things (IoT): a network of sometimes semi-autonomous things able to sense elements of the real world and communicate with each other and controlling systems. Already we have devices fitted in cars which can collect data about the performance of the car and can communicate this to service technicians and insurance companies. We also have activity trackers. These measure things like sleep, activity levels and heart rate on a continuous basis. They upload this data to servers for more detailed analysis.

Of course, it is still hard to imagine where this could go. Imagine that your calendar/organiser is able to determine where you are now, where you need to get to for your next meeting, and can arrange for an autonomous vehicle to take you there, all without you needing to do anything.

Note only does digitisation replace or enhance physical products, and create completely new entirely digital products, it also often develops whole new ecosystems of digitally connected organisations, people and devices which collaborate together in previously unimaginable ways.

In 2017 McKinsey and Co research suggested that 70% of companies approach digitisation without changing their overall strategy, but that the 30% that do, generate 3 times the profit. More recently, in 2021, an Infosys MIT Technology Review survey of more than 250 business leaders and senior executives revealed that more than half of the enterprises realised new business models because of participating in the data economy. (Source)

When I look at offices and factories full of people hunched over keyboards, screens and other equipment, I always get the sense that somehow it is the people working to satisfy the requirements of the machines. I read today a prediction that in 10 years time, most interfaces will not have a screen. I believe that is because the machines will talk to us, as we now talk to each other, and talk to each other silently using some form of wireless protocol. In a fully digitised world with a fully developed IoT, you can imagine that finally, it will be the machines who serve the people, fading into the background as they do.

Lessons

I think there are two key lessons from understanding this evolution of digital strategy:

1. Organisations who remain focused on e-commerce alone will eventually be outcompeted by those that embrace all four stages of digitisation. All organisations must now look at all four phases of digital in parallel in order to remain competitive.
2. Organisations where a high proportion of the total labour cost is direct (that is directly proportionate to the volume of products and services provided to customers) rather than indirect (that is devoted to researching and developing new and improved propositions) will fall behind. Employees should be firmly focused on creating new sources of value, and not delivering existing sources of value.

Your brain could be your password

One of the obstacles to online engagement is the number of passwords your customers have to remember.

Using popular sites such as Facebook or Google to provide authentication services is one way to avoid forcing your users to create yet another login and password. However, science is looking for even better solutions: this video from Mashable describes a number of alternatives, the most notable being the use of the unique patterns of brain waves each of us has.



This has obvious advantages over fingerprint and retina scanning, as I am sure we've all seen films where the villains remove someone's finger or eyeball to gain access!