Big data is an essential tool for retailers but it shouldn’t get in the way of good old-fashioned intuition and judgement.

It sometimes feels as if there should be a variant of Moore’s Law suggesting that the hype attached to big data doubles every two years.

That is not to downplay the huge benefits already being realised in personalisation, prediction and prescription: indeed the ability to use and act on big data is becoming a critical element of competitiveness.

At OC&C, our analytics division of data scientists is our fastest-growing and most-demanded capability. But crucially we regard it as a tool of strategy and an enhancement to decision-making, not a service in its own right.

The reason is perhaps best illustrated by the banking industry. Sophisticated quantitative tools crunching enormous volumes of real-time data did not alert bank boards to the risks of leverage or exposure to the property market. Big data did not spot the approaching cliff – in fact it drove the banks even faster over it. While profits rolled in, banking executives found it too convenient to ignore their intuition and instead seek false comfort from the ‘quant jockeys’.

Any model has implicit assumptions. Analysis starts with a hypothesis and any optimisation technique can blow up if driven too hard. It’s the chief executive’s job to understand what’s inside the black box and test that it remains fit for purpose.

The UK grocery industry is rightly admired for its world-class skills in collection and analysis of customer transaction and loyalty data. But that did not give sufficient warning of a macro change in shopping behaviours which has rendered swathes of hypermarket space redundant.

The big four’s adoption of rapid-response price benchmarking, reinforced by price promises, in reality formed a mutual non-aggression pact that opened the door to the discounters: a collective strategic error.

Big data will transform the speed and sophistication of decision-making in retail. Pricing and merchandising will become dynamic, digital marketing activity will be algorithmic, offers will be totally personalised, supply chain and operations will continue to become more efficient, and understanding of customer behaviour and sentiment will enable retailers to win more concessions from suppliers.

Retail bosses, while recognising the imperatives of big data, may find this unsettling. Years of experience in annual planning and reviews seem obsolete when decisions might be assessed in minutes or even executed in real time.

The volume and velocity of big data can feel overwhelming. But the principles of good management are just as important as ever. Retail bosses should refuse to get lost in complexity that can obscure the basic forces of customers and competition.

Big data cannot spot the black swans – the disruptive innovations that overturn an industry. No data set, however big, substitutes for intuition, creativity and savvy. Chief executives must strike a balance between thinking more statistically and maintaining a wide set of observational, qualitative and instinctive inputs. Algorithms support human judgement, but do not replace it.

  • Michael Jary, partner, OC&C Strategy Consultants