Myths of Big Data

Following up on the last two blogs (link 1 & 2), this one is dedicated to big data. With statements like “data is the new oil”, we may be at the peak of inflated expectations in the hype cycle before the sober reality sets in. Big data has now become a large industry, touted as cure for all that ails marketing.

Yet, reflecting on just my personal experience, we have a long way to go before big data delivers on its potential. Consider a couple of examples. I log into Facebook from Lausanne in Switzerland. I have been logging in for a couple of years from the same laptop and using the social site in the English language. Yet, what I see is the Facebook log in page in German. Perhaps, location trumps the laptop’s IP address. Still, even if it is based on location, in Lausanne, the log in page should be in French. Similarly, I recently moved to Singapore Management University. I now see numerous advertisements to enrol in various SMU programs!

Despite being a believer in big data, we need to have realistic expectations of big data. As marketers, we are at the start of a very long, journey.

The Myth of Big Data

The biggest myth is that there is something called big data. In reality, the construction of a useful database requires one to stitch together lots of small data from multiple sources. This is what makes it hard work.

  • How many different sources can one stitch together to capture all the digital exhaust from a customer. And, what is simply escaping in the atmosphere?
  • How to match data from several sources, each with its own approach to collecting, maintaining, and refreshing the data, and link it to a unique individual?

As one gets better at this, it allows greater integration across the different touchpoints through which customers experience the brand.

More Data is Better

Vendors, motivated by self-interest, keep pushing the idea that more granular the data, the better. And, of course, more data is always better. But big data is like sand grains which can create a sandstorm. It can blind you, it can disorient you. There is an old Bedouin saying that six minutes of a sandstorm is enough to make a man go mad.

  • Early in one’s research career, you learn that there is always too much data and not enough data simultaneously. A lot of the data that initially seems nice to have is unnecessary to solve the problem at hand. Furthermore, there is always more data one would have liked, but is unavailable.
  • Data, in itself, has no value. It is only a cost until it is married with analytics to extract actionable information. Key decisions are what to keep and what to ignore. Water in a glass is manageable, but in a flood, is overwhelming.

For marketers to exploit big data, they must learn to live with ambiguity. Rather than simply seeking more and more data, the technology and understanding to use the data available is perhaps more important.

I recall, early in my career, being asked to be an expert witness. My first question was: am I defending the market research or attacking it? In the face of a hostile lawyer, defending market research is challenging but attacking it is easy. There is no perfect data or model, one must be able to live with the noise. For managers, the focus should be, despite the problems with the data and model, what can one learn from it?

Real Time Data is Better

It seems obvious that data that is updated as close to real time as possible is better. While this may be true in some cases, but the accompanying “noise” can lead to greater confusion and errors.  Often, instead of a quick response, it is better to take the time to reflect. Only, after cleaning the data, understanding the problem, and improving the model, should one move to action.

  • No matter, how real time the data is, it is past data and one is trying to predict future behaviour.
  • For models to be useful, marketing managers should adopt lessons from the old Japanese quality movement of continuous improvement. But this requires testing of models rigorously. Before accepting a model, one must have the discipline to use hold out samples and randomized experiments. It is too easy for marketers to adopt the spray and pray.

Most things in life that are important and worthwhile are difficult to do. Big data is no exception. The models put in practice cannot simply be outsourced to data analysts or machine algorithms. Marketers are going to have to get their hands dirty and grapple with obtaining a deeper understanding of what went into the making of the model and the quality of the data that was utilized. If models based on big data become a “black box” for marketers, it will be a continuation of the much derided ‘faith based” marketing practices of the past.



Magic Dust

Former Tata Sons senior executive Nirmalya Kumar joins Singapore Management University – Forbes India

I invite you to follow me on Twitter @ProfKumar and LinkedIn.

Leave a reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s