You are here

A Smart Idea | The Evolution of Big Data

The arrival of Big Data has drastically changed the industry, completely altering the ways in which researchers are able to collect, handle and analyse vast amounts of data. This has been a necessity for 21stCentury Market Research, largely due to the rise of digital usage and the ever-increasing popularity of social media. Never before have we been able to use data in such a spectacular way, and the amount of information that can now be collected and analysed is staggering. Now, we could go on about the wonders of Big Data, but let’s face it; you’re probably tired of hearing about it already.

What really interests us is the evolution of Big Data. What happens next? Where does Market Research go from here?


The Smart step

Colin Strong, writing for Research Live, suggests that “Smart Data” should be the next step for researchers. Smart Data, he argues, is the combination of consumer data with behavioural information, and a social science perspective. To fully understand customer intention, Big Data just isn’t enough. A more humanistic approach, involving Psychology and Sociology may be necessary to use this data in the most effective way. Big Data can only provide objective information about people’s behaviour (what they show) and not what behaviour they believe they show or want to show. The limited ability to predict behaviour is a major disadvantage to Big Data analysis. To combat this, Strong suggests that researchers take the “Smart Data approach” when trying to understand consumer behaviour, and incorporate qualitative techniques to the Big Data findings.


Don’t underestimate the use of Big Data

It can be argued, however, that Big Data does not require the help of qualitative methods when trying to predict behaviour. Alex Pentland, in the Harvard Business Review, states that “current limitations on the interpretation of human behaviour mostly go away” with Big Data use. Experts, he claims, have the ability to tell an enormous amount about an individual’s behaviour through this data, more than the best survey research or focus group can do. They are able to do this because Big Data is beginning to reveal to us two important facts. The first, that our behaviour is largely determined by the social context we are in. The second is that our behaviour is much more predictable than we would suspect. Pentland suggests that these two facts allow researchers to observe only a few of our behaviours, and can then infer the rest, just by comparing us to the other people in your social environment.

It is debatable whether inferring behaviour is the correct way to go about predicting how consumers will act. It can result in incorrect or misleading information about an individual, reducing the reliability and validity of the research, and cause irrelevant or unsuitable information to be directed towards the consumer.


Where do we go from here?

The Smart Data approach appears to be the most likely progression and, through coexistence between Big Data and qualitative insight, the ability to understand and predict consumer behaviour will be better than ever. It will be interesting to see if the industry will take to the Smart Data approach and see it as the most effective way for brands to gain the greatest understanding about their consumers.

It is also important to remember that Smart Data (or even Big Data) will not replace the existing and well-established research methods but instead, they will accompany them and help deliver more accurate and holistic results. From our own experience we know that our work gives clients not only what they want, but also what they need, and provides them with the answers they are looking for. Smart Data and Big Data can often create more questions and, as a result, require brands to adjust their research expectations. Existing methods still do a very good job and continue to prove their worth time and time again. When looking for specifics, Big Data isn’t always the best answer, so be careful not to assume that “big is best”. Remember, size doesn’t always matter…


Will Smart Data become the next big thing? What are your thoughts on the evolution of Big Data?

Let us know and join the discussion here:

Twitter | LinkedIn | Facebook | Google+