It’s been about a year and a few months since the last time I posted about Big Data. I was in the middle of a master’s program at the time, and we dealt with huge data sets that entailed countless hours of SPSS and Excel analysis. I thrived in that environment and everything was blissful. We learned how to segment customers to learn the top groups on which to focus loyalty programs, as well as nascent customers who we can nurture into better ones. We looked at all sorts of data: transactional, respondent-level, customer profiles, surveys… it was an analyst/strategist’s dream environment.
But that was academia. Professors from this top program had leverage to partner with some of the biggest companies in multiple categories to gain access to data that can be analyzed by students. Some were syndicated data, which means companies have since updated their data sets and are not using currently using what was given to us, while others were live data, which means the companies were looking to our specialized skills to help them become better informed about their customers.
Outside of the academic world and into my professional life, I realize that the environment that my classmates and I played in was even more idyllic: we went straight into analysis because that was what we were required to do. Sure, we still had to clean the data, and we still had to spend hours organizing in a way that SPSS can ingest, but we didn’t have to deal with lack of data or management politics or clients who couldn’t be persuaded to transform their marketing strategies. Everyone was willing, collaborative and the data was ripe for analysis.
This is not the case in the real world. After graduation, I’m so eager to find projects where I can get my hands on data to analyze. If I can just work on CRM projects, consumer research and data analysis all day, I would. But before any kind of work can be done on data, it needs to be organized, cleaned, enriched first. This means data must first be converted into structured rows and columns because currently, the tools available can only analyze structured data. This means data must be prepared in a way that nothing is mismatched and errors are minimized, or else analysis will revert to the old mathematics adage: “garbage in, garbage out.” This means data must be combined with other data sets as appropriate to make sure gaping holes are minimized, data has multiple dimensions, in order to maximize analytic value.
All the steps that need to be done before analysis exist in the land of nightmares for an analyst. But this is the reality and us analyst go through these steps and push for that light at the end of the tunnel, because we know that there are no shortcuts and all the steps are necessary. It will make our lives easier when the time comes for analysis and deriving insights from the rock of data that we want to turn into gold.
The term “big data” has existed for more than 3 years now, but still, we are still at the nascent stage of this revolution. Clients, even the large, resource-rich corporations, are still struggling with how to deal with the surplus of data they have, let alone analyze it to begin extracting value. Some companies are ahead of the curve, especially those who have traditionally been in data-heavy industries like banking, insurance, pharmaceuticals. But not all. Some of the verticals I see the most potential in are: CPG, retail, hospitality (hotels), telecom. Or any vertical that have robust loyalty programs.
CRM data is gold because with it, you begin to understand consumer behavior. It can also be the foundation for other kinds of data you can combine with it (attitudinal data, transactional data, etc). And that’s the key: it must be continuously updated, cleaned, organized.
But if we really are to take a step back and start from the beginning, what companies need to make sure they have, at the most basic level, is the means to capture and create customer data. Without this, they will miss out on the tidal wave that has been transforming businesses, and is about to revolutionize economies.