How to Prepare for the Big Data Challenge

Dream. Dare. Do – that is Suyati’s work principle in a nutshell.

  • Author:
  • Nayab Naseer

With big data all set to become a key source of competitive advantage, companies are going all out to implement big data analytics. The success of such big data efforts, however, depends on having data to analyze in the first place. Many companies take this for granted when the reality is that even when data is available, it requires considerable amount of preparatory work to ensure that the data is fit and ready for big data analytics.

The following are the crucial preparatory works that should ideally predate the rollout of big data analytics:

1. Make data available online:
This entails digitizing paper documents, saving email communications and converting offline records to online databases. The all-pervasive nature of computers and internet notwithstanding, a bulk of the data still remains out of bounds and the basic requirement of big data analytics is to make such data available for analysis.

2. Increase Storage Capacity:
The key technical challenge related to the task of accumulating data digitally is enhancing storage capacity. It is common for the size of the data to surge to petabytes. Two options are scaling the existing infrastructure either vertically or horizontally. The more scalable and cost-effective option is horizontal expansion or scaling out with NAS (Network Accessible Storage) architecture which entails adding nodes as required rather than enhancing the capacity of the existing servers.

3. Structure the Data:
Aggregating and storing the data is only half the job. Big data analytics requires bringing such data and the data that comes in from unstructured and disparate sources – ranging from social media feeds to factory floor sensors and from billing terminals to customer feedbacks – into a format that can be easily read and analyzed. This entails structuring the data into common databases or converting files into a standard format.

4. Overhaul System Architecture:
In addition to increasing storage capacity and structuring data, big data analytics necessitate major changes in server and storage infrastructure, and information management architecture. Big data analytics call for extensible and scalable systems with the capability of integrating the disparate systems that feed the big data analysis effort. The disparate, siloed systems for different functions prevalent in most organizations today, remain incompatible for big data analytics.

5. Prepare Staff:
An underestimated dimension of the big data challenge is staffing. There is a severe shortage of big data professionals ranging from data scientists to Hadoop specialists. McKinsey estimates the need for 140,000 to 190,000 additional experts in statistical methods and data analysis technologies, and an additional 1.5 million data-literate managers with formal training in predictive analytics and statistics. And this is in the US alone. Companies who invest in training their staff will have a head start in the game.

Organizations that undertake such preparatory works before rolling out their big data systems such as Hadoop, MapReduce, and NoSQL stand a greater chance of success. Conversely, organizations that neglect such preparatory tasks run the risk of their efforts going waste as the big data analytics would generate distorted results.

Image Credit: InformationWeek.Com

Leave a Comment

Your email address will not be published. Required fields are marked *