July 01, 2013

Big Data in the Workforce: Taming the Data Beast is Crucial for Survival

The application of big data in the workforce industry is gaining momentum as the gap between skills required and skills available increases, coupled with the global uncertainty in economic conditions and its impact on both employers and employees.  

With increased access to information sharing via social media and a growing number of job portals, the flow of information is more rapid, generating huge volumes of data.

The HR function and the workforce industry, both of which are heavily dependent on skilled workers at a global level, are now starting to look actively at leveraging information regarding hiring and talent decisions from this large mass of data, which increases every day. 

What Does it Mean for Recruiters?

 Talent information is no longer simply limited to a resume – it has now spread to social sources such as LinkedIn, Facebook, Twitter, personal blogs, websites, and online surveys as well as exchanges of emails, phone calls, etc. 

Now a recruiter is in a better position to find talent and conduct a thorough evaluation before deciding to hire. However, this is only possible when all the data elements are arranged in a structured layout available as and when the recruiter needs it.

Getting the required information faster and in a user-friendly format is all about harnessing big data through different ways and means.

Where is the Gap?

A large majority of those employed in Human Resource functions lack the data analytic skills or statistical background needed to sort through and use this large pool of data. Historically, HR and analytics are considered to be separate subject areas, but leveraging usable information from big data requires both skillsets.  HR as a function needs to be transitioned from a reactive to proactive stance. In order to gain the maximum benefits from big data, HR should liaison with IT and other groups to meet their analysis requirements or incorporate analytical skills internally to develop matrices and metrics useful for the organization. The active inclusion of big data analysis in the HR function will help HR executives to predict market sentiments and formulate business strategies. While most HR departments rely on skilled team members to conduct predictive analysis, succession planning and other functions, these efforts are almost always focused exclusively on permanent employees, thus omitting a sometime large portion of the workforce.  

 Where is Data Useful?

  • A more efficient and data-driven hiring process
  • Positioning a company to find talent as needed
  • Finding the right talent in any geography
  • Determining wages
  • Background verification through social media
  • Optimizing the hiring turnaround time

Where Do We Stand Now?

A conventional Human Resource Information System (HRIS) lacks the capability to measure talent or deliver the insights that companies need. Fast and updated hiring information from multiple sources is a huge challenge, requiring a system integration of a traditional HRIS with big data information feeds from multiple sources, along with a fast information-processing infrastructure.

Big data and its impact can be broadly classified into volume, velocity and variety. Big data is a combination of structure and unstructured data, and currently we only look at 10% of the entire population of data (all structured). The ratio of structured to unstructured is 1:9.

Generally, there are four steps to a ‘data to decision’ approach: 1) Acquire 2) Organize 3) Analyze 4) Decide. The most challenging steps are to organize and analyze because the multi-dimensional data from multiple sources needs to be structured for further analysis.

Technology Profile

Among the different database systems that handle big data, the open source software framework, Hadoop is a leader for many reasons, including its computational paradigm (MapReduce) and exception distributed file system (HDFS). Hadoop copies multiple replicas of data blocks and distributes them on compute nodes throughout a cluster to enable a rapid and reliable computation of big data. Hadoop can handle all types of data from disparate systems – structured, unstructured, pictures, audio files, email, log files, and more – regardless of native format. In addition, Hadoop has a cost advantage compared to other providers as it is deployed on industry standard servers rather than expensive specialized systems. 

Hadoop has grown more rapidly than other big data application frameworks due to its usage by major companies such as Yahoo, Google and IBM, along with mid-level and small startups. Hadoop’s scalable, cost-effective, flexible, and fault-tolerant solution is already changing the dynamics of large scale computing

Job Trends for Big Data Application Companies

Job trends for big data application companies

                                                                                                              (Source: Indeed.com)

A recently published Wikibon report, “Big Data Vendor Revenue and Market Forecast, 2012-2017” lists over 60 big data vendors with a 2012 total revenue of over $11 billion.  These are the top ten big data vendors that drive a hundred percent of their revenues from big data products and services.

Top Ten Big Data Vendors

Top Ten Big Data vendors