February 01, 2016

Discover the Diamond in the Rough

It’s a commonly believed myth that big data is something that is only a technology issue or even just a data issue. Big data is really about enabling business users to make decisions that create value. While the technology toolset that enables organizations to harness their volumes of data is important, it’s important that companies striving to gain insights from their raw data do not limit their focus to the technology solutions. Rather, it is important that the goal be focused towards helping business leaders and managers make better sense of the real data they have, access it quickly, and make high-value decisions.

What is Data Mining?

Data mining is defined as the process of finding anomalies, patterns and correlations within large data sets towards predicting outcomes. These insights can be used to increase revenues, cut costs, improve customer relationships, reduce risks, and more.

In today’s world with its increasing volumes of data, data mining is becoming more and more important. The volume of data produced doubles every two years and unstructured data makes up 90 percent of the digital universe. Data mining allows companies to shift through the chaotic and repetitive noise in your data, understand what is relevant, and accelerate the pace of making data-driven decisions.

According to many experts, data mining consists of five major elements:

  1. Extracting, transforming, and loading transaction data into a data warehousing system
  2. Storing and managing the data in a multidimensional database system
  3. Providing data access to business analysts
  4. Analyzing the data by application software
  5. Presenting the data in a useful format, such as a graph or table

The ultimate goal of data mining is to find valuable gems of insight that are hidden among digital rubble.

Some Examples of What Data Mining Can Do

Data mining is used to simplify and summarize data in a way that is simple to understand, allowing business users to infer things about specific cases based on patterns observed. There are several main types of pattern detection that are commonly used, and illustrate the abilities of data mining.

Anomaly Detection: In large data sets, data mining can be used to determine if there is an anomaly in the general pattern. For example, the IRS could model typical tax returns and use anomaly detection to identify specific returns that are different for review and audit.

Association Learning: This type of data mining helps to drive recommendations based on observations. For example, if a customer purchased a wine opener and a wine tasting set, they might be open to purchasing wine glasses.

Classification: Data mining can also be used to classify new cases into pre-determined categories. An example of this is spam filters, where large sets of emails that have been identified as spam have enabled filters to see differences in word usage between legitimate and spam emails, and classify these messages according to rules.

Regression:  Data mining can also be used to construct predictive models based on different variables. For example, a social network could predict future engagement for a user based on past behavior, such as personal information shared, comments, etc.

How Different Industries Find Their Diamond of Insight

Data mining is not limited to only one industry or category. Rather it’s the center of analytics efforts of a wide range of sectors.

Communications:  Multimedia and telecommunication companies use data mining to sift through volumes of customer data, help to predict customer behavior, and offer targeted campaigns.

Education: Data mining can help provide unified, data-driven views of student progress, helping educators to predict student performance.

Retail: Large customer databases hold hidden insights that can help improve customer relationships, optimize marketing campaigns and forecast sales, allowing retailers to offer more targeted campaigns.

Manufacturing:  Data mining can help align supply plans with demand forecasts, as well as help in early detection of issues and quality assurance.