When I attended Duke University’s Fuqua School of Business in the late 90s, one of my favorite classes was Decision Analysis, taught by Professor Robert T. Clemen. It was a course where I could easily draw the line between academic theory and real-world applicability, where coin-flip decisions could be made with near certainty using data analysis.
Turns out, data-driven analysis can be used to make better decisions in just about anything —Black Friday, online dating, even politics. Just look at the election last week, as president Obama was propelled to victory by a talented team of data scientists.
Huge amounts of data were mined and analyzed to raise over one billion dollars in campaign funds, an amount far exceeding any other election in history. Sophisticated analyses were used to perfect fundraising e-mail campaigns and messaging that would yield the best results. Polling data was gathered in realtime to understand where the campaign was losing ground and to allocate resources surgically. Television ads were datadriven as well, enabling the campaign to specifically target persuadable voters and run ads during programming that appealed to a particular demographic profile. History will look at the 2012 election as the “Big Data” election.
There I said it —Big Data —arguably the most talked about trend in information technology today. As the presidential election illustrates, the concept has quickly spread from areas like retail and finance to just about every field of business you can imagine. So, what’s the big deal about Big Data?
Without overstating it, Big Data can be used to make any tough business decision. Everyday, companies are looking for answers to questions like:
- Should we enter new markets?
- How do we increase sales?
- How can we increase profitability?
- How do we drive up production?
- Should we hire or make other investments in the business?
The answer to these questions is quite possibly already in their own data.
The consulting firm Deloitte predicts that by the end of this year, over 90 percent of the Fortune 500 companies will have at least some Big Data initiative underway. With that said, Big Data may not be for every organization. Before starting a Big Data project, companies need to ask themselves what questions are they trying to answer. If these questions are critical and a priority for the business, then they should start laying the plumbing for Big Data now.
In order to extract the value out of corporate data, companies need to understand their data assets and exactly what data needs to be captured and cross referenced to produce insights. Additionally, Big Data computing will require specialized technology to support the volume, velocity and variety of data.
For example, most organizations have their data in structured relational databases like Oracle, but much of the data generated today is unstructured, high-volume web data or machine data. Technologies like Hadoop and “NoSQL” databases, such as Cassandra and MongoDB, are better designed to support massive data processing and storage. Emerging technologies such as Storm and Kafka are designed to provide real-time streaming analytics, which is critical for volume data feeds such as social networks. Even ad-hoc query tools such as Dremel have been introduced to support Big Data environments with low latency.
Big Data also brings new skill-set challenges. As companies look to answer the most relevant questions related to their businesses, they will need data analysts or “data scientists” to mine the data. And they should get started soon; according to a recent McKinsey study, the United States alone faces a shortage of up to 190,000 workers with analytical expertise, as well as another 1.5 million managers and analysts that have the skills to understand and make decisions based on Big Data analysis.
The Big Data movement is the recognition that there’s “gold in them there data stores!” There are tons of real-world examples of Big Data done right — just ask President Obama. However, it’s not something to dive into without first doing some serious soul-searching about your company’s goals. And it’s definitely crucial to have the right tools to support your unique corporate needs. But as professor Clemen always used to ask, “What would you pay for perfect information?”
