Skip to content

Confused About Big Data? Here Are 5 Things You Need To Do

2013 September 25

When the oil tycoon John Paul Getty was asked about his rules for success, he reportedly said, “Rise early, work hard, strike oil.”

It’s a funny line, but it seems positively quaint now.  It’s hard to imagine young entrepreneurs these days looking to find their fortune in resource extraction.  The days of wildcat wells have long been replaced by an industry that’s dominated  by geologists, geopolitics and an unruly band of petro-dictators.

Every age has its great opportunity.  At the turn of the century, automobiles attracted a flurry of startups.  The consumer culture drove the post-war period and, more recently, finance was hot.  The killer technology of our time is undeniably big data and building a data science capability is crucial for every enterprise.  Here’s what you need to know:

1. Identify the Opportunity

For people who aren’t in a tech related industry, data science can seem hopelessly far afield.  After all, successful executives have gotten that way because they work hard, know their business and serve their customers well.  Going to the time, trouble and investment to bring in some eggheaded data scientists just isn’t a priority for most managers.

Yet firms ignore big data at their peril.  A broad based study of 179 companies carried out by researchers at MIT and the University of Pennsylvania found that data driven firms performed 5%-6% better.  Compounded annually, that’s a decisive advantage.  A report by McKinsey found that the gap could be as high as 60% in the retail sector.

James Manyika, a partner at McKinsey and a chief author of the report, points out that big data capabilities have little to do with company size, IT investment or even the type of business.  In fact, he sees some of the biggest opportunities for components suppliers and distributors who generate valuable data through their normal course of business.

2. Build Skills

Gartner predicts that there will be demand for 4.4 million big data jobs by 2015, with 1.9 million of those positions being created in the United States.  However, the McKinsey report estimates that there will be a shortage of 140,000-190,000 data scientists and a deficit of 1.5 million managers who are capable of utilizing data driven insights.

The numbers tell a clear story.  Just as data skills are becoming crucial to compete in today’s marketplace, big data talent is becoming a scarce resource.  Any company without a concerted effort to recruit, train and integrate people with data skills will find themselves at a significant disadvantage.

Fortunately, as demand is growing, so are the resources to meet it.  UC Berkeley has launched a fully online Master’s program in data science which executives can take while they remain at work.  IBM has partnered with 1000 universities around the world to help meet the skills gap and even offers an open source curriculum for data science programs.

3. Collect Data

Probably the most important thing for managers to do is make data a priority.  We’re used to using information for a specific task, such as analyzing quarterly sales numbers together to generate market projections, but thinking about data as a resource in its own right is something relatively new.

McKinsey’s Manyika suggests that we focus on “three P’s”: Proprietary, Public and Purchased data.

Proprietary Data:  Many companies are sitting on treasure troves of information, without even knowing it.  As noted above, ordinary distributors and components suppliers often generate important data that can be reborn as entirely new businesses.  A company called has even figured out how to make a business out of the data from bounced emails.

Public Data:  A second important source are public databases that are offered free of charge, such as  So someone with proprietary industry data can correlate that information with broader economic trends.  These can be used to build a predictive model that can help improve decision making.

Purchased Data:  A number of data marketplaces, such as Microsoft’s Windows Azure Marketplace, have sprung up  In order to satisfy demand.  They offer access to valuable resources such as point-of-sale  and industry specific data.

In the digital economy, information is power.  So you can’t afford to ignore the resources that are available to you.  Once you have data, you can start looking for valuable patterns that can propel your business forward.

4. Unify Your Architecture

We’ve come to expect consistency from the brands we do business with.  Go to a major retailer like Bloomingdales or The Gap and you’ll see similar brand communication, selection and level of service, no matter which location you find yourself in.

However, that consistency often breaks down between physical and digital brand channels.  See something you like on a mobile site as you walk down the street and then enter a store, chances are the brick and mortar staff won’t have the first idea what you’re talking about.  Their organizations—and therefore their systems—just aren’t designed that way.

Bobby Emamian, CEO of Prolific Interactive, a mobile strategy and development firm, thinks that much of the problem is due to a historical focus on platforms, rather than functions.  Firms build an in-store POS system, then a web site and then a mobile app, all with separate architectures.

In order to move quickly but remain effective, he recommends using an API driven approach to integrate legacy systems before building a completely new architecture.  His firm is currently working on proprietary technologies that will help manage and facilitate this process.

5.  Adopt a Big Data Mindset

The power of data to transform industries is often not so much a matter of resources, or even skills, but mindset.  A good example is the story of Goldcorp, which was once a struggling mining business.  With the business in jeopardy and frustrated by the lack of progress, CEO Rob McEwan decided to do something truly revolutionary.

In what became known as the Goldcorp Challenge, he took 400 MB of proprietary data, put it online and offered $575,000 in prizes for anybody who could locate promising seams.  More than 1,400 contestants identified 110 new targets, 80% of which resulted in substantial new discoveries of gold.  Goldcorp is now a $25 billion company.

Much like Goldcorp’s McEwan, to compete in a big data world, managers will have to defy convention and adapt a new mindset.  The traditional, step-by-step strategic planning approach will no longer do.  We’ll have to adopt a more adaptive, Bayesian approach to strategy, where we look to capture value from patterns we find in real time data.

One thing is clear—there is a growing divide between firms that can manage in a big data world and those that can’t.  If you want to be able to compete in the future, you had better start now.

– Greg

2 Responses leave one →
  1. October 17, 2013

    Hi Greg, what xactly is meant by perform 5% better?

    Does it mean that their EBIT is 5% higher in absolute terms (say 15 instead of 10) or relatively e.g. 10.5 instead of 10, or does it mean that they are located to the right of a distribution of returns with 5% spread?

    this said the article is amazing.


  2. October 17, 2013


    There is a detailed explanation in the paper that I linked in (here’s the link again:, but it is in terms of both output and productivity, but not financial performance (although it would follow that would be affected too), compared to other firms. So it is a relative, not an absolute measure.

    Generally speaking, EBIT and related measures are not used in this type of analysis because you have some other factors there that might skew the analysis. Multifactor productivity is really what you want to look at, as well as overall output so that you can be sure the firms in question aren’t “shrinking to greatness”

    – Greg

Leave a Reply

Note: You can use basic XHTML in your comments. Your email address will never be published.

Subscribe to this comment feed via RSS