Renaissance Tech Dot Media

How to Make Use of All That Data You Have


Originally published in

Enterprises of all sizes today are deluged by data. EMC estimates that the amount of data worldwide is now doubling every two years. This rapid rate of expansion is expected to further accelerate as the Internet of Things drives exponential growth in data creation and storage.

Business data has also assumed an increasing variety of forms in recent years. It resides in multiple places — from legacy, on-premise finance systems to the cloud, and from mobile devices to social media repositories. Taken together, these storehouses of information tell the story of what a business is doing, how well it is performing, where it is likely going, and why.

As it grows in volume and complexity, enterprise data becomes increasingly difficult to decipher for the insights needed to manage a company to its highest potential. Traditional approaches, often tied to single applications and platforms, lack the power and flexibility to query multiple sources and formats, particularly as data volume increases.

Companies thus find themselves stymied by the wall of data facing them. McKinsey reports that just 1% of all enterprise data available is actually being used — and this primarily for anomaly detection and control, as opposed to predictive and business-optimizing purposes. Many companies admit that some of their most important decisions are made without being properly data-informed.

Overwhelming as the problem may seem, it is possible for any company to begin using its data to acquire the critical business insight needed to gain competitive advantage. Following is a brief guide for getting started on building a greater degree of data-driven business intelligence.

Frame your challenge

Do so as succinctly as possible. Experienced CFOs and analysts alike often get stuck right here. Eager to solve the company’s biggest, most vexing problems, their instinct is to try to get their arms around everything all at once. Resist this impulse. Focus on several key business problems you need to solve.

Fundamental operational issues are often a good place to start. Are you unable to determine production unit costs? Are you missing production quotas, failing to meet shipping deadlines, or unable to respond to customer order queries in a timely fashion? Of the problems you’ve identified, which are most pressing? Prioritize rigorously and identify a small number to begin resolving.

Organize your resources

Begin by identifying a data analytics team leader and ensure that the team’s mission has executive-level commitment. Identify team members from the various functional areas whose subject matter expertise supports the questions you are attempting to address.

Keep the group small. Populate it with those who possess appropriate technical, analytical, and collaborative skills and are capable of learning new methods in quick-study fashion. Carve out sufficient time for team members to contribute effectively within the context of their “regular” jobs. Embed the data analytics effort in associates’ formal goals and objectives.

Build your plan

Your team now needs to develop an approach for addressing the specific challenges identified. Have them begin by ensuring they fully understand the business problems at hand. Process maps should then be built that break down the different challenges into their more easily comprehended constituent parts, showing where interrelationships between and dependencies among them exist. Challenges in shipping, for example, may well be tied to shortfalls in core materials delivery, or in credit approval processes within the Finance area.

Identify and map the data sources relevant to each problem area. Then, with an eye to the priorities that have been established, build a project plan that moves the team along a detailed trajectory of data gathering, query writing and execution, data analysis, and problem resolution modeling. Address the challenges identified earlier in an informed, considered, data-driven manner.

At the same time, the team must be able to maintain a high degree of agility. Priorities may need to be adjusted based on discoveries made along the way. But before executing your plan — indeed, as part of setting the plan in motion — your team will need to build a properly tooled data analytics platform.

Assemble your analytics platform

With business process maps and data source diagrams in hand, your team will now need to determine the best tools and techniques for digging into this mix of information. Here you should consider consulting with external resources that specialize in advanced data analytics, because choosing the most appropriate toolset today is no trivial matter.

As companies have begun to grapple increasingly in recent years with the problems presented by “Big Data,” associated technologies have proliferated at a bewildering pace, providing software that moves far beyond the capabilities of Microsoft Excel. Depending on the nature and complexity of your data sets and the specific problems identified, you will very likely want to leverage several of these newer data science platforms and products, like Hadoop, Spark, Cassandra, and Storm – tools designed specifically to perform advanced, high-speed data processing, analysis, and visualization.

Most of these technologies are “open source,” developed by the highly regarded Apache Software Foundation. Each can be established within virtualized, cloud-based, pay-as-you-go environments. Your start-up costs for this effort can accordingly be kept to a minimum.

Execute and assess your analytics

Begin the execution phase of your plan by performing your first data extraction, aimed at collecting the raw data itself. Check the results: does it appear you’ve succeeded in pulling from all appropriate sources the sort of data you’d expect? For example, if you do business across the entire United States, does your data set actually report on all 50 states? If not, why not? If necessary, adjust your extract logic and rerun your process.

Now query your data. Apply the questions you believe will get to the heart of the problems you’ve prioritized to the raw data extracted. This may involve conducting additional data extractions, combining the resulting data sets, and conducting follow-on queries to help you develop a more complete understanding of cause and effect.

Consider your multiple data sources together, query them systematically, and adjust where necessary to refine the quality of your findings. You should now be able to see patterns and answers emerging out of the sea of data before you. Traditional business analytics typically answer the “what.” What you want to establish is the “why,” as that holds the key to understanding the “now what.”

You should now begin to see the power of building a comprehensive data analytics capability within your company. Developing meaningful business insight today involves combining and querying multiple, often highly voluminous and varied data sets in multi-dimensional ways. In turn, these insights should allow you to more quickly address your various business challenges and determine the effectiveness of your solutions.

Applying data analytics effectively will also help you discern actionable business insights “at the speed of business” — for example, determining the effectiveness of a sales webinar by conducting real-time online participant surveys.

Ultimately, your goal should be even more aggressive: to learn how to use data-derived knowledge to perform predictions regarding expected business outcomes. Achieving that higher-end capability will require moving beyond the essentials laid out here and embracing ever-more sophisticated data analytics techniques.