Discovering Core Metrics By Illuminating Dark Data
In the Beginning, There was Light
Back in the late 90’s, the “Information Age” began moving at the speed of light as global Internet connectivity became affordable and readily available to average businesses and households. Fiber optic networks were expanding at a rapid pace, and broadband communication was even traded as a commodity by Enron until the company’s collapse in 2001.
At the time, many people understood the raw power and potential benefit of a truly global network, but as with most innovations, lessons must first be learned before experience can be gained. Topics like Internet security, information privacy and master data management quickly became a central discussion in the public forum. Moving forward to present day, these topics (and many more) are still fresh on the minds of IT executives and professionals as new technologies become available and open the door to additional risks.
Despite many challenges, the Internet has enabled some incredible capabilities over the years. One of the most inherent and prolific examples is the ability to efficiently capture vast amounts of global data from various sources like mobile apps, desktop apps and even sensors that harvest real time operational data. The sheer volume of information that gets generated in one day is almost hard to comprehend. According to IBM, the world generates about 2.5 quintillion bytes of data every day. To put that in perspective, a modern DVD stores roughly 5 gigabytes of data, which means a person would need to buy almost 500 million DVD’s to store a day’s worth of global data.
Entering the Dark Ages
Ever demanding priorities, coupled with rapid and organic business growth, routinely force executives and their management teams to operate in reactive mode. They fight fires on a daily basis and have less time to focus on proactive activities like planning and strategic thinking.
Master data management is all about protecting your data and providing a framework that controls how data can evolve over time
Without periods of calm between storms, it can be challenging for an executive to know what reliable data is at their fingertips, so decisions tend to be made subjectively “by feel” rather than objectively “by metrics”. This statement isn’t meant to discount the value of subjective skills like intuition and experience, but rather distinguish between the two different types of decisions executives and managers make every day. When used correctly and within reason, metrics can help augment decision making and enhance overall situation awareness.
Given the nature and speed of business decisions, it is often easy to forget about the types of data your company may be collecting. Depending on the size of your business, you may not even know that certain systems exist. Silos, or gaps in communication, can make it hard for employees in large organizations to understand what data is available. Likewise, it can be just as challenging for small businesses to know how much data they should store, as well as how to organize data so it can scale with growth.
Regardless of your company’s size, what ends up happening over time is that much of the data collected by corporate systems is either archived, lost, forgotten, or a combination of all three. More importantly, the data is no longer in the forefront of anyone’s mind and has effectively gone “dark”.
What is Dark Data?
In the technology world, new buzz words pop up every day. With recent focus on Big Data and the large volumes of information being gathered by companies, Dark Data has become a trending topic.
According to Gartner, Dark Data is “the information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes (for example, analytics, business relationships and direct monetizing)”. According to IBM, “about 90 percent of data generated by most sensors and A to D converters on the market never get utilized, and 60 percent of that data loses its true value within milliseconds”.
As one might imagine, Dark Data also has storage and security implications. The longer data accumulates within system databases, the more money a company will spend on storage solutions. Likewise, security risk and exposure can also increase, especially as it relates to sensitive data.
Discovering Your Core Metrics
In order to emerge from the Dark Ages, it is important to understand what data you care about as a company. What really drives your customers to buy your product or use your services? How do you effectively increase revenue while managing your expenses? Is it reasonably possible to capture metrics around any of these questions?
Every day, we interact with companies that have discovered their core metrics and understand why capturing relevant data is so important. For example, the airline industry knows their “on time” statistic is a metric that drives customer satisfaction and ultimately ticketing revenue. Maritime logistics companies use a similar “on time” metric coupled with “safety” metrics to ensure products reach their destinations as expected and without incident.
Even if your company doesn’t have a good way to capture the data required for meaningful metrics, all companies can use financial numbers, trends and forecasts as general metrics of financial health. The important thing is to determine what metrics work best for your business and see if data exists to support it. Going forward, this allows you to proactively focus on collecting data you care about, and potentially eliminating data that has no real value.
To provide an actual business case, service companies like Contract Land Staff (CLS) use project management metrics to show customers if a project is on time and within budget. For many years at CLS, these metrics were generated manually behind the scenes through status updates with project teams and by digging through multiple data sources. The process was both time consuming and potentially error prone. Fortunately, CLS has a mantra of continuous process improvement, and the company’s Chief Project Officer, Phil Pack, wanted to implement a much more efficient solution with available technology.
The Renaissance – Bringing it All Together
In summary, IT executives and managers need to be proactive every day. Take the initiative to determine what data is important to you and your company. Help facilitate a collaborative environment and work with internal departments to identify owners of business data. Assigning owners is an important step, because it helps provide clear communication pathways that can break down internal silos. It also instills a sense of accountability and helps spread the workload of managing large volumes of data across the organization.
After building a cohesive team of data champions, implement a master data management strategy to effectively support the people and process that drive what you want to measure. Master data management is all about protecting your data and providing a framework that controls how data can evolve over time.
This process is not easy and will take time to accomplish, but is well worth it in the end. Through proactive management, collaboration, and fostering the right controls and procedures, you can help illuminate dark data within your organization and discover the true value of core metrics.