Shedding Light on Dark Data: How to Get Started

Tom H. C. Anderson
November 16th, 2016

Move over Big Data. There’s a new buzzword: dark data.

It’s actually not so new—Gartner coined the term a couple years back—but dark data is finally starting to catch on in market research circles and it represents a huge untapped opportunity for insights!

Gartner defined dark data as “the information assets organizations collect, process, and store during regular business activities, but generally fail to use for other purposes.”

The definition has since expanded to encompass not just internal data, but the broader spectrum of data that are readily available to organizations.

dark-data-text-analytics

The common denominators are 1) these data are largely unstructured and 2) they are not being analyzed. In fact, according to IDC, 90% of the unstructured data are never analyzed!

Why Search in the Dark?

dark-data-lamp-postMaybe you’ve heard this one?

A police officer comes upon a man crawling around on all fours under a streetlight one evening.

The man explains that he’s looking for his wallet.

“Where do you think you lost it?” asks the policeman.

“Across the street, but the light is so much better here,” says the man.

Popular among data scientists, I think this joke illustrates the irrationality of a lot of common thinking in research these days. We tend to search for insights in a relatively limited but easily accessible location—survey data—as if the only answers to be found must be there.

And even that relatively small pond isn’t being thoroughly fished. As I’ve blogged in the past, for most of us, even survey open-ends/comment data are still “dark data”!

At the risk of deluging you with metaphors, the fact remains that what we can find in our survey data is only the tip of the insights iceberg.

dark-data-text-aanalytics-ice-berg

We have at our disposal all manner of unstructured data for which text analytics are uniquely suited to organize and understand, including images and video—without any enrichment or visual content analysis. For example, images often contain file name and metadata descriptions in text format that can be analyzed with software like OdinText. Videos, too, often contain transcript data, and there are technologies like YouTube’s, which can handle audio-to-text translation.

A Few Things to Consider

Dark data can be Big Data. And very Big Dark Data can prove daunting (that’s partly why it stays dark in the first place).

But dark data can also be quite small we’ve found.

And just as Big Data isn’t necessarily valuable just because it’s big, dark data certainly isn’t valuable just because it’s dark.

Lastly, technology can’t make garbage data valuable and the complexities involved in analyzing some forms of dark data often require taking a sample or deciding exactly which parts of the data might prove most interesting to analyze.

Don’t Be Afraid of the Dark

There are tons of ways to start putting dark data to work for your organization. Here are recent examples of how clients are using OdinText currently to shed light on their dark data.

Communications

Phone transcripts, chat logs and email are often dark data that text analytics can help illuminate. Would it be helpful to understand how personnel deal with incoming customer questions? Which of your products are discussed with which of your other products or competitors’ products more often? What problems or opportunities are mentioned in conjunction with them? Are there any patterns over time?

We already have clients doing these types of analyses with OdinText. It is almost always exploratory at first, but these clients recognize the need to look.

Merging Disparate Dark Data Sources

How about integrating, say, audio file transcripts from a call center with click data from websites? There are plenty of cases where merging dark data sources can yield important insights that would not be attainable using conventional tools.

In such a case, you would typically start with the goal of understanding one or more KPIs. Thinking about what data you might have available to help understand, model and predict these would be the next step. How similar are these data, again, what is the value to understanding said KPI/s?

Ideally the data that is joined is similar in some respects, but it doesn’t necessarily have to be perfect.  We may be willing to overlook various problems in this data in hopes that the aggregate data (which may involve dropping in means, merging various text fields in different ways, etc.) will give us a better understanding of how to affect and manage against our KPI/s.

Again, I must stress that even this does not necessarily need to involve/yield Big Data. For instance, if you are a pharmaceutical company and the data in question are drug tests or small samples of doctors, even after the merge the data will still be relatively small by most standards.

Also the data need not be any more sophisticated than simple survey data or even in-depth interviews over the span of, say, 2-3 years. That said, it is always more interesting if marketing research opinion data—whether survey or some sort of more qualitative data—is accompanied by some real behavior or outcome like efficacy or sales.

My opinion on this sort of analysis has recently changed drastically as our clients have shown us that where there is a will, there is often both a way and one or more very lucrative insights!

Do a Demo Using Your Dark Data!

In a post last week about “what text analytics vendors won’t tell you”, I urged market researchers who are interested in text analytics to do a demo using your own data.

But better yet, why not take this opportunity to do a demo using your dark data?

What kind of dark data do you have? To what KPI’s or Insights do you think they may hold the key? How easy might it be to acquire access to and prepare this data for insights?

These are discussions worth having. Give us a call.

@TomHCAnderson

tomtextanalyticstips

7 thoughts on “Shedding Light on Dark Data: How to Get Started”

  1. Good stuff Tom Anderson! In almost any context that I’ve encountered, most companies sit on an abundance of data, often unknowingly and certainly always with a great potential for gaining valuable insights into their internal and external environment. The data labelled ‘Dark Data’ would often be used – in fragments – for transitions or transformations, as they become obviously useful in this context. Do you have any experience or know of cases where Dark Data has been used effectively to improve the market oriented approach, i.e. for the purpose of segmentation, market positioning, branding, etc.?

  2. We are working on a new case study draft as we speak. Assuming it gets approved by client will be able to share it soon.

    In a way, like Big Data, which is Big for some and not for others, it is the same for Dark Data. Something as simple as survey open ends/comment data for instance, is dark for some and not for others. Definitely it is very useful! Our ShellOil case study on our website is but one good example we have. But there are man other on other data sources including email (this is very new and very dark for most).

  3. So I believe in theory that all data that is in the sphere of Big Data is Dark data.
    At some point the data was stored and not looked into, then people would have seen the benefits of analysing this (this would have probably been done in RDBMS). Then came along Big Data so it does feel like the evolution of data somewhat.

  4. Nice Stuff.. Thanks Terence… I fully agree with your inputs. Can you also share your thoughts on how the DARK data is used in FMCG sector ? I am currently working in this domain and would like to understand how dark data can be effectively used here

  5. Every company in every industry has dark data. We’re coming across more and more of it not because we are thinking of it, but because our clients are bringing it to us asking “we think there might be value something text analytics can hep with?” in many cases the answer has been yes.

    Beyond the concept of Dark Data though, I think we also have “Gray Data”, data that gets a cursery look or very limited analysis, perhaps via sampling and occasional human only analysis which is not being tapped to it’s full power because the analysis applied is too basic, or just a small part of the data is leveraged, or it is not merged with the KPI/Variable of interest it should be merged with.

    In the visuals above one thing I think we failed to get across unfortunately is just how big this opportunity is. Dark (and certainly Gray) Data in my opinion exists in both the traditional enterprise data as well as on it’s own (and of course most of the Big Data is also Dark), so the size/opportunity in far larger than I was able to communicate above.

    BUT, just because it’s dark or big doesn’t mean it has automatic high value. Naturally ne has to evaluate it in relation to the important KPI’s that are of interest.

Comments are closed.

Search