Posts tagged Text Analytics Demo
Shedding Light on Dark Data: How to Get Started

Move over Big Data. There’s a new buzzword: dark data. It’s actually not so new—Gartner coined the term a couple years back—but dark data is finally starting to catch on in market research circles and it represents a huge untapped opportunity for insights!

Gartner defined dark data as “the information assets organizations collect, process, and store during regular business activities, but generally fail to use for other purposes.”

The definition has since expanded to encompass not just internal data, but the broader spectrum of data that are readily available to organizations.


The common denominators are 1) these data are largely unstructured and 2) they are not being analyzed. In fact, according to IDC, 90% of the unstructured data are never analyzed!

Why Search in the Dark?

dark-data-lamp-postMaybe you’ve heard this one?

A police officer comes upon a man crawling around on all fours under a streetlight one evening.

The man explains that he’s looking for his wallet.

“Where do you think you lost it?” asks the policeman.

“Across the street, but the light is so much better here,” says the man.

Popular among data scientists, I think this joke illustrates the irrationality of a lot of common thinking in research these days. We tend to search for insights in a relatively limited but easily accessible location—survey data—as if the only answers to be found must be there.

And even that relatively small pond isn’t being thoroughly fished. As I’ve blogged in the past, for most of us, even survey open-ends/comment data are still “dark data”!

At the risk of deluging you with metaphors, the fact remains that what we can find in our survey data is only the tip of the insights iceberg.


We have at our disposal all manner of unstructured data for which text analytics are uniquely suited to organize and understand, including images and video—without any enrichment or visual content analysis. For example, images often contain file name and metadata descriptions in text format that can be analyzed with software like OdinText. Videos, too, often contain transcript data, and there are technologies like YouTube’s, which can handle audio-to-text translation.

A Few Things to Consider

Dark data can be Big Data. And very Big Dark Data can prove daunting (that’s partly why it stays dark in the first place).

But dark data can also be quite small we’ve found.

And just as Big Data isn’t necessarily valuable just because it’s big, dark data certainly isn’t valuable just because it’s dark.

Lastly, technology can’t make garbage data valuable and the complexities involved in analyzing some forms of dark data often require taking a sample or deciding exactly which parts of the data might prove most interesting to analyze.

Don’t Be Afraid of the Dark

There are tons of ways to start putting dark data to work for your organization. Here are recent examples of how clients are using OdinText currently to shed light on their dark data.


Phone transcripts, chat logs and email are often dark data that text analytics can help illuminate. Would it be helpful to understand how personnel deal with incoming customer questions? Which of your products are discussed with which of your other products or competitors’ products more often? What problems or opportunities are mentioned in conjunction with them? Are there any patterns over time?

We already have clients doing these types of analyses with OdinText. It is almost always exploratory at first, but these clients recognize the need to look.

Merging Disparate Dark Data Sources

How about integrating, say, audio file transcripts from a call center with click data from websites? There are plenty of cases where merging dark data sources can yield important insights that would not be attainable using conventional tools.

In such a case, you would typically start with the goal of understanding one or more KPIs. Thinking about what data you might have available to help understand, model and predict these would be the next step. How similar are these data, again, what is the value to understanding said KPI/s?

Ideally the data that is joined is similar in some respects, but it doesn’t necessarily have to be perfect.  We may be willing to overlook various problems in this data in hopes that the aggregate data (which may involve dropping in means, merging various text fields in different ways, etc.) will give us a better understanding of how to affect and manage against our KPI/s.

Again, I must stress that even this does not necessarily need to involve/yield Big Data. For instance, if you are a pharmaceutical company and the data in question are drug tests or small samples of doctors, even after the merge the data will still be relatively small by most standards.

Also the data need not be any more sophisticated than simple survey data or even in-depth interviews over the span of, say, 2-3 years. That said, it is always more interesting if marketing research opinion data—whether survey or some sort of more qualitative data—is accompanied by some real behavior or outcome like efficacy or sales.

My opinion on this sort of analysis has recently changed drastically as our clients have shown us that where there is a will, there is often both a way and one or more very lucrative insights!

Do a Demo Using Your Dark Data!

In a post last week about "what text analytics vendors won't tell you", I urged market researchers who are interested in text analytics to do a demo using your own data.

But better yet, why not take this opportunity to do a demo using your dark data?

What kind of dark data do you have? To what KPI’s or Insights do you think they may hold the key? How easy might it be to acquire access to and prepare this data for insights?

These are discussions worth having. Give us a call.



Buyer Beware: What Text Analytics Providers Won’t Tell You.


Text Analytics May Not Be For You

[video width="1280" height="720" mp4=""][/video]

If you’re in market research and aren’t at least exploring text analysis software, you’re part of a shrinking minority. But you probably know this already, if only from the preponderance of conference presentations, blogs and trade articles on the topic. Yes, text analytics are all the rage these days.

You may feel under the gun to catch up, but if you’re late to the game, you may be comforted to know that for many people, text analytics aren’t living up to the hype.

Nearly every researcher I come in contact with at conferences and through my professional network is at least actively investigating text analysis if they haven’t already adopted a solution. And in either case, they’re frequently underwhelmed.

It’s my experience that there are two primary reasons for this:

  1. Despite the hype, text analytics are NOT for everyone


  1. Researchers too often buy technology when what they need is an insights solution

How Do I Know Before I Buy?

You heard me. Not everyone needs text analytics. And that’s because not all data merit or are even suited to text analysis, nor is every business question.

I see researchers buying buzzwords like “sentiment” or “artificial intelligence,” when what they need is to understand what drives customer satisfaction, what drives loyalty, what drives behavior or revenue…

The first things you should ask are: 1. Do I need text analytics to answer my business questions? and 2. Do I have the right data for it?

Most text analytics software providers will give you a resounding “Yes!” to both questions. (Shocking, I know.)

Then they’ll set up a dog-and-pony demo, possibly using data similar to yours, but it isn’t your data.

What they won’t do is take the time to honestly determine whether or not what they’re trying to sell you makes sense for you.

And they won’t confirm it by running your actual data before you buy.

Do a Demo with Your Own Data Before the Year is Out!

It’s Q4, which means that from now until the end the year, my colleagues and I are going to get a lot of calls from researchers who need to spend what’s left of their budgets and who are convinced that they need a text analysis tool.

But before we even discuss an OdinText demo, we will take the time to figure out for certain whether they actually need us.

And if it looks like a fit, we’ll arrange a demo using their actual data to be sure.

Whether you haven’t yet found a text analytics solution that works for you or you just feel pressured to adopt a tool before the year is out, I am telling you that text analytics may not be the right answer for you.

Find out for sure. Before you close the books on 2016, call us to arrange a phone consultation and take an OdinText demo with YOUR OWN DATA!

We won’t sell you what you don’t need.


Tom H. C. Anderson

OdinText Inc.

888.891.3115 x 701