Posts tagged Survey Open-ends
What Americans Really Think about Trump’s Immigration Ban and Why

Text Analysis of What People Say in Their Own Words Reveals More Than Multiple-Choice Surveys It’s been just over a week since President Trump issued his controversial immigration order, and the ban continues to dominate the news and social media.

But while the fate of Executive Order 13769—“Protecting the Nation from Foreign Terrorist Entry into the United States”—is being hashed out in federal court, another fierce battle is being waged in the court of public opinion.

In a stampede to assess where the American people stand on this issue, the news networks have rolled out a parade of polls. And so, too, once again, the accuracy of polling data has been called into question by pundits on both sides of the issue.

Notably, on Monday morning the president, himself, tweeted the following:

Any negative polls are fake news, just like the CNN, ABC, NBC polls in the election. Sorry, people want border security and extreme vetting.

— Donald J. Trump (@realDonaldTrump) February 6, 2017

Majority Flips Depending on the Poll

It’s easy to question the accuracy of polls when they don’t agree.

Although on the whole these polls all indicate that support is pretty evenly divided on the issue, the all-important sound bite of where the majority of Americans stand on the Trump immigration moratorium flips depending on the source:

  • NBC ran with an Ipsos/Reuters poll that found the majority of Americans (49% vs. 41%) support the ban.

  • Fox News went with similar results from a poll by Quinnipiac College (48% in favor vs. 42% opposed).

  • CNN publicized results from an ORC Poll with the majority opposed to the ban (53% vs. 47%).

  • A widely reported Gallup poll found the majority of Americans oppose the order (55% to 42%).

There are a number of possible reasons for these differences, of course. It could be the way the question was framed (as suggested in this Washington Post column); it could be the timing (much has transpired and has been said between the dates these polls were taken); maybe the culprit is sample; perhaps modality played a part (some were done online, others by phone with an interviewer), etc.

My guess is that all of these factors to varying degrees account for the differences, but the one thing all of these polls share is that the instrument was quantitative.

So, I decided to see what if anything happens when we try to “unstructure” this question, which seemingly lends itself so perfectly to a multiple-choice format. How would an open-ended version of the same question compare with the results from the structured version? Would it add anything of value?

Part I: A Multiple-Choice Benchmark

The first thing we did was to run a quantitative poll as a comparator using a U.S. online nationally representative sample* of n=1,531 (a larger sample, by the way, than any of the aforementioned polls used).

In carefully considering how the question was framed in the other polls and how it’s being discussed in the media, we decided on the following wording:

“Q. How do you personally feel about Trump's latest Executive Order 13769 ‘Protecting the Nation from Foreign Terrorist Entry into the United States’ aka ‘A Muslim Ban’”?

We also went with the simplest and most straightforward closed-ended Likert scale—a standard five-point agreement scale. Below are the results:

TextAnalyticsTrumpOrder1.png

Given a five-point scale, the most popular answer by respondents (36%) was “strongly disagree.” Interestingly, the least popular choice was “somewhat disagree” (6.6%).

Collapsing “strongly” and “somewhat” (see chart below) we found 4% more Americans (43%) disagree with Trump’s Executive Order than agree with it (39%). A sizeable number (18%) indicated they aren’t sure/don’t know.

Trump-Text-Analytics-2.png

Will It Unstructure? - A Text Analytics PollTM

Next, we asked another 1500 respondents from the same U.S. nationally online representative source* EXACTLY the same question, but instead of providing choices for them to select from, we asked them to reply in an open-ended comment box in their own words.

We ran the resulting comments through OdinText, with the following initial results:

Trump-OdinText.png

As you can see, the results from the unstructured responses were remarkably close to those from structured question. In fact, the open-ended responses suggest Americans are slightly closer to equally divided on the issue, though slightly more disagree (a statistically significant percentage given the sample size).

This, however, is where the similarities between unstructured and structured data end.

While there is nothing more to be done with the Likert scale data, the unstructured question data analysis has just begun…

Low-Incidence Insights are Hardly Incidental

It’s worth noting here that OdinText was able to identify and quantify many important, but low-incidence insights—positive and negative— that would have been treated as outliers in a limited code-base and dismissed by human coders:

  • “Embarrassment/Shame” (0.2%)

  • “Just Temporary” (0.5%)

  • “Un-American” (0.9%)

  • “Just Certain/Specific Countries” (0.9%)

  • “Unconstitutional/Illegal” (2%)

  • “Not a Muslim Ban/Stop Calling it that” (2.9%)

An Emotionally-Charged Policy

EMOTIONAL-SENTIMENT-ANALYSIS-TRUMP.png

It shouldn’t come as a surprise to anyone that emotions around this particular policy run exceptionally high.

OdinText quickly quantified the emotions expressed in people’s comments, and you can see that while there certainly is a lot of anger—negative comments are spread across anger, fear/anxiety and sadness—there is also a significant amount of joy.

What the heck does “joy” entail, you ask? It means that enough people expressed unbridled enthusiasm for the policy along the lines of, “I love it!” or “It’s about time!” or “Finally, a president who makes good on his campaign promises!”

Understanding the Why Behind People’s Positions

Last, but certainly not least, asking the same question in an open-ended format where respondents can reply in their own words enables us to also understand why people feel the way they do.

We can then quantify those sentiments using text analytics and see the results in context in a way that would not have been possible using a multiple-choice format.

Here are a few examples from those who disagree with the order:

  • “Just plain wrong. It scored points with his base, but it made all Americans look heartless and xenophobic in the eyes of the world.”

  • “Absolutely and unequivocally unconstitutional. The foundation, literally the reason the first European settlers came to this land, was to escape religious persecution.”

  • “I don't like and it was poorly thought out. I understand the need for vetting, but this was an absolute mess.”

  • “I think it is an overly confident action that will do more harm than good.”

  • “I understand that Trump's intentions mean well, but his order is just discriminating. I fear that war is among us, and although I try my best to stay neutral, it's difficult to support his actions.”

Here are a few from those who agree:

  • “I feel it could have been handled better but I agree. Let’s make sure they are here documented correctly and backgrounds thoroughly checked.”

  • “I feel sometimes things need to be done to demonstrate seriousness. I do feel bad for the law abiding that it affects.”

  • “Initially I thought it was ridiculous, but after researching the facts associated with it, I'm fine with it. Trump campaigned on increasing security, so it shouldn't be a surprise. I think it is reasonable to take a period of time to standardize and enforce the vetting process.”

  • “I feel that it is not a bad idea. The only part that concerns me is taking away from living the American Dream for those that aren’t terrorists.”

  • “good but needed more explanation”

  • “OK with it - waiting to see how it pans out over the next few weeks”

  • “I think it is good, as long as it is temporary so that we can better vet those who would come to the U.S.”

And just as importantly, yet oft-overlooked those who aren’t completely sure:

  • “not my circus”

  • “While the thought is good and just for our safety, the implementation was flawed, much like communism.”

Final Thoughts: What Have we Learned?

First of all, we saw that the results in the open-ended format replicated those of the structured question. With a total sample of 3000, these results are statistically significant.

Second, we found that while emotions run high for people on both sides of this issue, comments from those who disagree with the ban tended to be more emotionally charged than from those who agreed with the ban. I would add here that some of the former group tended not to distinguish between their feelings about President Trump and the policy.

We also discovered that supporters of the ban appear to be better informed about the specifics of the order than those who oppose it. In fact, a significant number of the former group in their responses took the time to explain why referring to the order as “a Muslim ban” is inaccurate and how this misconception clouds the issue.

Lastly, we found that both supporters and detractors are concerned about the order’s implementation.

Let me know what you think. I’d be happy to dig into this data a bit more. In addition, if anyone is curious and would like to do a follow-up analysis, please contact me to discuss the raw data file.

@TomHCAnderson

Ps. Stay tuned for Part II of this study, where we’ll explore what the rest of the world thinks about the order!

*Note: Responses (n=3,000) were collected via Google Surveys. Google Surveys allow researchers to reach a validated (U.S. General Population Representative) sample by intercepting people attempting to access high-quality online content—such as news, entertainment and reference sites—or who have downloaded the Google Opinion Rewards mobile app. These users answer up to 10 questions in exchange for access to the content or Google Play credit. Google provides additional respondent information across a variety of variables including source/publisher category, gender, age, geography, urban density, income, parental status, response time as well as google calculated weighting. Results are +/- 1.79% accurate at the 95% confidence interval.

Top 2017 New Year’s Resolutions Text Analyzed (In Their Own Words)

Will it Unstructure? Part I of a New Series of Text Analytics Tests Happy New Year!

As I was preparing to celebrate the New Year with my family and pondering the year ahead, my mind wandered to all of those Top New Year’s Resolutions lists that you see the last week in December every year. It seems to me that the same resolutions with very similar incidence populate those lists each year, usually with something around diet and/or exercise as the most popular resolution.

After spending several minutes investigating, it occurred to me that these lists are almost always compiled using quantitative instruments with static choice answers pre-defined by researchers—therefore limited in options and often biased.

Here’s a good example of a study that has been repeated now for a few years by online financial institution GOBankingRates.com.

While their 2017 survey was focused solely on financial resolutions, their 2016 survey was broader and determined that “Live Life to The Fullest” was the most popular resolution (45.7%), followed by “Live a Healthier life” (41.1%) etc. [see chart below].

NewYearsRessolutionsStructured-300x188.png

The question I had, of course, was what would this look like if you didn’t force people to pick from a handful of arbitrary, pre-defined choices?

Will It Unstructure?

You may be familiar with the outlandish but wildly popular “Will it Blend?” video series by Blendtec, where founder Tom Dickson attempts to blend everything from iPhones to marbles. It’s a wacky, yet compelling way to demonstrate how sturdy these blenders are!

Well, today I’m announcing a new series of experiments that we’re calling “Will it Unstructure?

The idea here is to take structured questions from surveys, polls and so forth we come across and ask: Will it Unstructure? In other words, will asking the same question in an open-ended fashion yield the same or different results?

(In the future, we’ll cover more of these. Please send us suggestions for structured questions you’d like us to test!)

Will New Year’s Resolutions Unstructure? A Text Analytics PollTM

So, back to those Top New Year’s Resolution lists. Let’s find out: Will it Unstructure?

Over New Year’s weekend we surveyed n=1,536 respondents*, asking them the same question that was asked in the GoBankingRates.com example I referenced earlier: “What are your 2017 resolutions?”

*Representative online general population sample sourced via Google Surveys.

Below is a table of the text comments quickly analyzed by OdinText.

WillItUnstructure1OdinText.png

As you can see, there’s a lot more to be learned when you allow people to respond unaided and in their own words. In fact, we see a very different picture of what really matters to people in the coming year.

Note: The GoBankingRates.com survey allowed people to select more than one answer.

Predictably, Health (Diet and/or Exercise) came in first, but with a staggeringly lower incidence of mentions compared to the percent of respondents who selected it on the GoBankingRates.com survey: 19.4% vs. 80.7%.

Moreover, we found that ALL of the top resolution categories in the GoBankingRates.com example actually appeared DRAMATICALLY less frequently when respondents were given the opportunity to answer the same question unaided and in their own words:

  • “Living life to the fullest” = 1.1% vs. 45.7%

  • Financial Improvement (make/save more and/or cut debt) = 2.9% vs. 57.6%

  • Spend more time with family/friends = 0.2% vs. 33.2%

Furthermore, the second most-mentioned resolution in our study didn’t even appear in the GoBankingRates.com example!

What we’ll call “Spirituality” here—a mix of sentiments around being kinder to others, staying positive, and finding inner peace—appeared in 8.3% of responses, eclipsing each of the top resolutions from the GoBankingRates.com example except diet/exercise.

After that we see a wide variety of equally often mentioned and sometimes contradictory resolutions. Now, bear in mind that some of these responses—“Drink more alcohol,” for example—were probably made tongue-in-cheek. Interestingly, even in those cases, more than one person said the same thing, which suggests it may mean something more. (I.e., could this have been filed under “Have Fun/Live Life to the Fullest”?)

These replies are all low incidence, sure, but they certainly provide a fuller picture. For instance, who would’ve predicted that “getting a driver’s license/permit” or “getting married” would be a New Year’s resolution? I would add that among these low incidence mentions, a text analysis a way to understand the relative differences in frequency between various answers.

Disturbingly, 0.3% (five people) said their 2017 resolution is to die. Whether or not these responses were in jest or serious is debatable. Our figure is coincidentally not so far off from estimates from reputable sources with expertise on the subject. For example, according to Emory University, in the past year approx. 1.0% of the U.S. population (2.3 million people) developed a suicide plan and 0.5% (1 million people) attempted suicide.

All of this said, obviously the GoBankingRate.com survey was not a scientific instrument. We selected it at random from a lot of similar “Top New Year’s Resolutions” surveys available.

These results are all, of course, relatively subject to interpretation and we can debate them on a number of fronts, but at the end of the day it’s unmistakably clear that a quantitative instrument with a finite set of choices tells an entirely different story than people do when they have the opportunity to respond unaided and in their own words.

Bonus: Top Three Most Important Events of 2016

Since the whole New Year’s resolutions topic is a little overdone, I ran an additional question just for fun: “Name the Three Most Important Things That Happened in 2016.”

Here are the results from OdinText ranked in order of occurrence in 2016..

MostMemorableEventsOf2016textanalysis.png

If I had to answer this question myself I would probably say Donald Trump winning the U.S. Presidential Election, Russian aggression/hacking and Brexit.

But, again, not everyone places the same weight on events. So here’s yet another example of how much more we can learn when we ask people to reply unaided, in their own words.

Thanks for reading!

REMINDER: Let me know what questions you would like us to use for future posts on the “Will it Unstructure?” series!

Wishing you and yours a happy, healthy new year!

@TomHCAnderson

Tom H. C. Anderson

Five Reasons to NEVER Design a Survey without a Comment Field

Marketing Research Confessions Part II - Researchers Say Open-Ends Are Critical!

My last post focused on the alarmingly high number of marketing researchers (~30%) who, as a matter of policy, either do not include a section for respondent comments (a.k.a. “open-ended” questions) in their surveys or who field surveys with a comment section but discard the responses.

The good news is that most researchers do, in fact, understand and appreciate the value of comment data from open-ended questions.

Indeed, many say feedback in consumers’ own words is indispensable.

Among researchers we recently polled:

  • 70% would NEVER launch tracker OR even an ad-hoc (66%) survey without a comment field
  • 80% DO NOT agree that analyzing only a subset of the comment data is sufficient
  • 59% say comment data is AT LEAST as important as the numeric ratings data (and many state they are the most important data points)
  • 58% ALWAYS allocate time to analyze comment data after fielding

In Their Own Words: “Essential”

In contrast to the flippancy we saw in comments from those who don’t see any need for open-ended survey questions, researchers who value open-ends felt pretty strongly about them.

Consider these two verbatim responses, which encapsulate the general sentiment expressed by researchers in our survey:

“Absolutely ESSENTIAL. Without [customer comments] you can easily draw the wrong conclusion from the overall survey.”

“Open-ended questions are essential. There is no easy shortcut to getting at the nuanced answers and ‘ah-ha!’ findings present in written text.”

As it happens, respondents to our survey provided plenty of detailed and thoughtful responses to our open-ended questions.

We, of course, ran these responses through OdinText and our analysis identified five common reasons for researchers’ belief that comment data from open-ended questions is critically important.

So here’s why, ranked chronologically in ascending order by preponderance of mentions and in their own words

 Top Five Reasons to Always Include an Open-End

 

#5 Proxy for Quality & Fraud

“They are essential in sussing out fraud—in quality control.”

“For data quality to determine satisficing and fraudulent behavior

“…to verify a reasonable level of engagement in the survey…”

 

#4 Understand the ‘Why’ Behind the Numbers

“Very beneficial when trying to identify cause and effect

“Open ends are key to understand the meaning of all the other answers. They provide context, motivations, details. Market Research cannot survive without open ends”

Extremely useful to understand what is truly driving decisions. In closed-end questions people tend to agree with statements that seem a reasonable, logical answer, even if they have not considered them before at all

“It's so critical for me to understand WHY people choose the hard codes, or why they behave the way the big data says they behave. Inferences from quant data only get you so far - you need to hear it from the horse’s mouth...AT SCALE!”

“OEs are windows into the consumer thought process, and I find them invaluable in providing meaning when interpreting the closed-ended responses.”

 

#3 Freedom from Quant Limitations

“They allow respondents more freedom to answer a question how they want to—not limited to a list that might or might not be relevant.”

“Extremely important to gather data the respondent wants to convey but cannot in the limited context of closed ends.”

“Open-enders allow the respondent to give a full explanation without being constrained by pre-defined and pre-conceived codes and structures. With the use of modern text analytics tools these comments can be analyzed and classified with ease and greater accuracy as compared to previous manual processes.”

“…fixed answer options might be too narrow.  Product registration, satisfaction surveys and early product concept testing are the best candidates…”

allowing participants to comment on what's important to them

 

#2 Avoiding Wrong Conclusions

“We code every single response, even on trackers [longitudinal data] where we have thousands of responses across 5 open-end questions… you can draw the wrong conclusion without open-ends. I've got lots of examples!”

“Essential - mitigate risk of (1) respondents misunderstanding questions and (2) analysts jumping to wrong conclusions and (3) allowing for learnings not included in closed-ended answer categories”

“Open ended if done correctly almost always generate more right results than closed ended.  Checking a box is cheap, but communicating an original thought is more valuable.”

 

#1 Unearthing Unknowns – What We Didn’t Know We Didn’t Know

“They can give rich, in-depth insights or raise awareness of unknown insights or concerns.”

“This info can prove valuable to the research in unexpected ways.”

“They are critical to capture the voice of the customer and provide a huge amount of insight that would otherwise be missed.”

“Extremely useful.  I design them to try and get to the unexpected reasons behind the closed-end data.”

“To capture thoughts and ideas, in their own words, the research may have missed.”

“It can give good complementary information. It can also give information about something the researcher missed in his other questions.”

“Highly useful. They allow the interviewee to offer unanticipated and often most valuable observations.”

 

Ps. Additional Reasons…

Although it didn’t make the top five, several researchers cited one other notable reason for valuing open-ended questions, summarized in the following comment:

“They provide the rich unaided insights that often are the most interesting to our clients

 

Next Steps: How to Get Value from Open-Ended Questions

I think we’ve established that most researchers recognize the tremendous value of feedback from open-ended questions and the reasons why, but there’s more to be said on the subject.

Conducting good research takes knowledge and skill. I’ve spent the last decade working with unstructured data and will be among the first to admit that while the quality of tools to tackle this data have radically improved, understanding what kind of analysis to undertake, or how to better ask the questions are just as important as the technology.

Sadly many researchers and just about all text analytics firms I’ve run into understand very little about these more explicit techniques in how to actually collect better data.

Therefore I aim to devote at least one if not more posts over the next few weeks to delve into some of the problems in working with unstructured data brought up by some of our researchers.

Stay tuned!

@TomHCAnderson

 

Ignoring Customer Comments: A Disturbing Trend

One-Third of Researchers Think Survey Ratings Are All They Need

You’d be hard-pressed to find anyone who doesn’t think customer feedback matters, but it seems an alarming number of researchers don’t believe they really need to hear what people have to say!

 

2in5 openends read

In fact, almost a third of market researchers we recently polled either don’t give consumers the opportunity to comment or flat out ignore their responses.

  • 30% of researchers report they do not include an option for customer comments in longitudinal customer experience trackers because they “don’t want to deal with the coding/analysis.” Almost as many (34%) admit the same for ad hoc surveys.
  • 42% of researchers also admit launching surveys that contain an option for customer comments with no intention of doing anything with the comments they receive.

Customer Comments Aren’t Necessary?

2 in 5 researchers it is sufficient to analyze only a small subset of my customers comments

Part of the problem—as the first bullet indicates—is that coding/analysis of responses to open-ended questions has historically been a time-consuming and labor-intensive process. (Happily, this is no longer the case.)

But a more troubling issue, it seems, is a widespread lack of recognition for the value of unstructured customer feedback, especially compared to quantitative survey data.

  • Almost half (41%) of researchers said actual voice-of-customer comments are of secondary importance to structured rating questions.
  • Of those who do read/analyze customer comments, 20% said it’s sufficient to just read/code a small subset of the comments rather than each and every

In short, we can conclude that many researchers omit or ignore customer comments because they believe they can get the same or better insights from quantitative ratings data.

This assumption is absolutely WRONG.

Misconception: Ratings Are Enough

I’ve posted on the serious problems with relying exclusively on quantitative data for insights before here.

But before I discovered text analytics, I used to be in the same camp as the researchers highlighted in our survey.

My first mistake was that I assumed I would always be able to frame the right questions and conceive of all possible relevant answers.

I also believed, naively, that respondents actually consider all questions equally and that the decimal point differences in mean ratings from (frequently onerous) attribute batteries are meaningful, especially if we can apply a T Test and the 0.23% difference is deemed “significant” (even if only at a directional 80% confidence level).

Since then, I have found time and time again that nothing predicts actual customer behavior better than the comment data from a well-crafted open-end.

For a real world example, I invite you to have a look at the work we did with Jiffy Lube.

There are real dollars attached to what our customers can tell us if we let them use their own words. If you’re not letting them speak, your opportunity cost is probably much higher than you realize.

Thank you for your readership,

I look forward to your COMMENTS!

@TomHCAnderson

[PS. Over 200 marketing researchers professionals completed the survey in just the first week in field (statistics above), and the survey is still fielding here. What I was most impressed with so far was ironically the quality and thought fullness of the two open ended comments that were provided. Thus I will be doing initial analysis and reporting here on the blog during the next few days. So come back soon to see part II and maybe even a part III of the analysis to this very short but interesting survey of research professionals]