Posts tagged Polling
Marketing Research Blooper Reveals Lots of Surprises and Two Important Lessons

April Foolishness: What Happens When You Survey People in the Wrong Language?

I’m going to break with convention today and, in lieu of an April Fool’s gag, I’m going to tell you about an actual goof we recently made that yielded some unexpected but interesting results for researchers.

As you know, last week on the blog we highlighted findings from an international, multilingual Text Analytics Poll™ we conducted around culture. This particular poll spanned 10 countries and eight languages, and when we went to field it we accidentally sent the question to our U.S. sample in Portuguese!

Shockingly, in many cases, people STILL took the time to answer our question! How?

First, bear in mind that these Text Analytics Polls™ consist of only one question and it’s open-ended, not multiple choice. The methodology we use intercepts respondents online and requires them to type an answer to our question before they can proceed to content they’re interested in.

Under the circumstances, you might expect someone to simply type “n/a” or “don’t understand” or even some gibberish in order to move on quickly, and indeed we saw plenty of that. But in many cases, people took the time to thoughtfully point out the error, and even with wit.

Verbatim examples [sic]:

“Are you kidding me, an old american who can say ¡adios!”

“Tuesday they serve grilled cheese sandwiches.” “What the heck is that language?”

“No habla espanol”

“i have no idea what that means”

“2 years of Spanish class and I still don't understand”

Others expressed themselves more…colorfully…

“No, I don't speak illegal immigrant.”

“Speak English! I'm switching to News 13 Orlando. They have better coverage than FT.”

Author’s note:I suspect that last quote was from someone who was intercepted while trying to access a Financial Times article. ;-)

While a lot of people clearly assumed our question was written in Spanish, still others took the time to figure out what the language was and even to translate the question!

“I had to use google translate to understand the question.”

“what the heck does this mean i don't speak Portuguese”

But what surprised me most was that a lot of Americans actually answered our question—i.e., they complied with what we had asked—even though it was written in Portuguese. And many of those replies were in Spanish!!!

We caught our mistake quickly enough when we went to machine-translate the responses and we were told that replies to a question in Portuguese were now being translated from English to English, but two important lessons were learned here:

Takeaway One: Had we made this mistake with a multiple-choice instrument, we either might not have caught it until after the analysis or perhaps not at all. Not only would respondents not have been able to tell us that we had made a mistake, but they would’ve had the easy option of just clicking a response at random. And unless those random clicks amounted to a conspicuous pattern in the data, we could’ve potentially taken the data as valid!

Takeaway Two: The notion that people will not take the time to thoughtfully respond to an open-ended question is total bunk. People not only took the time to answer our question in detail when it was correctly served to them in their own language, but they even spared a thought for us when they didn’t understand the language!

I want to emphasize here that if you’re one of those researchers (and I used to be among this group, by the way) who thinks you can’t include an open-ended question in a quantitative instrument, compel the respondent to answer it, and get a meaningful answer to your question, you are not only mistaken but you’re doing yourself and your client a huge disservice.

Take it from this April fool, open-ended questions not only tell you what you didn’t know; they tell you what you didn’t know you didn’t know.

Thanks for reading. I’d love to hear what you think!

@TomHCAnderson

P.S. Find out how much more value an open-ended question can add to your survey using OdinText. Contact us to talk about it.

About Tom H. C. Anderson

Tom H. C. Anderson is the founder and managing partner of OdinText, a venture-backed firm based in Stamford, CT whose patented SAS platform is used by Fortune 500 companies like Disney, Coca-Cola and Shell Oil to mine insights from complex, unstructured and mixed data. A recognized authority and pioneer in the field of text analytics with more than two decades of experience in market research, Anderson is the recipient of numerous awards for innovation from industry associations such as CASRO, ESOMAR and the ARF. He was named one of the "Four under 40" market research leaders by the American Marketing Association in 2010. He tweets under the handle @tomhcanderson.

When Oprah is President We Can Celebrate Family Day While Skiing!

Text Analytics Poll™ Shows What We’d Like Instead of Presidents Day It’s been less than a week since our Valentine’s Day poll unearthed what people dislike most about their sweethearts, and already another holiday is upon us! Though apparently for most of us it’s not much of a holiday at all; well over half of Americans say they do nothing to commemorate ‘Presidents Day.’

You’ll note I put the holiday in single quotes. That’s because there’s some confusion around the name. Federally, it’s recognized as Washington’s Birthday. At the state level, it’s known by a variety of names—President’s Day, Presidents’ Day, Presidents Day and others, again, depending on the state.

But the name is not the only inconsistency about Presidents Day. If you’re a federal employee OR you happen to be a state employee in a state where the holiday is observed OR you work for an employer who honors it, you get the day off work with pay. Schools may or may not be closed, but that again depends on where you live.

As for what we’re observing exactly, well, that also depends on the state, but people generally regard the holiday as an occasion to honor either George Washington, alone, or Washington and Abraham Lincoln, or U.S. presidents, in general.

Perhaps the one consistent aspect of this holiday is the sales? It’s particularly popular among purveyors of automobiles, mattresses, and furniture.

Yes, it’s a patriotic sort of holiday, but on the whole, we suspected that ‘Presidents Day’ fell on the weaker end of the American holiday spectrum, so we investigated a little bit…

About this Text Analytics Poll™

In this example for our ongoing series demonstrating the efficiency, flexibility, and practicality of the Text Analytics Poll™ for insights generation, we opted for a light-hearted poll using a smaller sample* than usual. While text analytics have obvious value when applied to larger-scale data where human reading or coding is impossible or too expensive, you’ll see here that OdinText also works very effectively with smaller samples!

I’ll also emphasize that the goal of these little Text Analytics Polls™ is not to conduct a perfect study, but to very quickly design and field a survey with only one open-ended question, analyze the results with OdinText, and report the findings in here on this blog. (The only thing that takes a little time—usually 2-3 days—is the data collection.)

So while the research is representative of the general online population, and the text analytics coding applied with 100% consistency throughout the entire sample, this very speedy exercise is meant to inspire users of OdinText to use the software in new ways to answer new questions. It is not meant to be an exhaustive exploration of the topic. We welcome our users to comment and suggest improvements in the questions we ask and make suggestions for future topics!

Enough said, on to the results…

A Holiday In Search of a Celebrant in Search of a Holiday…

Poll I: Americans Celebrate on the Slopes, Not in Stores

When we asked Americans how they typically celebrate Presidents Day, the vast majority told us they don’t. And those few of us lucky enough to have the day off from work tend to not do much outside of sleeping.

But the surprise came from those few who actually said they do something on Presidents Day!

We expected people to say they go shopping on Presidents Day, but the most popular activity mentioned (after nothing and sleeping) was skiing! And skiing was followed by 2) barbecuing and 3) spending time with friends—not shopping.

Poll II: Change it to Family Day?

So, maybe as far as holidays go, Presidents Day is a tad lackluster? Could we do better?

We asked Americans:

Q. If we could create a new holiday instead of Presidents Day, what new holiday would you suggest we celebrate?

While some people indicated Presidents Day is fine as is, among those who suggested a new holiday there was no shortage of creativity!

The three most frequently mentioned ideas by large margins for replacement of Presidents Day were 1) Leaders/Heroes Day, 2) Native American Day (this holiday already exists, so maybe it could benefit from some publicity?) and 3) Family Day (which is celebrated in parts of Canada and other countries).

People also seemed to like the idea of shifting the date and making a holiday out of other important annually occurring events that lent themselves to a day off in practical terms like Election Day, Super Bowl Monday and, my personal favorite, Taxpayer Day on April 15!

Poll III: From Celebrity Apprentice to Celebrity POTUS

Donald Trump isn’t the first person in history to have not held elected office before becoming president, but he is definitely the first POTUS to have had his own reality TV show! Being Presidents Day, we thought it might be fun to see who else from outside of politics might interest Americans…

 Q: If you could pick any celebrity outside of politics to be President, who would it be?

 

Looks like we could have our first female president if Oprah ever decides to run. The media mogul’s name just rolled off people’s tongues, followed very closely by George Clooney, with Morgan Freeman in a respectable third.

Let Them Tell You in Their Own Words

In closing, I’ll remind you that none of these data were generated by a multiple-choice instrument, but via unaided text comments from people in their own words.

What never ceases to amaze me about these exercises is how even when we give people license to say whatever crazy thing they can think up—without any prompts or restrictions—people often have the same thoughts. And so open-ends lend themselves nicely to quantification using a platform like OdinText.

If you’re among the lucky folks who have the holiday off, enjoy the slopes!

Until next time, Happy Presidents Day!

@TomHCAnderson

PS.  Do you have an idea for our next Text Analytics Poll™? We’d love to hear from you. Or, why not use OdinText to analyze your own data!

[*Today’s OdinText Text Analytics PollTM sample of n=500 U.S. online representative respondents has been sourced through of Google Surveys. The sample has a confidence interval of +/- 4.38 at the 95% Confidence Level. Larger samples have a smaller confidence level. Subgroup analyses within the sample have a larger confidence interval.]

About Tom H. C. Anderson

Tom H. C. Anderson is the founder and managing partner of OdinText, a venture-backed firm based in Stamford, CT whose eponymous, patented SAS platform is used by Fortune 500 companies like Disney, Coca-Cola and Shell Oil to mine insights from complex, unstructured and mixed data. A recognized authority and pioneer in the field of text analytics with more than two decades of experience in market research, Anderson is the recipient of numerous awards for innovation from industry associations such as CASRO, ESOMAR, and the ARF. He was named one of the “Four under 40” market research leaders by the American Marketing Association in 2010. He tweets under the handle @tomhcanderson

Mush Aside, Top Things that Bug Us about Our Sweethearts

Valentine’s Day Text Analytics Poll™ Uncovers What People Really Think about their Special Someones Ah, February 14th—the one day of the year when anything you want to eat comes shaped in a heart!

Yes, love is in the air today, friends. We show it with cards and candy, roses and Build-a-Bears, theater tickets, dinner reservations and sexy unmentionables…

On Valentine’s Day we express our true feelings for that special someone!

Well, maybe we profess our love, at least. But what do we hold back?

The truth is, anyone who’s been in one can tell you that relationships aren’t all sweet nothings and love songs. (Except mine. My wife is a reader. I love you, honey! You’re perfect!)

So, at the risk of cynicism on the ultimate Hallmark holiday, we decided to get a little real here with some help from OdinText. We asked each of three randomly-selected gen pop samples of 1,500 a question designed to give us the good, bad or the ugly about their sweethearts—stuff you might not find in a card store.

What Irks Me about You

I love being married. It's so great to find one special person you want to annoy for the rest of your life. — Rita Rudner 

We asked 1,500 Americans to reply to the following question in their own words in a comment box:

“What do you like least about your significant other?”

Omitting a significant number of liars and honeymooners who said “nothing,” here are the responses after running all 1,500 comments through OdinText…

Ok, who did not predict “hair” would be the number three response? Especially against common relationship problems like money/financial issues and lack of communication (which came out relatively low)?

What I Bring to this Relationship

My friends tell me I have an intimacy problem. But they don't really know me.  — Garry Shandling

Valentine’s Day is all about appreciating the way our sweethearts enrich our lives, but we all like to think we have something special to offer in return.

We asked people one of two questions:

  1. “How would your significant other describe you to a good friend?”

Or

  1. “How would you describe your significant other to a good friend?”

Here’s what they told us, with responses to question 1 in white and question 2 in red:

As you can see from the results, it appears a lot of us consider our sweethearts to be our best friends and we value them for their love and support. But we’re twice as likely to describe ourselves as being the funny one in the relationship (and the crazy one, too).

Life is Lonely without You

If we take matrimony at its lowest, we regard it as a sort of friendship recognized by the police. —Robert Louis Stevenson

They say distance makes the heart grow fonder, so we asked people this:

“If you were without your significant other for an entire week, what things would you miss the most?”

And there you have it. I think the responses to this question show overwhelmingly why our sweethearts put up with so much from us.

Wishing you and your special someone a lovely Valentine’s Day!

And for you single folks out there, take heart: You’re not only saving money today; you’re not shackled to a snoring, messy, flatulent sweetheart with a bad attitude.

@TomHCanderson

PS. Could a Text Analytics Poll™ answer your burning marketing questions?  Contact us to see if a single-question open-ended survey makes sense for you! Using OdinText is easy.

About Tom H. C. Anderson

Tom H. C. Anderson is the founder and managing partner of OdinText, a venture-backed firm based in Stamford, CT whose eponymous, patented SAS platform is used by Fortune 500 companies like Disney, Coca-Cola and Shell Oil to mine insights from complex, unstructured and mixed data. A recognized authority and pioneer in the field of text analytics with more than two decades of experience in market research, Anderson is the recipient of numerous awards for innovation from industry associations such as CASRO, ESOMAR and the ARF. He was named one of the "Four under 40" market research leaders by the American Marketing Association in 2010. He tweets under the handle @tomhcanderson.

Who Are You Voting Against?

Text Analysis Shows Dislike May Decide Presidential Election (A Text Analytics PollTM ) Exit pollsters today will ask thousands of Americans “Who did you vote for?” when they probably should be asking “Who did you vote against?”

A survey we just completed suggests that the outcome of the 2016 U.S. Presidential Election may hinge on which candidate is disliked more intensely by the other side.

One simple question posed interchangeably for the candidates produced such an unexpectedly visceral emotional reaction that one could reasonably conclude a vote for either candidate in many cases may be primarily about preventing the other candidate from being elected.

More than Just the Lesser of Two Evils

They’re both unpopular. We knew that already.

A slew of polls going back to the start of the general election and most recently by Washington Post/ABC News have repeatedly indicated that Hillary Clinton and Donald Trump are the two least popular candidates for U.S. president in the history of political polling.

What conventional, multiple-choice polling does not reveal, although it certainly supports this conclusion, is that apparently this election will not just be a matter of just holding one’s nose and voting for the lesser of two evils.

Unaided responses to one open-ended question analyzed using OdinText suggest that what may drive many voters to cast their ballots for either candidate today is an intense distaste for the alternative.

People’s distaste for each candidate is so intense that when asked to tell us what Hillary Clinton and Donald Trump stands for, respectively, respondents didn’t name a policy issue, they named a character flaw.

Top of Mind: The Crook and the Hatemonger

We took a general population sample* of 3000 Americans via Google surveys, split it in half randomly, and asked each half the same single question substituting only the candidate’s name:

“Without looking, off the top of your mind, what issues does [insert candidate name] stand for?”

The comments—presumably the issues that are truly top of mind for people in this election—were analyzed with OdinText and are captured in the chart below.

odintexttrumpclintonissues

You’ll note that for each candidate (red for Trump, blue for Clinton), respondents frequently offered a negative character perception instead of naming a political issue or policy supported by the candidate.

Indeed, the most popular response for Hillary Clinton involved the perception of dishonesty/corruption and the third most popular response for Donald Trump was perceived racism/hatemongering.

In both cases, the data tell us that people are unusually fixated on perceived problems they have with the candidates personally.

More Joy, Less Anger for Trump

Though the responses tended to be short and direct, a look at the words used to describe the candidates provides a pretty clear picture of the emotions associated with each candidate.

The OdinText visualization below shows the most striking emotional differences between Clinton and Trump around respondents’ levels of joy and anger. [See OdinText Emotions Plot Below, Trump Red, Clinton Blue]

clintonvstrumptextanalytics

While descriptions for both candidates exhibit a lot of anger, the proportion of anger in comments for Clinton is significantly higher (16.4% VS 12.3% for Trump).

The higher level of joy identified in this analysis is partly due to Trump’s positive campaign slogan “Make America Great Again,” which had significantly higher recall among respondents than Clinton’s slogan “Stronger Together.” Among those surveyed, 33 people in our sample specifically referenced the Trump slogan, while only one person referenced Clinton’s slogan—a notable difference in percentage terms (2.2% vs 0.07%, respectively)

More Effective Messaging for Trump

In terms of actual issues identified by respondents, Clinton was most often associated with championing women and civil rights, while Trump was identified with immigration and a pro-America, protectionist platform.

Here one could argue that the Trump campaign has done a more effective job of establishing a signature issue for the candidate.

While neither campaign has done a significantly better job of educating voters on its candidate’s policies than the other (8.2% vs 8.6% for Trump and Clinton, answering “I don’t know”), it may be that the simple message of “Make America Great Again” has clearer meaning to people than Clinton’s “Stronger Together.”

Indeed, the top issue identified for Trump was immigration (12.8% VS 2.3% for Clinton), while the number one issue for Clinton was the negative trait “corruption/lies” (12.5% VS. 1.4% for Trump).

This may prove problematic for the Clinton camp.

When voters don’t like their choices, they tend to stay home. If voter turnout is high today, it won’t be because people are unusually enthusiastic about the candidates; it will be because one of these candidates is so objectionable that people can’t in good conscience abstain from voting.

@TomHCAnderson

 

[*Note: N=3,000 responses were collected via Google Surveys 11/5-11/7 2016. Google Surveys allow researchers to reach a validated (U.S. General Population Representative) sample by intercepting people attempting to access high-quality online content—such as news, entertainment and reference sites—or who have downloaded the Google Opinion Rewards mobile app. These users answer up to 10 questions in exchange for access to the content or Google Play credit. Google provides additional respondent information across a variety of variables including source/publisher category, gender, age, geography, urban density, income, parental status, response time as well as google calculated weighting. All 3,000 comments where then analyzed using OdinText to understand frequency of topics, emotions and key topic differences. Out of 65 topics total topics identified using OdinText 19 topics were mentioned significantly more often for Clinton, and 21 topics were significantly more often mentioned for Trump. Results are +/- 2.51% accurate at the 95% confidence interval. ]