Posts in Sentiment Analysis
Advertising Effecitveness +OdinText

ad testing +OdinText [Authors note. As I am writing this blog post early Monday morning after the Super Bowl, I have already completed the initial ad testing analysis. It’s the case where modern AI and analytics software (OdinText) is faster than the data collection process/vendor we’re relying on. We’ve asked an open ended comment question among n=3,000 respondents about which super bowl ads they like/dislike and why. Eager to have the analysis complete as soon as possible, the analysis is already done, and blog written based on n=1,011 initial responses received. But since 1,998 more are expected I’m painfully waiting to publish results until the rest of the fielding comes in. The bad part is waiting for the sample. The good part is knowing that now repeating the analysis will literally take less than 1 minute. Just uploading the data into OdinText, and then the brand names and advertisement likes and dislikes will automatically be coded, analyzed and charted in seconds. I just have to review if anything has changed materially and make small updates in my copy below in such case. As it turned out, more data did change findings, and so I did have to change my blog copy. Ah, the joys of modern analytics!]

 

The Advertising Pundits weighed in on which ads were best and worst even before the Super Bowl aired. We tend to do things a little differently at OdinText and allow data, not opinion to drive.

Of course, for "best" and "worst" not to be subjective, we need some definition of desired outcomes. Last year we looked at a simple formula to evaluate efficacy consisting of Awareness + Positive Sentiment/Liking of the ads.

For instance, you may not remember this because of the low sentiment, but last year 85 Lumber was one of the companies with the highest Awareness after the super bowl. However, because it also had low sentiment and relevancy (as it dealt with the explosive issue of immigration/Trump's wall in a somewhat ambiguous way). It's probably the case that it ended up doing better among its core customer segments than among the general population, but since Super Bowl Ads are expensive, I argued that all things equal, a strategy with a broader target in mind, which aims to leave a positive impact among this broader group, should provide a better ROI. Looking at it another way, to have the most significant positive impact we want to maximize both awareness and sentiment almost equally.

With those assumptions and comments from over 3,000 respondents, OdinText's AI predicted which of the Super Bowl Ads were successful, and which were not. Below I've shown 10 Brands/ads, the best performing 6 and the worst 4.

OdinText Ad Ratings

10 superbowl ads rated

THE WINNERS

#1. THE NFL

https://www.youtube.com/watch?v=KUoD-gPDahw

 

In a year where there has been a lot of controversy surrounding NFL players taking a knee, and with a few of our respondents explicitly stating that they had boycotted the Super Bowl this year, it interesting to see the NFL advertising, and doing it so well. The NFL's Dirty Dancing with Manning and Beckham performed best, I believe in part because of its high relevance to the audience, but also for garnering high awareness together with very high positive sentiment/liking. In fact, only one other ad came close in sentiment.

#2 AMAZON ALEXA

https://www.youtube.com/watch?v=J6-8DQALGt4

 

That second most well-liked ad was Amazon's Alexa. Not as much because of its awareness (which was rather low in comparison), but because of its extremely high sentiment. The audience loved the various famous actors playing the voice of Alexa at least as much as they enjoyed NFL players Dirty Dancing.

#3 TIDE

https://www.youtube.com/watch?time_continue=1&v=doP7xKdGOKs

In 3rd place we have Tide. They earned the spot less so for sentiment (though viewers did like the ad). The reason Tide did so well was primarily due to the awareness it garnered. Tide had THE HIGHEST awareness of any Super Bowl Ad. However contrary to some of the Advertising Pundits opinions, it just wasn't quite as consistently well-liked by viewers as Dirty Dancing, and Alexa.

If you are in the camp who believe Awareness is everything, then Tide should have an even higher spot.

#3 DORITOS (& MOUNTAIN DEW ICE)

https://www.youtube.com/watch?time_continue=10&v=4eKYR_iL5eU

Doritos + Mountain Dew Ice was so close in our model, that I’m going to give them a tie for 3rd. Not awareness like Tide, but for balancing both positive sentiment and awareness perfectly. It's mix of awareness and liking was in the same proportions as NFL Dirty Dancing, just at a slightly smaller scale.

Obviously considering the audience and occasion, just like the NFL ad, Doritos especially is a highly relevant product, and as importantly the humorous approach with two extremely popular yet not commonly seen together stars (Namely Morgan Freeman and Peter Dinglage/Tyron from Game of Thrones) succeeded in the unique Combo messaging of Fire & Ice.

#5 BUD-LIGHT (NOT BUDWEISER)

https://www.youtube.com/watch?v=CxGUmtRLm5g

https://www.youtube.com/watch?v=lvTE_5c7buk

 

Budweiser is almost expected to do well. So, in a way, it may be surprising to see it doesn’t make it into our analysis (Beer, in general, did poorly especially Miller Ultra). It really should be so easy for Budweiser though. Here's a case where the occasion is more than just relevant, it's almost as if the brand has a historic Super Bowl halo effect. That said, their performance was less than impressive.

While the idea of stopping the Budweiser line to make water in an emergency could be touching for some, reminding consumers you have a good fun product may be a safer strategy than asking for kudos for merely being a good corporate citizen?

And that’s where Bud-Light’s Knight did better. Beer should be about fun…

#6 TOYOTA

https://www.youtube.com/watch?v=QwF3ipuNyfc

Here's a case where awareness was quite low, but the ad was still more liked than average compared to the other brands. Toyota and our #8 brand just barely made the list. While the setting was right "The Super Bowl," in the end perhaps the ‘Priest, an Imam, a Monk and a Rabbi' may have felt a bit less like a joke, and more like preaching…

THE LOSERS

#1 COCA-COLA

https://www.youtube.com/watch?v=-R-EEdvDrUU

Like Budweiser, we expect a lot from Coca-Cola when it comes to advertising. They’ve been pushing the diversity message for a few years now. It may be that pulling at heart strings is far harder to do than making people laugh. Coca-Cola had lower than average sentiment coupled with relatively low awareness. Not a winning combination.

#2 DODGE RAM

https://www.youtube.com/watch?v=SlbY1tGARUA

 

Dodge Ram did better than Coca-Cola at least, especially on awareness, but even concerning sentiment/liking.

The negative aspect of course in large part was the appropriateness of Martin Luther King's message at the beginning.

When it comes to ads like these though, I think we must assume, as was the case for 85 Lumber last year, that perhaps the brand knows what it's doing. They aren't there to please everyone (as you would hope is the goal of Pepsi and Coca-Cola), but to message their core audience with a ‘We Get You – Even if Everyone Else Doesn't'. And so, awareness wise, Ram did better than Amazon, Bud-Light, and Pepsi. But on an overall basis, they get dinged by the overall sentiment due to the some would say clumsy ‘MLK + Patriotic' messaging. Only time and sales will tell…

#3 T-MOBILE

https://www.youtube.com/watch?v=C-rumHvmqCA

T-Mobile was less well liked than you’d think, who doesn’t like babies right? Turns out people are getting tired of the “social responsibility ads” in their entertainment, at least that’s what they told us.

#4 DIET COKE (TWISTED MANGO)

https://www.youtube.com/watch?v=T6T6YrPA4aM

The Booby Prize. Ok, so here is a bad ad. In PR they used to say, any PR is good PR. But Diet Coke didn't do too well on either of our metrics. It had low awareness combined with even lower sentiment/liking. Diet Coke Mango, because, just no…

 

HALF TIME

A WORD ABOUT PEPSI

https://www.youtube.com/watch?v=2z3EUY1aXdY

https://www.youtube.com/watch?v=0gHYd67OumQ

Pepsi, what can I say.  You may be surprised that yet again, Pepsi performed poorly compared to the other brands I mentoned considering that their name was all over the Super Bowl during the Half Time Show. And yet, it may be that the real winner of Halftime is the brand of the performer, which this year was Justin Timberlake. We saw a similar pattern last year as well.

 

@TomHCAnderson

PS. See Your Data +OdinText

 

Best 10 Text Analytics Tips Posts of The Year

Our Top 10 Most Read Data and Text Mining Posts of 2017

Thank you for reading our blog this year. The OdinText blog has quickly become even more popular than the Next Gen Market Research blog, and I really appreciate the thoughtful feedback we’ve gotten here on the blog, via Twitter, and email.

In case you’re curious, here are the most popular posts of the year:

#10 NFL Players Taking a Knee is More Complex and Polarizing Than We Think If a Topic is Worth Quantifying – It’s Also Worth Understanding The Why’s Behind It

#9 Text Analytics Picks The 10 Strongest Super Bowl Ads New Text Analytics Poll Shows Which Super Bowl Ads Really Performed Best

#8 Why Your HR Survey is a Lie and How to Get The Truth OdinText Discovers Job Satisfaction Drivers in Anonymous Employee Data

#7 Of Tears & Text Analytics (An OdinText User Story – Text Analytics Guest Post (AI Meets VOC))

#6 65 CEO’s Share Thoughts on Insights (Insights Associations Inaugural CEO Summit – A Future Tied to Collaboration and Technology)

#5 Why Machine Learning is Meaningless (Beware of Buzzwords! The Truth about ‘Machine Learning’ and ‘Artificial Intelligence’)

#4 Do You Speak Teen? OdinText Announces 2nd Annual List of Top 10 Slang Terms (How Text Analytics Can Help Marketers Move at the Speed of Slang)

#3 Text Analysis Reveals Potential French Election French Election Upset (Text Analytics Poll Showed How Close Le Pen Came to ‘Trumping’ Macron)

#2 Text Analytics Poll: Why We Unfriend on Facebook (You Can’t Handle The Truth (And Other Top Reasons Why We Unfriend on Facebook)

#1 What Americans Really Think About Trump’s Immigration Ban and Why (Text Analysis of What People Say in Their Own Words Reveals More Than Multi-Choice Surveys)

 

I thought I’d also check what our top 5 posts were from last year as well, here they are in case you missed them:

Top Posts From 2016

#1 Text Analysis Answers Is The Quran Really More Violent Than The Bible (3 Parts)

#2 Attensity Sold – What Does it Mean?

#3 Customer Satisfaction Surveys: What do Satisfied VS Dissatisfied Customers Talk About?

#4 What’s Really Wrong With Polling?

#5 What Your Customer Satisfaction Research Isn’t Telling You

Thanks again for reading and commenting. As always I welcome your thoughts and questions via LinkedIn, or feel free to request info on anything you’ve read above here.

Happy New Year!

@TomHCAnderson

Labor Day Text Analytics Poll™ Spells Trouble for Labor Unions

Text Analysis Reveals Fragile Positioning of Labor Unions in 2017 America Happy Labor Day to our U.S. OdinText Users and Friends!

Over the holiday weekend we thought we’d run one of our quick Text Analytics Polls™, which highlight how a single open-ended question can be used to very quickly and accurately generate deeper insights on almost any subject compared to conventional multiple-choice surveys (with fewer questions and at comparable speed and cost).

In honor of the holiday, we thought we would ask about labor—specifically organized labor unions. We asked 1,500 Americans: “What are your thoughts and impressions of Workers Unions?” The responses—again, people’s comments in their own words—were rapidly analyzed and quantified using the OdinText advanced analytics software platform and are reported below…

Indifference Even on Labor Day

odintext poll

The results suggest organized labor in America today is precariously positioned, with 60% of respondents indicating they are either indifferent (35%) or outright opposed to labor unions (25%).

The Why’s Behind the Sentiment

Unlike conventional multiple-choice surveys, using unstructured questions and text analytics in polling enables us to not only quantify opinions, but also the reasons behind those opinions.

text poll

Most of those in support of unions don’t feel the need to elaborate much. They simply state that labor unions are a great thing and/or mention that they or a family member belongs to a union. The top reasons provided among supporters is that labor unions are a “necessity” (8.7%) and that they protect workers’ rights (4%).

Interestingly, while those who oppose unions are fewer in number than those who support them (25% vs 40%), their reasons tend to be slightly more articulated, with the most popular being that unions are “no longer useful/necessary” (9.3%), and that they are often “corrupt” (4.1%) and “foster laziness” (3.3%).

Takeaways

Labor unions have been on a gradual decline in the US for some time, and OdinText’s analysis of these comments points to a serious positioning problem, which, if left unaddressed, will probably lead to continued decline and, ultimately, irrelevance.

As many respondents indicated they are indifferent about unions as those who support them.  In politics and in marketing, indifference (or lack of loyalty) can spell death, but it also presents an opportunity to persuade. The question then becomes who has stronger, more cogent messaging in place?

Proponents are not inclined to (or could not) offer much explanation for why they support organized labor; conversely, opponents in their responses tend to offer slightly more detailed explanations, with the most frequent being that labor unions are generally obsolete. The distance between indifference and opposition (“unions are obsolete”) in this case isn’t much. So the challenge facing organized labor in America may be to justify its very existence.

That this result came from a poll conducted over Labor Day weekend, I think, speaks volumes, too.

@TomHCAnderson

PS. If you would like to learn more about how easy, fast and powerful Text Analytics Polling with OdinText can be, feel free to join our live Webinar on the 14th.

PPS. Friendly reminder, today is also the last day for NGMR Award Nominations. Consider nominating a worthy company or colleague here.

*Note: n=1,500 responses were collected  9/1-9/3 2017 via Google Surveys which allow researchers to reach a validated U.S. General Population Representative sample by intercepting people attempting to access high-quality online content. Results are +/- 2.53% accurate at the 95% confidence interval. Data was analyzed using OdinText 9/3/17. Request more info on OdinText here.

 

ABOUT ODINTEXT  The leader in Text Analytics for Marketing Research, OdinText is a patented SaaS (software-as-a-service) platform for natural language processing and text mining. For more information or to request a demo, visit Get The Job Done With OdinText!

New Sentiment Analysis Reveals Reasons Behind Stance on Confederate Statues

Text Analytics Poll™ shows asking respondents to provide reasons for their opinions may increase cognition and decrease “No Opinion”

Asking People WHY They Support/Oppose Civil War Monuments May Affect Results. Judging from the TV news and social media, the entire country is up in arms over the status of Confederate Civil War monuments. What really is the mood of the country in regard to these statues?

A quick Google search turned up the chart below, which to YouGov’s credit broke out not just Democrats VS Republicans, but also blacks VS whites. On a high level this structured survey question, which allowed respondents to answer a standard five-point agreement scale from ‘strongly approve’ to ‘strongly disapprove,’ seems to indicate that “almost half of Americans (48%)” want the Charlottesville Robert E. Lee statue to stay.

While emotions as depicted on TV and social media are running  high, there doesn’t seem to be much reasoned discussion about WHY people feel so strongly on either side. Therefore, we were curious if rather than just asking a closed-ended agreement scale, what would happen instead if respondents were asked to elaborate on their choice with a reason?

Note: the goal here is not to uncover all the best reasons for or against keeping the statues. If that was the case we could approach a handful of social science professors with expertise in history, civil rights or ethics and psychology. Instead, we were curious to see if simply asking someone to consider a reason for their choice (even if they could not give a very good one) would affect the proportions of those agreeing or disagreeing. Of course, we were also curious about how many reasons each side might enumerate and what the quality of those reasons might be.

We asked a random sample of 1,500 Americans the following:

Q. Should Confederate Civil War Monuments be allowed in the US, why or why not?

Asking respondents to provide a reason, and using Text Analytics to measure sentiment, provided an almost identical number in favor of removing Confederate Civil War statues (29%) as the simple Likert scale poll; however, it halved the number of “Don’t Know/Don’t Care” responses (just 10%), apparently to the benefit of those who support keeping Confederate Civil War statues intact (61%).

EMOTIONS VS EXPLICIT REASONS

Let’s have a look at the reasons each side provided…

First, it’s noteworthy but not surprising that a number of the comments registered high emotional valence – especially anger – among both groups. Among those who favor keeping the statues, there is also significantly more fear/anxiety expressed in their comments.

As for the specific reasons, among those who want statues to remain, ‘history’ (implicitly the preservation of) is the most frequently mentioned reason by far (46%), and that history shouldn’t be deleted (3%), and history is both Good and Bad (2%).

The main argument among those who want to remove the statues is that Confederates were losers and traitors (9%) and that these statues should be limited to museums and battle grounds (8%), that glorifying what these men stood for is wrong (6%), as well as more general mentions of its symbolism of hate or slavery (6%).

A QUICK LOOK AT REGION

We took a quick look at answers by geography. Southerners were 5% more likely than total to mention the historic importance of the statues (35% VS 30% in total). They were also half as likely to have made the argument that statues for losers/traitors aren’t appropriate (1.7% VS 2.8% in total).

Americans in the Northeast region were significantly more likely than average to say they weren’t sure or didn’t care (7% VS 5% in total), and were also significantly more likely to mention the importance of “remembering” (3% VS 1% in total).

Americans in the West Region were significantly less likely to mention the importance of ‘History’ (25% VS 30% in Total).

The Verdict Changes When Asked Why

The court of public opinion in a standard Likert scale instrument appears fairly evenly split on whether or not to remover Confederate Civil War monuments, but when we ask people to explain why they hold a position on this matter in their own words, we see a significant shift in the data toward keeping these monuments intact.

Most respondents didn’t offer any surprises in terms of their explanations for why they support/oppose keeping the monuments. Indeed, a few arguments on both sides have already been fleshed out in the media, and this may have affected how people responded.

The ah-ha for us in this exercise was that the “don’t care/don’t knows” shrank by half when respondents were asked to provide a reason for their opinion. Whether this is a matter of causality, of course, is debatable. But it does suggest that allowing people to explain in their own words will produce a different, possibly more accurate picture, as well as which reasons have strongest appeal.

@TomHCAnderson

*Note: n=1,500 responses were collected via Google Surveys 8/19-8/21 2017. Google Surveys allow researchers to reach a validated U.S. General Population Representative sample by intercepting people attempting to access high-quality online content or who have downloaded the Google Opinion Rewards mobile app. Results are +/- 2.53% accurate at the 95% confidence interval. Data was analyzed using OdinText 8/21/17. Request more info on OdinText here.

About Tom H. C. Anderson Tom H. C. Anderson is the founder and managing partner of OdinText, a venture-backed firm based in Stamford, CT whose eponymous, patented SAS platform is used by Fortune 500 companies like Disney, Coca-Cola and Shell Oil to mine insights from complex, unstructured and mixed data. A recognized authority and pioneer in the field of text analytics with more than two decades of experience in market research, Anderson is the recipient of numerous awards for innovation from industry associations such as CASRO, ESOMAR and the ARF. He was named one of the “Four under 40” market research leaders by the American Marketing Association in 2010. He tweets under the handle @tomhcanderson.

Text Analytics Explores Whether All Culture Is Becoming American? Part 3
Emotion Speaks Louder than Words Across 11 Cultures, 10 Countries and 8 Languages!

Welcome to Part 3 of our international, multilingual exploration of culture using text analytics!

In Part 1 of this series, I provided a topline analysis of comments from more than 15,500 people spanning 11 cultures in 10 countries and eight languages in response to one question:

“How would you explain <insert country> culture to someone who isn’t at all familiar with it?”

Part 2 took a deeper dive into the key similarities and differences among cultures in our study, revealing how respective members see themselves.

But things got really interesting when OdinText analyzed people’s comments for emotion. Here we have a bit of a surprise. One might expect people’s descriptions of their cultures to be generally positive and for the range of emotions to be fairly narrow, but this was hardly the case. In fact, the emotional analysis revealed much more than just people’s impressions of their own cultures; this exercise tapped into state of mind! You’ll see what I mean in the spider charts for our emotional analysis and verbatim* comments included below.

*Note: Verbatim comments are either translated or [sic]

U.S.A.  (High Positive Sentiment)

Americans are Angry! Twice as Angry as the international average. The Anger is accompanied by high levels of Fear/Anxiety and even Disgust, an emotion we don’t see often outside of food categories and which in this case appears to be related to the recent presidential election.

Joy is also lower than average (and trust is slightly below average), which begs the question: How could we also have a somewhat higher than average overall positive sentiment? The answer lies in a very polarized/divided populous, almost half of whom are bullish and joyful in their descriptions!

[USA Emotions Blue - International Mean Red]

Verbatim examples:

“Expect cordiality and indifference equally, as well as politeness and kindness that may turn to anger and malice. We are all different, and reflections of the world around us. We expect to be treated fairly and bear grudges beyond what is necessary. Racism is a dread poison that has seeped into the veins of our country. While none truly want to take the antidote. There is no standard in our country, people are all different as America breeds individuality.” – FEAR/ANXIETY (and mixed emotion)

“the expression of self in the most obnoxious form one can think of…Fat, dumb and ugly, Loud and obnoxious, Donald Trump - the ugly American” – DISGUST

“I honestly don't know what American culture is. We're such a large country, not at all homogenous. I think we have regional cultures and I would be comfortable explaining southern culture to someone. In the south, most people are neighborly, incredibly polite, and have a strong sense of pride for their region. I would have said a unifying feature of American citizenry was out unified devotion to country, but even that is questionable at times. Overall, I think it depends where in America one is.” – TRUST

“Freedom. Even with all the stuff going on, we still have the best country in the world because we have freedom of speech, choice, and worship…” – JOY

 

UNITED KINGDOM (Average Positive Sentiment)

In the UK, emotion around culture scored pretty average with one notable exception: Fear/Anxiety registered almost twice as high as the international average (although neither was as pronounced as what Americans expressed).

Verbatim examples:

Difficult to say as different parts of Britain have different cultures… difficult to understand Polite hypocritical compassionate confusing people” – FEAR/ANXIETY

“We have a prime minister we didn't elect, England messed up Scotland independence, Brexit is a disaster but we never give up.” – FEAR/ANXIETY

Unsure, confused and variedIt's dead” – FEAR/ANXIETY

[NOTE: comments like “unsure, confused and varied” is a common theme in many of the cultural descriptions, not just for the UK]

AUSTRALIA (Very High Positive Sentiment)

Australians described their culture as laid back, and the emotions they expressed back it up. Their comments contained far less (about half as much) Anger than the international average, lower Sadness and significantly higher Joy. Australian comments also don’t reflect much Surprise, with very few using terms such as “amazing.” Comments are more often relaxed (and often mention this term).

Verbatim examples:

“Its full of kindness, reslectfuly, courageness and happy” - JOY

“limited. But great mateship” – JOY

“Inclusive, relaxed, full of laughter” – JOY

“Laid back, relaxed and able to laugh at ourselves” – JOY

 

BRAZIL (Low Positive Sentiment)

Even though Carnival was a frequently mentioned feature in descriptions of Brazilian culture, life for Brazilians isn’t one big party. Brazilians’ culture comments are significantly more likely than average to contain Anger. They also contain fewer Trust mentions. Most of these sentiments involved frustration with corruption and/or crime. Paradoxically, at the same time, we found low instances of Anticipation and Fear/Anxiety, indicating Brazilians have somewhat resigned themselves or have grown accustomed to these conditions. Moreover, Joy is neither significantly lower nor higher than the international average.

Verbatim examples:

“…Because of the [income] distribution … very Robin Hood, ie acceptable to steal from large companies and also the government. So bank robberies without victims are not perceived negatively by the population, stealing TV signals, tax evasion, political and corruption in general is high, there is strong prejudice against the poor. unqualified civil servants are lazy (stealing their government salaries) High use of pesticides in food, eliminating its nutrients.”  – ANGER (multiple examples)

“A mixture of cultures, and now with evil people in charge making it very difficult to live with the current culture” – ANGER

“good. I believe in Brazil, that one day it will be great” - JOY

 

FRANCE (Average Positive Sentiment)

French comments contain less Surprise than average. In other words, they are less likely than average to use terms like “amazing” and “extraordinary” to describe their culture. This may be because French culture, conceptually, is so familiar and established in the minds of the French, yet the opposing emotion to Surprise—Anticipation—is also not significantly higher than average. French comments describing their culture are also somewhat less likely than average to contain Anger.

Verbatim example:

“We cannot explain French culture. We can only share its ideology, although doing so has evident limitations. I consequently, and personally, see it as wealth gained by mixing cultures: extraordinary traditions gained through the people who have lived here before us. – SURPRISE (rare example)

 

MEXICO (High Positive Sentiment)

Mexicans exhibit a high level of positivity in describing their culture, with their comments containing almost twice the amount of Joy as the international mean. Similarly, their Anger is also almost half that of the ten-country aggregate. Mexicans are also notable for their amount of Surprise—almost three times the average!

Verbatim examples:

“It is very rich and has many very beautiful and amazing things, traditions are super beautiful and have much biodiversity” – JOY and SURPRISE

“As a wonderful and amazing and different gift to what can be seen elsewhere in the world.” – JOY and SURPRISE

“Full of diversity and incredible things that transport you back in time to a magical place” – SURPRISE

“Mexican culture as a set of traditions and art that defined not only the beauty but the feeling of the nation is very particular as we have a very cheerful culture.” – JOY

 

SPAIN (Low Positive Sentiment)

When describing Spanish culture, the Spanish are three times less likely than others to mention issues related to Trust. Surprisingly, they also exhibit almost twice the average level of Sadness. And importantly, we found significantly higher amounts of Anger in Spanish comments about their culture, often related to corruption.

Verbatim Examples:

“For me, the Spanish culture is summed up in the torture of an animal (bull) and very rich food like potato omelette and paella” - ANGER

“bulls, crisis, corruption, political thieves, injustice, cachondeo” – ANGER

“Culture rather low and in many cases ridiculous. Eat and drink like monkeys and hang as much as possible with whoever is around. Idiots, political vermin, thieves and plunderers posing as big cahunas, big wealthy guys, the magnates of oil companies. These guys at the oil companies, they are just clowns but because they work there they become very wealthy, they steal and get a lot of money from the oil companies. They are thieves, corrupt. They become rich. They call this success. Like Rafa Mora or Belen Esteban they are very mediocre people in this country. I am ashamed of these people.” – ANGER and SADNESS

 

GERMANY (Low Positive Sentiment)

Germans have far less positive sentiment in descriptions and about half the proportion of Joy compared to the international average. Like the French, there is also very little Surprise in their comments. It’s not that negative emotions like Anger and Sadness are significantly higher, but rather the lack of positive emotions is significant.

Verbatim Examples:

“Well organized, industrious, intelligent, technically well developed.” – JOY (infrequent example of German Joy)

“Conservative, many rules, precise but also pleasure in little things, family” – JOY (infrequent example of German Joy)

 

JAPAN (Very Low Positive Sentiment)

By “Very Low Positive Sentiment” we do not mean that Japanese sentiment was negative, but that the Japanese sentiment was absent. The Japanese are very reserved and conservative, so it should come as no shock that the degree to which they expressed emotions, generally, was significantly lower than average.

Verbatim Examples:

“Though it is the culture of an island isolated by sea, it is special in its ability to ‘mimic’, and therefore it has developed into a simultaneously unique and multifaceted culture” – JOY

“The origins of the great culture of the Samurai” – JOY

“Japanese culture is special in that it is mellow and refined and is characterized by many gorgeous things. For example, tea ceremony, calligraphy, flower arranging, etc., at first appear to be quite quiet and plain endeavors, however such an impression belies a perfection and refined beauty that exists therein.” – JOY

“Japanese culture is a culture of hospitality and care” – TRUST

 

CANADA-ENGLISH (High Positive Sentiment)

Peace of mind doesn’t appear to be much of a problem for English-speaking Canadians, whose comments reflected significantly low Anger and high Trust. They also exhibited significantly less Fear/Anxiety than the international average, and a modestly higher level of JOY.

Verbatim examples:

“Look great from the outside, is great on the inside. But does have its flaws. Not to mention prejudice, inequality and racism is still embedded in large portions of our culture. Media also does a great job of covering stories that don't matter and are not actually informing.” – JOY (mixed/modest)

“Canadians are usually warm and welcoming people. We are mostly immigrants and understand peoples needs and desires to strive for a better life. We tend to supposrt one another yet respect peoples privacy.” - TRUST

“Like America only with gun control, socialized health care, and French on the packaging. And a much cuter leader.” – TRUST

“Friendly, fair, safe and welcoming” – TRUST

 

CANADA-FRENCH (Lower Positive Sentiment)

The Quebecois’ level of Joy is significantly higher than the international average, but it’s accompanied by equally high levels of Anger and Fear/Anxiety. This combination was unique in our data, perhaps as it represents a strong, well-understood and distinct culture that is defensively positioned within a larger, somewhat opposing culture that sometimes feels threatening. Comments from French Canadians—in contrast to those of the actual French from France—contained quite a lot of emotion. There were also significantly higher levels of Trust, and Sadness scored slightly above average.

Verbatim examples:

“People welcoming, open and proud. rich and diverse culture.” – JOY

“Mixture of French European roots in a North American context. Culture which developed from a difficult kind, hard winter. But a warm and supportive culture, proud of its language on an English-speaking continent.” – JOY

“A welcoming culture, which focuses on French and fights for its rights. Who are past present and future is important. Which is multi-ethnic” – ANGER

“people proud of its language and its history. Quebec is slightly open, yet desperate to preserve its values.” – FEAR/ANXIETY

“We are tolerant but do not humiliate us. Our history is full of situations where we have been crushed but wounds heal slowly. We are proud revelers but we lack confidence in us. We need to assert ourselves in the world and we are receiving from everyone so obviously there is no danger for us.” – FEAR/ANXIETY

What Have We Learned?

First of all, thank you to everyone for the incredible interest you’ve shown and for joining us on this journey!

While I’ll leave the final word on the cultural impact of globalization to anthropologists and others specializing in the study of culture, this surface-level read strongly suggests that we are becoming more alike. Multiculturalism, in particular, has become an important component of cultural identities across many countries and cultures. The data also obviously show that 1. significant differences endure, 2. their dimensions and 3. the degree to which they matter.

Somewhat surprisingly, the hero today may have been the emotional analysis, which told us that cultural identity is not necessarily a static construct, and that how people think about their culture at a given point in time is strongly influenced by current affairs and circumstances, hence the variation in emotions expressed and their intensity.

But what’s really striking about this exercise is that we were able to run these analyses and visualizations and glean all of these insights from data collected from a SINGLE open-ended question.

Look at how much we learned!

Imagine for a moment trying to collect this same information using a multiple-choice instrument. You’d need more than one, and I still don’t think it would be possible to achieve the same insights.

Then there’s the scale to consider. We analyzed responses from more than 15,500 people in their own words.

Lastly, we accomplished this using OdinText across 11 cultures, 10 countries and eight languages in fewer than two hours! (It actually took longer to prepare this blog post than it did to translate and analyze the data!)

In summary, research innovation today is generally assessed in three questions:

  • Is it better? Yes! This approach yielded insight that would have been impossible to achieve with a conventional, multiple-choice survey.
  • Is it faster? Yes! Manual coding alone would’ve taken days or weeks. OdinText did it in fewer than two hours.
  • Is it cheaper? Yes! This international project was affordable enough to conduct on a whim.

Whatever the size of your organization or your resources, this project demonstrates that you can now conduct, translate and analyze a multinational, multilingual study among key consumers in key markets and capture meaningful insights quickly, affordably and easily without even getting up from your desk using OdinText.

Contact us here to talk about it.

Thanks for reading. I’d love to hear what you think!

@TomHCAnderson (@OdinText)

About Tom H. C. Anderson

Tom H. C. Anderson is the founder and managing partner of OdinText, a venture-backed firm based in Stamford, CT whose eponymous, patented SAS platform is used by Fortune 500 companies like Disney, Coca-Cola and Shell Oil to mine insights from complex, unstructured and mixed data. A recognized authority and pioneer in the field of text analytics with more than two decades of experience in market research, Anderson is the recipient of numerous awards for innovation from industry associations such as CASRO, ESOMAR and the ARF. He was named one of the "Four under 40" market research leaders by the American Marketing Association in 2010. He tweets under the handle @tomhcanderson.

Text Analytics: It's Not Just for BIG Data

In a world focused on the value of Big Data, it's important to realize that Small Data is meaningful, too, and worth analyzing to gain understanding. Let me show you with a personal example. If you're a regular reader of the OdinText blog, you probably know that our company President, Tom Anderson, writes about performing text analytics on large data sets.  And yes, OdinText is ideal for understanding data after launching a rapid survey then collecting thousands of responses.

However for this blog post, I'm going to focus on the use of Text Analytics for smaller, nontraditional data set:  emails.

SMALL Data (from email) Text Analytics

I recently joined OdinText as Vice President, working closely with Tom on all our corporate initiatives. I live in a small town in Connecticut with an approximate population of 60,000.  Last year I was elected to serve our town government as an RTM member along with 40 other individuals.  Presently, our town's budget is $290M and the RTM is designing the budget for the next year.

Many citizens email elected members to let them know how they feel about the budget.  To date, I have received 280 emails. (Before you go down a different path with this, please know that I respond personally to each one -- people who take the time to write me deserve a personal response.  I did not and will not include in this blog post how I intend to vote on the upcoming budget, nor will I include anything about party affiliations. And I certainly will not share names.)

As the emails were coming in, I started to wonder … what if I ran this the data I was receiving through OdinText?  Would I be able to use the tool to identify, understand and quantify the themes in the people’s thoughts on how I should vote on the budget?

The Resulting Themes from Small Data Analytics

A note about the methodology:  Each email that I received contained the citizen's name, their email address and content in open text format.  Without a key driver metric like OSAT, CSAT or NPS to analyze the text against, I chose to use overall sentiment. Here is what I learned

Emails about the town budget show that our citizens feel Joy but RTM members need to recognize their Sadness, Fear and Anger

Joy:

“I have been a homeowner in Fairfield for 37 years, raised 4 kids here and love the community.”

Sadness:

“I am writing you to tell you that I am so unhappy with the way you have managed our town.”

Fear:

“My greatest concern seems to be the inability of our elected members to cut spending and run the town like a business”

Anger:

“We live in a very small house and still have to pay an absurd amount of money in taxes.”

Understanding the resulting themes in their own words

Reduce Taxes (90.16%)

“Fairfield taxes are much higher than surrounding communities.”

“Fairfield taxes are out of line with similar communities”

“The town has to stop raising taxes at such a feverish rate.”

“High taxes are slowly eroding the town of Fairfield.”

Moving if Taxes are Increased (25.13%)

“I am on a fixed income at 64, and cannot afford Fairfield’s taxes now. Please recognize that I cannot easily sell my house, due to the economy & the amount of homes on the market here”

“regret to say most of our colleagues and friends have an "exit strategy" to leave Fairfield”

“Our town is losing residents who are fed up and have moved or are moving to Westport and other towns with lower mil rates”

Reduce Spending (33.33%)

“... bring spending under control”

“Stop the spending please”

“... needs to trim fat at the local level, cut services, stop spending money”

“We need to keep taxes down as much as possible - even if it means spending cuts.”

Education ‘don’t cut’ (8.74%)

“… takes great pride in its education system”

“… promise of an excellent public education”

“… fiscal responsibility; however, not at the expense of the children and their right to an excellent education.”

Education ‘please cut’ (9.83%)

“Let's shave funding from all programs including education”

“... deeply questioning our education budget”

“... reduce the Education budget”

“I have a cherished budgetary item that I want protected--the library. Cut that last, after you cut education, police, official salaries”

Big Value from Small Data in Little Time

I performed this text analysis in 30 minutes. Ironically, it has taken me longer to write this blog post than it did to quantify the text from all those emails. Yet the information and understanding I have gleaned will empower me as I make decisions on this important topic. A small investment in small data has paid off in a BIG way.

Tim Lynch - @OdinText

Share Your Text Analytics Success with us at The Sentiment Analysis Symposium

Emotion—Influence—Activation: Call for Speakers, 2017 Sentiment Analysis Symposium  Writing today to OdinText users as well as other fellow practitioners, especially those on the client side.

I’m working with Seth Grimes Chairman of the Sentiment Analysis Symposium to get the call out for speakers as well as panelists for an interesting and interactive discussion at the event this Summer.

OdinText has been a long time supporter of the event which this year takes place June 27-28 in New York. The Seniment Analysis Symposium tackles the business value of sentiment, opinion, and emotion in our big data world.

Emotion is one of the keys to customer (and patient, voter, and market) understanding. The symposium is _the_ place to stay current with the technologies and their research and insights applications. Please join us in June, as either an attendee or presenter...

The key to a great conference is great speakers. Whether you're a business visionary, experienced user, technologist, or consultant, please consider presenting. You may submit your proposal here. Choose from among the suggested topics or surprise us. Help us build on our track record of bringing attendees useful, informative technical and business content (along with excellent networking opportunities). Submit by January 31 if possible.

We're inviting talks that focus on customer experience, brand strategy, market research, media & publishing, social insights, healthcare, and financial markets. On the tech side, show off what you know about natural language processing, machine learning, speech and emotion AI, and the data economy.

Please help us create another great symposium! I look forward to seeing you at the event. Feel free to reach out if you have any questions. @TomHCAnderson

About Tom H. C. Anderson

Tom H. C. Anderson is the founder and managing partner of OdinText, a venture-backed firm based in Stamford, CT whose eponymous, patented SAS platform is used by Fortune 500 companies like Disney, Coca-Cola and Shell Oil to mine insights from complex, unstructured and mixed data. A recognized authority and pioneer in the field of text analytics with more than two decades of experience in market research, Anderson is the recipient of numerous awards for innovation from industry associations such as CASRO, ESOMAR and the ARF. He was named one of the “Four under 40” market research leaders by the American Marketing Association in 2010. He tweets under the handle @tomhcanderson

Text Analytics Picks the 10 Strongest Super Bowls Ads

New Text Analytics PollTM Shows Which Super Bowl Ads Really Performed Best Well, it’s been five days since the Super Bowl, and pretty much everyone has cranked out a “definitive” best-and-worst ad list or some sort of top 10 ranking. And frankly, I think a lot of them are based on the wrong metrics.

Without a doubt, what makes a Super Bowl ad great differs from what makes a “normal” ad great. So what exactly qualifies a Super Bowl ad as a success or failure?

We could look at purchase consideration or intent, likelihood to recommend, or any of a dozen or more other popular advertising metrics, but that’s not what Super Bowl advertising effectiveness is really about.

Word of mouth has always been a big one and nowadays that means social media buzz. But does buzz equate to success? Ask the folks at Budweiser or Lumber 84.

Bottom line: This is a very expensive reach buy, first and foremost, and it’s a branding exercise.  I’ve shelled out $5 million (plus production costs) for 30 seconds to make a lasting and largely unconscious impression on the world’s biggest television audience.

As far as I’m concerned there need only be three objectives then:

  1. I want you to remember the ad;
  2. I want you to remember it’s my ad;
  3. I want you to feel positive about it.

Whether or not my ad met all of these criteria can be answered with one single unstructured question in a Text Analytics PollTM and quickly be analyzed by NLP software like OdinText with more valid results than any multiple-choice instrument.

Why a Text Analytics PollTM ?

Using a Likert scale to assess recall or awareness will only provide an aided response; I can’t ask you about an ad or brand without mentioning it. So I don’t really know if the ad was actually that memorable. And while a quantitative instrument can tell me whether or not you liked or disliked an ad, it also won’t tell me why.

Conversely, I can get the “why” from traditional qualitative tools like focus groups or IDIs, but not only would those insights be time-consuming, labor-intensive and expensive to gather, they wouldn’t be quantified.

But if I ask you to just tell me what you remember in your own words using a comment box, I can find out which ad was truly memorable, ascertain whether or not you truly recall the brand, determine whether the ad left a positive or negative impression on you and get a much deeper understanding of why. I can achieve all of this using one open-ended question. And with text analytics software like OdinText, I can quantify these results.

Which Super Bowl Ads Did “Best”?

We asked a random, gen pop sample of n=4,535 people (statistics with a confidence interval of +/- 1.46) one simple question:

“What Super Bowl ad stood out the most to you and why?”

Author’s note: We ran this survey Sunday night and closed it Monday night. We were originally planning to post the results on Tuesday, but decided to postpone it in favor of sharing what we felt were more pressing results from a Text Analytics PollTM we had conducted around President Trump’s immigration ban.

As you can see in the table below, this one simple question told us everything we needed to know…

Top 10 Super Bowl Ads: Memorability of Ad & Brand, and Degree of Positive Sentiment

The following ads are ranked according to memorability—respondents’ unaided recall of both the ad and the brand—accompanied by positive/negative sentiment breakout (blue for positive, orange for negative) in reverse order. Author’s note: The verbatim examples included here are [sic]

#10 Pepsi

 

 

As the sponsor of the Lady Gaga halftime show, one might expect Pepsi to do very well, but Lady Gaga may have literally stolen the show from Pepsi! In fact, the halftime show was actually mentioned more often in the comment data than Pepsi, and the two were infrequently mentioned together. Meanwhile, Pepsi’s ads were relatively unmemorable and much of the awareness we saw was in the form of negative sentiment.

Author’s note: Interestingly, social media monitoring services like Sprinkler had reported Pepsi “owned” the Super Bowl ad chatter on social media. I’ll say it not for the first time: social media (aka Twitter) can be full of spam often generated by agents of the brand.

 

#9 Buick

This is a case where the star of the ad, Cam Newton, didn’t eclipse the sponsor. People liked the pro footballer playing with the little kids and the tie-in to football seemed to work well. We saw this with Tom Brady in a different ad, too.

Buick with Cam Newton, cute and funny

I like the Buick ad because it let a bunch of kids play football with Cam Newton.

So what’s not to like, you say? How did it garner even a 13% unfavorable rating?

cam newton pushed little kids

The buick commercial, the concept was boring

Buick, it was not even funny

 

#8 Skittles

 

Skittles, made my kids laugh

The Skittles ad because it was funny and sort of relate-able. It shows how far one is willing to do something for someone.

Humor generally always does well, so what’s not to like?

The skittles commercial it made no sense

skittles, stupid with the burglar

Skittles, it was creepy. And what was with the gopher at the end?

 

#7 T-Mobile

Popular and a little risqué… [Note Also, Sprint Ads were often mis-remembered as T-Mobile, perhaps Halo effect and a reason Sprint didn't make the Top 10...]

The T-Mobile ‘fake your own death to escape Verizon bill’ it was very funny, and got its point across very well

T-mobile. very funny parodying 50 shades of gray to Verizon ‘screwing its customers!’

T-Mobile with Justin. Maybe because I'm a T-Mobile subscriber? Or Justin Bieber was dressed so well in a suit, and then he starts dancing and jumping like a maniac. The contrast makes it funny.

T mobile add where guy faking death. Most memorable. Light hearted. Got point across.

BUT not everyone is a Belieber

The t mobile justin biber. It was kinda lame

T-Mobile w/Justin Bieber - inane, juvenile, bordering on insulting

T-Mobile Unlimited Moves. It wasnt funny and Justin Bieber looked like the six flags guy.

T-Mobile, awkward dancing as they attempted to appeal to teenagers

 

#6  Audi

Audi took on gender equality with an appeal to fathers of daughters. The resulting ad was memorable in 6th place:

The audi one because it was meaningful

Audi - moving story and loved the message of what to tell daughters!

Audi. I have a daughter

Audi - moving story and loved the message of what to tell daughters!

However, not everyone liked mixing politics or social issues with their football (as we will see again for some of the other top ads):

AUDI and 84 Lumber. Keep your political message out of my entertainment

Least liked Audi because it was a liberal ad

 

#5 Coca-Cola

Ironically, even without sponsoring the halftime show, Coca-Cola beat Pepsi.

The coke commercial was really meaningful and symbolic

Coca Cola because of the embracing of diversity

Coca Cola True portrayal of America's diversity

The coke ad. I liked the pro-refugee stance.

coke america is beautiful commercial, very admirable

Coca Cola Commercial because it's all about being connected

Coke , showing we are still interconnected regardless of ethnicity

I liked the coca cola ad at the very beginning. I've seen it before but I think the message is so powerful and the commercial is beautifully executed.

But the ad was not received well by many, likely in part due to the politically-charged climate. Several advertisers ran messages that struck people as being politically biased or advancing a political agenda—something not everyone cared for…

Didn’t appreciate Coca Cola trying to make a political statement

I didn't like the Coke commercial. They showed it two years ago and the year before.

Google and Coke because they shoved their political views into my face.

 

#4 Mr. Clean

Who would have predicted MR. Clean for fourth place? The brand made good use of humor, and it stood out from the other ads by targeting women (but appealed to members of both genders).

Mr clean, it was funny - Female

Mr. Clean because I'm bald -  Male

Mr. Clean, relatable, memorable, hilarious. -Female

The Mr. Clean commercial, it was funny, tasty, and got the point across. Incredibly well done ad. – Male

Mr clean because my wife pointed it out – Male

mr clean because it relates to family, and parents that stay at home and clean. it was family friendly - Female

mr clean everything else sucked – Gender Not Specified

Some men though didn’t see the humor and or get the point, calling it “weird”. It wasn’t really that they disliked it intensely; they just felt it wasn’t for them.

 

#3 Lumber 84

Not many had heard of this company before the Super Bowl, but I’ll bet you know who they are now. The third most memorable ad, yes, but more than half of those who remembered it had nothing nice to say!

First, among those who liked the ad:

It was so touching

Audi, 84 lumber, both showed compassionate ads

84 lumber - it's the only one I can remember

84 Lumber - Showed what America is actually supposed to be.

they were obviously trying to get across a non- traditional message that didn't seem to be advertising. Also it was beautifully and compellingly produced.

Lumber 84 showed that not everyone wants a wall and that we understand there is power in diversity.

But the execution confused people and whatever the intention, the sponsor stepped into a controversy. Here the emotional sentiment (particularly anger) ran high and was prevalent in comments like “romanticized crime” and “forced politics”:

The Journey 84 ad, it just left me confused

The 84 lumber commercial. It didn't make sense

it was about illegals sneaking into America, i won't be shopping their anymore

Lumber 84 because it was politically offensive

84 lumber, clearly a political statement and uncalled for

84 lumber, Made no sense, Not going to look something up

#2 Kia

Ironically, with other brands going serious and political, Kia poked some fun with help from Melissa McCarthy. Kia’s investment in humor and McCarthy paid off in a big way, scoring the highest combination of memorability and positive sentiment, although to an extent the comedian eclipsed the brand.

Loved melissa McCarthy because she is hilarious and i love her.

Kia it was funny and not somber like most the others

The Buick one, the world of tanks ones and the eco friendly Melissa one because they were the funniest

The one with Melissa McCarthy because it made me laugh

KIA becuase it didn't feel like it was trying to sell me anything, just entertain with brand placement

 

#1 Budweiser

Yes, Budweiser took first place in terms of recall, but the perception of a political bent cost the king of beers. The ad, which featured one of the founders struggling as an immigrant, was apparently in the works before the Trump Immigration Order controversy. But even if that was the case, by choosing to air it Budweiser took a risk.

Likes:

I liked the Budweiser commercial reminded us all that not all white Europeans were always welcome in the US.

Budweiser. I love the reminder that we are all immigrants

Budweiser immigration. Shows Trump is an idiot, but we all know that

The Budweiser ad about how they were founded by an immigrant, because it was actually relevant to their company history

Budweiser, it was a beautiful immigrant's tale. Not overtly political

The Budweiser commercial because it shows what a true immigrant had to go through and even though many people thought it was to take a shot at Trump's travel ban it had nothing to do with it.

Dislikes:

Budweiser. Too liberal.

budweiser, too pro immigration

bud, adolfus was not ILLEGAL !

The Budweiser ad about immigration. Too political.

Budweiser, they shot themselves in the foot being that the man who immigrated into the U.S. did so legally.

Budweiser. Football/all sports should not involve politics. We need to relax sometimes.

So…who won?

Isn’t it obvious? I’d say Kia. Sure, Budweiser scored higher unaided awareness, but a significant portion of that was negative.

But it's all in the data, what do you think?

A Final Note on Text Analytics PollsTM 

It occurred to me in writing this post that about 11 years ago almost to the day I predicted that the survey of the future would be a one-question open-end, because that’s all people really want to tell you, and that’s all you’ll need.

Turns out I may have been right.

This week, we’ve shared results from three such surveys, a technique we've dubbed “Text Analytics PollTM .

These incredibly short, one-question polls allow us to field quickly to large samples with minimal burden on the respondent. And text analysis software such as OdinText enables us to quantify these huge quantities of comments.

But the real advantage to using text analytics polls is that the responses tell us so much more than whether someone agrees/disagrees or likes/dislikes. Using text analytics we can uncover why from respondents in their own words.

Thanks again for reading!

@TomHCAnderson @OdinText

Could a text analytics poll answer your burning marketing questions?  Contact us to see if a single-question open-ended survey makes sense for you!

 

About Tom H. C. Anderson

Tom H. C. Anderson is the founder and managing partner of OdinText, a venture-backed firm based in Stamford, CT whose eponymous, patented SAS platform is used by Fortune 500 companies like Disney, Coca-Cola and Shell Oil to mine insights from complex, unstructured and mixed data. A recognized authority and pioneer in the field of text analytics with more than two decades of experience in market research, Anderson is the recipient of numerous awards for innovation from industry associations such as CASRO, ESOMAR and the ARF. He was named one of the "Four under 40" market research leaders by the American Marketing Association in 2010. He  tweets under the handle @tomhcanderson.

 

 

 

Poll: What Other Countries Think of Trump’s Immigration Order

Text Analytics PollTM Shows Australians, Brits, and Canadians  Angry About Executive Order Temporarily Barring Refugee (Part II of II)In my previous post, we compared text analysis of results from an open-ended survey instrument with a conventional Likert-scale rating poll to assess where 3,000 Americans really stand on President Trump’s controversial executive order temporarily barring refugees and people from seven predominately-Muslim countries from entering the U.S.

Today, we’re going to share results from an identical international study that asked approx. 9,000 people—3,000 people from each of three other countries—what they think about the U.S. immigration moratorium ordered by President Trump.

But first, a quick recap…

As I noted in the previous post, polling on this issue has been pretty consistent insomuch as Americans are closely divided in support/opposition, but the majority position flips depending on the poll. Consequently, the accuracy of polling has again been called into question by pundits on both sides of the issue.

By fielding the same question first in a multiple-choice response format and a second time providing only a text comment box for responses, and then comparing results, we were able to not only replicate the results of the former but gain a much deeper understanding of where Americans really stand on this issue.

Text analysis confirmed a much divided America with those opposing the ban just slightly outnumbering (<3%) those who support the order (42% vs 39%). Almost 20% of respondents had no opinion or were ambivalent on this issue.

Bear in mind that text analysis software such as OdinText enables us to process and quantify huge quantities of comments (in this case, more than 1500 replies from respondents using their own words) in order to arrive at the same percentages that one would get from a conventional multiple-choice survey.

But the real advantage to using an open-ended response format (versus a multiple-choice) to gauge opinion on an issue like this is that the responses also tell us so much more than whether someone agrees/disagrees or likes/dislikes. Using text analytics we uncovered people’s reasoning, the extent to which they are emotionally invested in the issue, and why.

Today we will be looking a little further into this topic with data from three additional countries: Australia, Canada and the UK.

A note about multi-lingual text analysis and the countries selected for this project…

Different software platforms handle different languages with various degrees of proficiency. OdinText analyzes most European languages quite well; however, analysis of Dutch, German, Spanish or Swedish text requires proficiency in said language by the analyst. (Of course, translated results, including and especially machine-translated results, work very well with text analytics.)

Not inconspicuously, each of the countries represented in our analysis here has an English-speaking population. But this was not the primary reason that we chose them; each of these countries has frequently been mentioned in news coverage related to the immigration ban: The UK because of Brexit, Australia because of a leaked telephone call between President Trump and its Prime Minister, and Canada due to its shared border and its Prime Minister’s comments on welcoming refugees affected by the immigration moratorium.

Like our previous U.S. population survey, we used a nationally-representative sample of n=3000 for each of these countries.

Opposition Highest in Canada, Lowest in the UK

It probably does not come as a surprise to anyone who’s been following this issue in the media that citizens outside of America are less likely to approve of President Trump’s immigration moratorium.

I had honestly expected Australians to be the most strongly opposed to the order in light of the highly-publicized and problematic telephone call transcript leaked last week between President Trump and the Australian Prime Minister (which, coincidentally, involved a refugee agreement). But interestingly, people from our close ally and neighbor to the north, Canada, were most strongly opposed to the executive order (67%). The UK had proportionately fewer opposing the ban than Australia (56% vs. 60%), but the numbers of people opposed to the policy in both countries significantly lagged the Canadians. Emotions Run High Abroad Deriving emotions from text is an interesting and effective measure for understanding people’s opinions and preferences (and more useful than the “sentiment” metrics often discussed in text analytics and, particularly, in social media monitoring circles).

The chart below features OdinText’s emotional analysis of comments for each of the four countries across what most psychologists agree constitute the eight major emotion categories:

We can see that while the single highest emotion in American comments is joy/happiness, the highest emotion in the other three countries is anger. Canadians are angriest. People in the UK and Australians exhibit somewhat greater sadness and disgust in their comments. Notably, disgust is an emotion that we typically only see rarely in food categories. Here it takes the form of vehement rejection with terms such as “sickened,” “revolting,” “vile,” and, very often, “disgusted.” It is also worth noting that in cases, people directed their displeasure at President Trump, personally.

Examples:

"Trump is a xenophobic, delusional, and narcissistic danger to the world." – Canadian (anger) “Most unhappy - this will worsen relationships between Muslims and Christians.” – Australian (sadness) "It's disgusting. You can't blame a whole race for the acts of some extremists! How many white people have shot up schools and such? Isn't that an act of terror? Ban guns instead. He's a vile little man.” –Australian (disgust)

UK comments contain the highest levels of fear/anxiety:

"I am outraged. A despicable act of racism and a real worry for what political moves may happen next." – UK (fear/anxiety)

That said, it is also important to point out that there is a sizeable group in each country who express soaring agreement to the level of joy:

“Great move! He should stop all people that promote beating of women” – Australian (joy) “Sounds bloody good would be ok for Australia too!” – Australian (joy) “EXCELLENT. Good to see a politician stick by his word” – UK (joy) “About time, I feel like it's a great idea, the United States needs to help their own people before others. If there is an ongoing war members of that country should not be allowed to migrate as the disease will spread.” – Canadian (joy)

Majority of Canadians Willing to Take Refugee Overflow Given Canada’s proximity to the U.S., and since people from Canada were the most strongly opposed to President Trump’s executive order, this raised the question of whether Canadians would then support a measure to absorb refugees that would be denied entrance to the U.S., as Prime Minister Justin Trudeau appears to support.

(Note: In a Jan. 31 late-night emergency debate, the Canadian Parliament did not increase its refugee cap of 25,000.)

 

A solid majority of Canadians would support such an action, although it’s worth noting that there is a significant difference between the numbers of Canadians who oppose the U.S. immigration moratorium (67%) and the number who indicated they would be willing to admit the refugees affected by the policy.

When asked a follow-up question on whether “Canada should accept all the refugees which are turned away by USA's Trump EO 13769,” only 45% of Canadians agreed with such a measure, 33% disagreed and 22% said they were not sure.

Final Thoughts: How This Differs from Other Polls Both the U.S. and the international versions of this study differ significantly from any other polls on this subject currently circulating in the media because they required respondents to answer the question in a text comment box in their own words, instead of just selecting from options on an “agree/disagree” Likert scale.

As a result, we were able to not only quantify support and opposition around this controversial subject, but also to gauge respondents’ emotional stake in the matter and to better understand the “why” underlying their positions.

While text analysis allows us to treat qualitative/unstructured data quantitatively, it’s important to remember that including a few quotes in any analysis can help profile and tell a richer story about your data and analysis.

We also used a substantially larger population sample for each of the countries surveyed than any of the conventional polls I’ve seen cited in the media. Because of our triangulated approach and the size of the sample, these findings are in my opinion the most accurate numbers currently available on this subject.

I welcome your thoughts!

@TomHCAnderson - @OdinText

About Tom H. C. Anderson Tom H. C. Anderson is the founder and managing partner of OdinText, a venture-backed firm based in Stamford, CT whose eponymous, patented SAS platform is used by Fortune 500 companies like Disney, Coca-Cola and Shell Oil to mine insights from complex, unstructured and mixed data. A recognized authority and pioneer in the field of text analytics with more than two decades of experience in market research, Anderson is the recipient of numerous awards for innovation from industry associations such as CASRO, ESOMAR and the ARF. He was named one of the "Four under 40" market research leaders by the American Marketing Association in 2010. He tweets under the handle @tomhcanderson.

What Americans Really Think about Trump’s Immigration Ban and Why

Text Analysis of What People Say in Their Own Words Reveals More Than Multiple-Choice Surveys It’s been just over a week since President Trump issued his controversial immigration order, and the ban continues to dominate the news and social media.

But while the fate of Executive Order 13769—“Protecting the Nation from Foreign Terrorist Entry into the United States”—is being hashed out in federal court, another fierce battle is being waged in the court of public opinion.

In a stampede to assess where the American people stand on this issue, the news networks have rolled out a parade of polls. And so, too, once again, the accuracy of polling data has been called into question by pundits on both sides of the issue.

Notably, on Monday morning the president, himself, tweeted the following:

Majority Flips Depending on the Poll

It’s easy to question the accuracy of polls when they don’t agree.

Although on the whole these polls all indicate that support is pretty evenly divided on the issue, the all-important sound bite of where the majority of Americans stand on the Trump immigration moratorium flips depending on the source:

  • NBC ran with an Ipsos/Reuters poll that found the majority of Americans (49% vs. 41%) support the ban.

 

 

  • CNN publicized results from an ORC Poll with the majority opposed to the ban (53% vs. 47%).

 

  • A widely reported Gallup poll found the majority of Americans oppose the order (55% to 42%).

 

There are a number of possible reasons for these differences, of course. It could be the way the question was framed (as suggested in this Washington Post column); it could be the timing (much has transpired and has been said between the dates these polls were taken); maybe the culprit is sample; perhaps modality played a part (some were done online, others by phone with an interviewer), etc.

My guess is that all of these factors to varying degrees account for the differences, but the one thing all of these polls share is that the instrument was quantitative.

So, I decided to see what if anything happens when we try to “unstructure” this question, which seemingly lends itself so perfectly to a multiple-choice format. How would an open-ended version of the same question compare with the results from the structured version? Would it add anything of value?

Part I: A Multiple-Choice Benchmark

The first thing we did was to run a quantitative poll as a comparator using a U.S. online nationally representative sample* of n=1,531 (a larger sample, by the way, than any of the aforementioned polls used).

In carefully considering how the question was framed in the other polls and how it’s being discussed in the media, we decided on the following wording:

“Q. How do you personally feel about Trump's latest Executive Order 13769 ‘Protecting the Nation from Foreign Terrorist Entry into the United States’ aka ‘A Muslim Ban’”?

We also went with the simplest and most straightforward closed-ended Likert scale—a standard five-point agreement scale. Below are the results:

Given a five-point scale, the most popular answer by respondents (36%) was “strongly disagree.” Interestingly, the least popular choice was “somewhat disagree” (6.6%).

Collapsing “strongly” and “somewhat” (see chart below) we found 4% more Americans (43%) disagree with Trump’s Executive Order than agree with it (39%). A sizeable number (18%) indicated they aren’t sure/don’t know.

Will It Unstructure? - A Text Analytics PollTM

Next, we asked another 1500 respondents from the same U.S. nationally online representative source* EXACTLY the same question, but instead of providing choices for them to select from, we asked them to reply in an open-ended comment box in their own words.

We ran the resulting comments through OdinText, with the following initial results:

As you can see, the results from the unstructured responses were remarkably close to those from structured question. In fact, the open-ended responses suggest Americans are slightly closer to equally divided on the issue, though slightly more disagree (a statistically significant percentage given the sample size).

This, however, is where the similarities between unstructured and structured data end.

While there is nothing more to be done with the Likert scale data, the unstructured question data analysis has just begun…

Low-Incidence Insights are Hardly Incidental

It’s worth noting here that OdinText was able to identify and quantify many important, but low-incidence insights—positive and negative— that would have been treated as outliers in a limited code-base and dismissed by human coders:

  • “Embarrassment/Shame” (0.2%)
  • “Just Temporary” (0.5%)
  • “Un-American” (0.9%)
  • “Just Certain/Specific Countries” (0.9%)
  • “Unconstitutional/Illegal” (2%)
  • “Not a Muslim Ban/Stop Calling it that” (2.9%)
  • ...

 

An Emotionally-Charged Policy

It shouldn’t come as a surprise to anyone that emotions around this particular policy run exceptionally high.

OdinText quickly quantified the emotions expressed in people’s comments, and you can see that while there certainly is a lot of anger—negative comments are spread across anger, fear/anxiety and sadness—there is also a significant amount of joy.

What the heck does “joy” entail, you ask? It means that enough people expressed unbridled enthusiasm for the policy along the lines of, “I love it!” or “It’s about time!” or “Finally, a president who makes good on his campaign promises!”

Understanding the Why Behind People’s Positions

Last, but certainly not least, asking the same question in an open-ended format where respondents can reply in their own words enables us to also understand why people feel the way they do.

We can then quantify those sentiments using text analytics and see the results in context in a way that would not have been possible using a multiple-choice format.

Here are a few examples from those who disagree with the order:

  • “Just plain wrong. It scored points with his base, but it made all Americans look heartless and xenophobic in the eyes of the world.”
  • “Absolutely and unequivocally unconstitutional. The foundation, literally the reason the first European settlers came to this land, was to escape religious persecution.”
  • “I don't like and it was poorly thought out. I understand the need for vetting, but this was an absolute mess.”
  • “I think it is an overly confident action that will do more harm than good.”
  • “I understand that Trump's intentions mean well, but his order is just discriminating. I fear that war is among us, and although I try my best to stay neutral, it's difficult to support his actions.”

Here are a few from those who agree:

  • “I feel it could have been handled better but I agree. Let’s make sure they are here documented correctly and backgrounds thoroughly checked.”
  • “I feel sometimes things need to be done to demonstrate seriousness. I do feel bad for the law abiding that it affects.”
  • “Initially I thought it was ridiculous, but after researching the facts associated with it, I'm fine with it. Trump campaigned on increasing security, so it shouldn't be a surprise. I think it is reasonable to take a period of time to standardize and enforce the vetting process.”
  • “I feel that it is not a bad idea. The only part that concerns me is taking away from living the American Dream for those that aren’t terrorists.”
  • “good but needed more explanation”
  • “OK with it - waiting to see how it pans out over the next few weeks”
  • “I think it is good, as long as it is temporary so that we can better vet those who would come to the U.S.”

And just as importantly, yet oft-overlooked those who aren’t completely sure:

  • “not my circus”
  • “While the thought is good and just for our safety, the implementation was flawed, much like communism.”

Final Thoughts: What Have we Learned?

First of all, we saw that the results in the open-ended format replicated those of the structured question. With a total sample of 3000, these results are statistically significant.

Second, we found that while emotions run high for people on both sides of this issue, comments from those who disagree with the ban tended to be more emotionally charged than from those who agreed with the ban. I would add here that some of the former group tended not to distinguish between their feelings about President Trump and the policy.

We also discovered that supporters of the ban appear to be better informed about the specifics of the order than those who oppose it. In fact, a significant number of the former group in their responses took the time to explain why referring to the order as “a Muslim ban” is inaccurate and how this misconception clouds the issue.

Lastly, we found that both supporters and detractors are concerned about the order’s implementation.

Let me know what you think. I’d be happy to dig into this data a bit more. In addition, if anyone is curious and would like to do a follow-up analysis, please contact me to discuss the raw data file.

@TomHCAnderson

Ps. Stay tuned for Part II of this study, where we’ll explore what the rest of the world thinks about the order!

*Note: Responses (n=3,000) were collected via Google Surveys. Google Surveys allow researchers to reach a validated (U.S. General Population Representative) sample by intercepting people attempting to access high-quality online content—such as news, entertainment and reference sites—or who have downloaded the Google Opinion Rewards mobile app. These users answer up to 10 questions in exchange for access to the content or Google Play credit. Google provides additional respondent information across a variety of variables including source/publisher category, gender, age, geography, urban density, income, parental status, response time as well as google calculated weighting. Results are +/- 1.79% accurate at the 95% confidence interval.

 

About Tom H. C. Anderson

Tom H. C. Anderson is the founder and managing partner of OdinText, a venture-backed firm based in Stamford, CT whose eponymous, patented SAS platform is used by Fortune 500 companies like Disney, Coca-Cola and Shell Oil to mine insights from complex, unstructured and mixed data. A recognized authority and pioneer in the field of text analytics with more than two decades of experience in market research, Anderson is the recipient of numerous awards for innovation from industry associations such as CASRO, ESOMAR and the ARF. He was named one of the "Four under 40" market research leaders by the American Marketing Association in 2010. He tweets under the handle @tomhcanderson.