Posts in Analytics
Listening to Employees Where it Really Matters

World-Renowned Hospitals Use OdinText to Listen to Valuable Employee Feedback and Prioritize Resources

Human-Resources.png

This year again, OdinText was honored to be voted among the Most Innovative Marketing Research Suppliers in the World (2018 Greenbook Research Industry Trends GRIT Report).

As is custom, each of the Top Marketing Research Suppliers are invited to submit a case study for inclusion in the annual e-book showcasing the best of the best in consumer insights.

Last year our case study highlighted how OdinText can use customer comment data to better understand drivers of customer satisfaction and NPS (Net Promoter Score), as well as  predict  return behavior and revenue! This year we chose a case which highlights how OdinText can be used for Workforce Analytics to leverage employee feedback for continuous improvement, increased employee engagement and satisfaction.

As the Director of Human Resources, Greene Memorial Hospital and Soin Medical Center comments on the experience with OdinText in the Health Care & Human Resources:

Odin-Health-Worjforce-Text-Analytics-300x194.png

“The magnitude and detail of OdinText is amazing! [OdinText] pinpoints exactly areas that we can really work on. Other vendors just give us material and we have to hunt and peck. For not knowing anything about our industry, this software is amazing! You know atmosphere, what’s changing and what’s not… This blows me away.”

You can view an abbreviated case study in the e-book or on the Greenbook site tomorrow. However we are also happy to share a slightly more detailed case with you. To find out how world class hospitals improve through stakeholder feedback follow the link below:

alt text

alt text

Turning staff feedback into action: OdinText artificial intelligence platformreveals what drives employeesatisfaction at Kettering HealthNetwork’s Greene Memorial Hospitaland Soin Medical Center!

Again Thank You to Greenbook and everyone who selected us as most innovative, and congratulations to all the other great companies this year!

If you are curious how YOUR Data +OdinText will provide more powerful Insights Guaranteed you can request more information here.

Tom H. C. Anderson Chief Research Officer @OdinText

The State of Marketing Research Innovation

What You Missed at IIEX 2018 – 3 Takeaways Walking the floor at the Insights Innovation Exchange (IIEX) for a day and a half with our new CEO, Andy Greenawalt, we spoke to several friends, client and supplier side partners, and ducked into quite a few exciting startup sessions.

Three things struck me this year:

-Insights Technology is Finally Getting More Innovative. By that I mean there are no longer just the slight immaterial modifications to existing ways of doing things, but actual innovation that has disruptive implications (passive monitoring, blockchain, image recognition, more intelligent automation…).

As expected most of this innovation is coming from startups, many of which, while they have interesting ideas, have little to no experience in marketing research - and have yet to prove their use cases.

-A Few Marketing Research Suppliers are picking up their consulting game. Surprisingly perhaps, in this area it seems that change is coming from the Qualitative side. For a while qualitative looked like a race to the bottom in terms of price, even more so than what was happening in Quantitative Research. But there are now a handful of Image/Brand/Ideation ‘Agencies’ whose primary methodologies are qualitative who are leading the way to a higher value proposition. There are a couple, but I will mention two I’ve been most impressed with specifically, Brandtrust and Shapiro+Raj, Bravo!

-The Opportunity. I think the larger opportunity if there is one, lies in the ability of the traditional players to partner with and help prove the use cases of some of these newer startup technologies. Incorporating them into consulting processes with higher end value propositions, similar to what the qualitative agencies I noted above have done.

This seems to be both an opportunity and a real challenge. Can Old help New, and New help Old? It may be more likely that the end clients, especially those that are more open to DIY processes will be the ones that select and prove the use cases of these new technologies offered by the next generation of startups, and therefore benefit the most.

While this too is good, I fear that by leaving some of the traditional companies behind we will lose some institutional thinking and sound methodology along the way.

Either way, I’m more optimistic on new Marketing Research Tech than I’ve ever been.

Keep in mind though, Innovation in Marketing Research should be about more than just speed and lower cost (automation). It should be even more about doing things better, giving the companies and clients we work for an information advantage!

@TomHCAnderson

Best 10 Text Analytics Tips Posts of The Year

Our Top 10 Most Read Data and Text Mining Posts of 2017

Thank you for reading our blog this year. The OdinText blog has quickly become even more popular than the Next Gen Market Research blog, and I really appreciate the thoughtful feedback we’ve gotten here on the blog, via Twitter, and email.

In case you’re curious, here are the most popular posts of the year:

#10 NFL Players Taking a Knee is More Complex and Polarizing Than We Think If a Topic is Worth Quantifying – It’s Also Worth Understanding The Why’s Behind It

#9 Text Analytics Picks The 10 Strongest Super Bowl Ads New Text Analytics Poll Shows Which Super Bowl Ads Really Performed Best

#8 Why Your HR Survey is a Lie and How to Get The Truth OdinText Discovers Job Satisfaction Drivers in Anonymous Employee Data

#7 Of Tears & Text Analytics (An OdinText User Story – Text Analytics Guest Post (AI Meets VOC))

#6 65 CEO’s Share Thoughts on Insights (Insights Associations Inaugural CEO Summit – A Future Tied to Collaboration and Technology)

#5 Why Machine Learning is Meaningless (Beware of Buzzwords! The Truth about ‘Machine Learning’ and ‘Artificial Intelligence’)

#4 Do You Speak Teen? OdinText Announces 2nd Annual List of Top 10 Slang Terms (How Text Analytics Can Help Marketers Move at the Speed of Slang)

#3 Text Analysis Reveals Potential French Election French Election Upset (Text Analytics Poll Showed How Close Le Pen Came to ‘Trumping’ Macron)

#2 Text Analytics Poll: Why We Unfriend on Facebook (You Can’t Handle The Truth (And Other Top Reasons Why We Unfriend on Facebook)

#1 What Americans Really Think About Trump’s Immigration Ban and Why (Text Analysis of What People Say in Their Own Words Reveals More Than Multi-Choice Surveys)

 

I thought I’d also check what our top 5 posts were from last year as well, here they are in case you missed them:

Top Posts From 2016

#1 Text Analysis Answers Is The Quran Really More Violent Than The Bible (3 Parts)

#2 Attensity Sold – What Does it Mean?

#3 Customer Satisfaction Surveys: What do Satisfied VS Dissatisfied Customers Talk About?

#4 What’s Really Wrong With Polling?

#5 What Your Customer Satisfaction Research Isn’t Telling You

Thanks again for reading and commenting. As always I welcome your thoughts and questions via LinkedIn, or feel free to request info on anything you’ve read above here.

Happy New Year!

@TomHCAnderson

Why Your HR Survey is a Lie and How to Get the Truth

OdinText Discovers Job Satisfaction Drivers in Anonymous Employee Data

Employee satisfaction surveys go by different names – “stakeholder satisfaction,” “360-degree surveys,” “employee engagement.” I blog a lot about the shortcomings of survey data and problems like respondent and data quality due to bad sample, long surveys, and poorly structured questions (which assumes the researcher already knows the best questions to ask), but I haven’t talked much about human resources/employee surveys.

HR surveys have a different and bigger problem than consumer surveys. It’s not the sample; we know exactly who our employees are. We know their email, phone numbers and where they sit. Heck, we can even mandate 100% survey participation (and some companies do). In fact, I’ve spoken to Human Resources directors who actually survey their employees once per week. The reasoning goes something like, “our millennial employees are in high demand and we want to keep them happy.” But that’s not a problem, per se; in fact, I’m a believer in frequent data collection.

The Problem with Employee Satisfaction Surveys

Hr employee satisfaction survey research

NO ONE taking an employee survey trusts that their data will be kept anonymous and confidential. This is the case even when third parties promising to keep all data at an aggregate level are hired to conduct the fieldwork.

It really doesn’t matter that this mistrust may be unfounded or invalid, only that it exists. And as it happens, it isn’t entirely unfounded. In fact, I know a former senior HR manager at a Fortune 500 who admitted to successfully pressuring vendors into providing de-anonymized, individual-level data.

Even if you as an employee believed your data would remain anonymous, you might nonetheless be wary of being completely forthcoming. For instance, if you were one of three or four direct reports asked to rate your satisfaction with either your company or your manager on a five-point Likert scale, it might feel foolhardy to answer with anything less than a top-3-box score. There would be a lot of interest in who the ‘disgruntled’ employee was, after all.

This is a data problem, and there are two solutions:

  1. Text Analysis of Employee Satisfaction Comment Data

Unstructured, free-form comment data can be a window into the soul! I might never risk giving my company or supervisor anything below a top-2-box satisfaction rating on a Likert scale, but there are other ways to unearth the truth. For example, consider these open-ended questions:

Q: “What do you think about the prospects for your company’s success in the next two years?”

Or maybe a specific question about a boss I didn’t like? Such as:

Q: “Tell us about your relationship with your boss. Does he/she provide you with adequate X?”

While the respondent would obviously still be very careful about how he/she answers – probably even more so – it would be nearly impossible not to divulge important clues about how he/she really feels.

Why? Because we really aren’t very good at lying. We can’t help leaving emotional clues in spoken or written text that reveal our hidden emotions based on word choice.

Even in the absence of any negative terms or emotions, just the appearance of significantly lower levels of basic positive sentiment within a department or within a specific manager’s group might signal a serious problem.

  1. Anonymizing Employee Satisfaction Data

The other solution is to collect data that truly is more anonymous. This is a second unmet opportunity to improve employee satisfaction and engagement research. The trick is not just providing an option for anonymous feedback such as a suggestion box, but making it top-of-mind and encouraging more frequent anonymous feedback.

Obviously, many companies know their customers are discussing them and giving them ratings both anonymously and with their real profiles on various sites—Amazon.com, BazaarVoice.com, Airbnb, TripAdvisor and Yelp, to name just a few.

But what about employee data? Back during the dotcom boom, working for Snowball.com/Ign.com I recall everyone at the company and other dotcom’s regularly visiting F*ckedCompany.com (the asterisk was added by me) where anonymous feedback about management, impending layoffs, etc., would be shared. This isn’t necessarily a good thing for anyone except investors wanting to short a certain company.

Today there are sites like GlassDoor.com where employees rate larger companies on both work satisfaction, in general, and even the interview process. The problem with this site is that it tends to be focused more on larger companies (think Home Depot and Lowes), though there are many ratings for middle-market and smaller companies, too.

I think in the future there will be even more opportunities to access public reviews of satisfaction at companies, but also hopefully more private ways to collect unbiased data on your company’s employee satisfaction.

What to Expect from Text Analysis of Good/Anonymous Employee Data?

While I’ll be writing more on this topic in the future, what prompted the idea for this blog post was one of our most recent clients, TruckersReport.com. As the name suggests, TruckersReport.com is a trucking industry professional community that includes an anonymous discussion board.

Recently, OdinText deployed to analyzed anonymous, unstructured comments as well as structured ratings data from the TruckersReport.com’s website. Some rather unexpected findings were covered in a joint press release. For example, OdinText discovered 11 features driving job satisfaction, and salary didn’t make the top three! You may request a more detailed report here.

I look forward to your thoughts and questions related to either the above study or human resources analytics in general.

 

@TomHCAnderson

Learn How to Smash Data Silos at the Marketing Metrics and Analytics Summit This Week

I’m Going on a “Data Raid” with Three Client-Side Practitioners in NYC on Thursday! Consumer decisions are not made in a vacuum, so why do we continue to expect to get the whole truth from siloed data? These are puzzle pieces, not the whole puzzle!

The biggest insights opportunity for just about any organization today lies in bringing disparate data sources together to develop an entirely new, holistic and infinitely more useful picture of what’s driving the results we’re seeing (or not seeing, as the sad case may too often be).

This used to be a pipe dream, but today it is feasible. Unfortunately, one of the greatest barriers to achieving holistic insights is the organization, itself!

At the 2017 Marketing Metrics and Analytics Summit in NYC this week, I’ll be joining three savvy insights jocks on a roundtable to tackle “Breaking Down Black Boxes,” or, more specifically, How to Address Communication Barriers When It Comes to Big Data.”

It’s a critically important topic, and I’m eager to unpack it along with Heineken’s Director of Data Science & Analytics, Shivanku Misra, Pandora’s Consumer Insights  Director, Andrea Lopus Cardozo, and Matthew Koppel, Senior Manager of Digital Marketing at Next Issue Media.

There are several challenges with intra-organizational information-sharing. Frankly, data and insights are power, and so there are reasons that aren’t necessarily healthy, let alone optimal, behind why data-siloing persists in organizations today.

The good news is that it’s far harder than it used to be to hoard data. As software becomes more powerful in terms of handling merged data and analyzing different kinds of data, everyone’s data is more exposed.

This presents a tremendous opportunity for individuals or departments who are equipped to analyze these data themselves. I know I may not sound like a team player here, but in my experience when a committee is entrusted with mixed data, the process is slow, wasteful, and overly political/bureaucratic. The most effective mixed data initiatives I’ve seen have involved smaller, more agile actors within the organization who take on skunkworks projects that often incorporate other departments’ data to really outshine their internal competition.

I’m sure we’ll be discussing ways to circumvent miscommunication and “forging harmonious and mutually-beneficial working relationships,” but let’s face it: what’s work without a bit of friendly competition? 😉

I’m really looking forward to this event and hearing from some of the other very interesting speakers like Wharton Professor of Marketing Peter Fader.  If you can get into NYC on Wednesday or Thursday, you should really consider attending. I’d love to see you there!

Feel free to use my speaker code [ODIN] for a $200 discount!

@TomHCAnderson

 

PS. If you’re not able to attend, or for some reason we don’t get a chance to speak at the event, please don’t hesitate to reach out. I love discussing how mixed data analysis can be an insights game-changer, and an important first step is getting access to the right data.

About Tom H. C. Anderson

Tom H. C. Anderson is the founder and managing partner of OdinText, a venture-backed firm based in Stamford, CT whose eponymous, patented SAS platform is used by Fortune 500 companies like Disney, Coca-Cola and Shell Oil to mine insights from complex, unstructured and mixed data. A recognized authority and pioneer in the field of text analytics with more than two decades of experience in market research, Anderson is the recipient of numerous awards for innovation from industry associations such as CASRO, ESOMAR and the ARF. He was named one of the “Four under 40” market research leaders by the American Marketing Association in 2010. He tweets under the handle @tomhcanderson.

Marketing Research Blooper Reveals Lots of Surprises and Two Important Lessons

April Foolishness: What Happens When You Survey People in the Wrong Language?

I’m going to break with convention today and, in lieu of an April Fool’s gag, I’m going to tell you about an actual goof we recently made that yielded some unexpected but interesting results for researchers.

As you know, last week on the blog we highlighted findings from an international, multilingual Text Analytics Poll™ we conducted around culture. This particular poll spanned 10 countries and eight languages, and when we went to field it we accidentally sent the question to our U.S. sample in Portuguese!

Shockingly, in many cases, people STILL took the time to answer our question! How?

First, bear in mind that these Text Analytics Polls™ consist of only one question and it’s open-ended, not multiple choice. The methodology we use intercepts respondents online and requires them to type an answer to our question before they can proceed to content they’re interested in.

Under the circumstances, you might expect someone to simply type “n/a” or “don’t understand” or even some gibberish in order to move on quickly, and indeed we saw plenty of that. But in many cases, people took the time to thoughtfully point out the error, and even with wit.

Verbatim examples [sic]:

“Are you kidding me, an old american who can say ¡adios!”

“Tuesday they serve grilled cheese sandwiches.” “What the heck is that language?”

“No habla espanol”

“i have no idea what that means”

“2 years of Spanish class and I still don't understand”

Others expressed themselves more…colorfully…

“No, I don't speak illegal immigrant.”

“Speak English! I'm switching to News 13 Orlando. They have better coverage than FT.”

Author’s note: I suspect that last quote was from someone who was intercepted while trying to access a Financial Times article. ;-)

While a lot of people clearly assumed our question was written in Spanish, still others took the time to figure out what the language was and even to translate the question!

“I had to use google translate to understand the question.”

“what the heck does this mean i don't speak Portuguese”

But what surprised me most was that a lot of Americans actually answered our question—i.e., they complied with what we had asked—even though it was written in Portuguese. And many of those replies were in Spanish!!!

We caught our mistake quickly enough when we went to machine-translate the responses and we were told that replies to a question in Portuguese were now being translated from English to English, but two important lessons were learned here:

Takeaway One: Had we made this mistake with a multiple-choice instrument, we either might not have caught it until after the analysis or perhaps not at all. Not only would respondents not have been able to tell us that we had made a mistake, but they would’ve had the easy option of just clicking a response at random. And unless those random clicks amounted to a conspicuous pattern in the data, we could’ve potentially taken the data as valid!

Takeaway Two: The notion that people will not take the time to thoughtfully respond to an open-ended question is total bunk. People not only took the time to answer our question in detail when it was correctly served to them in their own language, but they even spared a thought for us when they didn’t understand the language!

I want to emphasize here that if you’re one of those researchers (and I used to be among this group, by the way) who thinks you can’t include an open-ended question in a quantitative instrument, compel the respondent to answer it, and get a meaningful answer to your question, you are not only mistaken but you’re doing yourself and your client a huge disservice.

Take it from this April fool, open-ended questions not only tell you what you didn’t know; they tell you what you didn’t know you didn’t know.

Thanks for reading. I’d love to hear what you think!

@TomHCAnderson

P.S. Find out how much more value an open-ended question can add to your survey using OdinText. Contact us to talk about it.

About Tom H. C. Anderson

Tom H. C. Anderson is the founder and managing partner of OdinText, a venture-backed firm based in Stamford, CT whose patented SAS platform is used by Fortune 500 companies like Disney, Coca-Cola and Shell Oil to mine insights from complex, unstructured and mixed data. A recognized authority and pioneer in the field of text analytics with more than two decades of experience in market research, Anderson is the recipient of numerous awards for innovation from industry associations such as CASRO, ESOMAR and the ARF. He was named one of the "Four under 40" market research leaders by the American Marketing Association in 2010. He tweets under the handle @tomhcanderson.

Text Analytics: It's Not Just for BIG Data

In a world focused on the value of Big Data, it's important to realize that Small Data is meaningful, too, and worth analyzing to gain understanding. Let me show you with a personal example. If you're a regular reader of the OdinText blog, you probably know that our company President, Tom Anderson, writes about performing text analytics on large data sets.  And yes, OdinText is ideal for understanding data after launching a rapid survey then collecting thousands of responses.

However for this blog post, I'm going to focus on the use of Text Analytics for smaller, nontraditional data set:  emails.

SMALL Data (from email) Text Analytics

I recently joined OdinText as Vice President, working closely with Tom on all our corporate initiatives. I live in a small town in Connecticut with an approximate population of 60,000.  Last year I was elected to serve our town government as an RTM member along with 40 other individuals.  Presently, our town's budget is $290M and the RTM is designing the budget for the next year.

Many citizens email elected members to let them know how they feel about the budget.  To date, I have received 280 emails. (Before you go down a different path with this, please know that I respond personally to each one -- people who take the time to write me deserve a personal response.  I did not and will not include in this blog post how I intend to vote on the upcoming budget, nor will I include anything about party affiliations. And I certainly will not share names.)

As the emails were coming in, I started to wonder … what if I ran this the data I was receiving through OdinText?  Would I be able to use the tool to identify, understand and quantify the themes in the people’s thoughts on how I should vote on the budget?

The Resulting Themes from Small Data Analytics

A note about the methodology:  Each email that I received contained the citizen's name, their email address and content in open text format.  Without a key driver metric like OSAT, CSAT or NPS to analyze the text against, I chose to use overall sentiment. Here is what I learned

Emails about the town budget show that our citizens feel Joy but RTM members need to recognize their Sadness, Fear and Anger

Joy:

“I have been a homeowner in Fairfield for 37 years, raised 4 kids here and love the community.”

Sadness:

“I am writing you to tell you that I am so unhappy with the way you have managed our town.”

Fear:

“My greatest concern seems to be the inability of our elected members to cut spending and run the town like a business”

Anger:

“We live in a very small house and still have to pay an absurd amount of money in taxes.”

Understanding the resulting themes in their own words

Reduce Taxes (90.16%)

“Fairfield taxes are much higher than surrounding communities.”

“Fairfield taxes are out of line with similar communities”

“The town has to stop raising taxes at such a feverish rate.”

“High taxes are slowly eroding the town of Fairfield.”

Moving if Taxes are Increased (25.13%)

“I am on a fixed income at 64, and cannot afford Fairfield’s taxes now. Please recognize that I cannot easily sell my house, due to the economy & the amount of homes on the market here”

“regret to say most of our colleagues and friends have an "exit strategy" to leave Fairfield”

“Our town is losing residents who are fed up and have moved or are moving to Westport and other towns with lower mil rates”

Reduce Spending (33.33%)

“... bring spending under control”

“Stop the spending please”

“... needs to trim fat at the local level, cut services, stop spending money”

“We need to keep taxes down as much as possible - even if it means spending cuts.”

Education ‘don’t cut’ (8.74%)

“… takes great pride in its education system”

“… promise of an excellent public education”

“… fiscal responsibility; however, not at the expense of the children and their right to an excellent education.”

Education ‘please cut’ (9.83%)

“Let's shave funding from all programs including education”

“... deeply questioning our education budget”

“... reduce the Education budget”

“I have a cherished budgetary item that I want protected--the library. Cut that last, after you cut education, police, official salaries”

Big Value from Small Data in Little Time

I performed this text analysis in 30 minutes. Ironically, it has taken me longer to write this blog post than it did to quantify the text from all those emails. Yet the information and understanding I have gleaned will empower me as I make decisions on this important topic. A small investment in small data has paid off in a BIG way.

Tim Lynch - @OdinText

Poll: What Other Countries Think of Trump’s Immigration Order

Text Analytics PollTM Shows Australians, Brits, and Canadians  Angry About Executive Order Temporarily Barring Refugee (Part II of II)In my previous post, we compared text analysis of results from an open-ended survey instrument with a conventional Likert-scale rating poll to assess where 3,000 Americans really stand on President Trump’s controversial executive order temporarily barring refugees and people from seven predominately-Muslim countries from entering the U.S.

Today, we’re going to share results from an identical international study that asked approx. 9,000 people—3,000 people from each of three other countries—what they think about the U.S. immigration moratorium ordered by President Trump.

But first, a quick recap…

As I noted in the previous post, polling on this issue has been pretty consistent insomuch as Americans are closely divided in support/opposition, but the majority position flips depending on the poll. Consequently, the accuracy of polling has again been called into question by pundits on both sides of the issue.

By fielding the same question first in a multiple-choice response format and a second time providing only a text comment box for responses, and then comparing results, we were able to not only replicate the results of the former but gain a much deeper understanding of where Americans really stand on this issue.

Text analysis confirmed a much divided America with those opposing the ban just slightly outnumbering (<3%) those who support the order (42% vs 39%). Almost 20% of respondents had no opinion or were ambivalent on this issue.

Bear in mind that text analysis software such as OdinText enables us to process and quantify huge quantities of comments (in this case, more than 1500 replies from respondents using their own words) in order to arrive at the same percentages that one would get from a conventional multiple-choice survey.

But the real advantage to using an open-ended response format (versus a multiple-choice) to gauge opinion on an issue like this is that the responses also tell us so much more than whether someone agrees/disagrees or likes/dislikes. Using text analytics we uncovered people’s reasoning, the extent to which they are emotionally invested in the issue, and why.

Today we will be looking a little further into this topic with data from three additional countries: Australia, Canada and the UK.

A note about multi-lingual text analysis and the countries selected for this project…

Different software platforms handle different languages with various degrees of proficiency. OdinText analyzes most European languages quite well; however, analysis of Dutch, German, Spanish or Swedish text requires proficiency in said language by the analyst. (Of course, translated results, including and especially machine-translated results, work very well with text analytics.)

Not inconspicuously, each of the countries represented in our analysis here has an English-speaking population. But this was not the primary reason that we chose them; each of these countries has frequently been mentioned in news coverage related to the immigration ban: The UK because of Brexit, Australia because of a leaked telephone call between President Trump and its Prime Minister, and Canada due to its shared border and its Prime Minister’s comments on welcoming refugees affected by the immigration moratorium.

Like our previous U.S. population survey, we used a nationally-representative sample of n=3000 for each of these countries.

Opposition Highest in Canada, Lowest in the UK

It probably does not come as a surprise to anyone who’s been following this issue in the media that citizens outside of America are less likely to approve of President Trump’s immigration moratorium.

I had honestly expected Australians to be the most strongly opposed to the order in light of the highly-publicized and problematic telephone call transcript leaked last week between President Trump and the Australian Prime Minister (which, coincidentally, involved a refugee agreement). But interestingly, people from our close ally and neighbor to the north, Canada, were most strongly opposed to the executive order (67%). The UK had proportionately fewer opposing the ban than Australia (56% vs. 60%), but the numbers of people opposed to the policy in both countries significantly lagged the Canadians. Emotions Run High Abroad Deriving emotions from text is an interesting and effective measure for understanding people’s opinions and preferences (and more useful than the “sentiment” metrics often discussed in text analytics and, particularly, in social media monitoring circles).

The chart below features OdinText’s emotional analysis of comments for each of the four countries across what most psychologists agree constitute the eight major emotion categories:

We can see that while the single highest emotion in American comments is joy/happiness, the highest emotion in the other three countries is anger. Canadians are angriest. People in the UK and Australians exhibit somewhat greater sadness and disgust in their comments. Notably, disgust is an emotion that we typically only see rarely in food categories. Here it takes the form of vehement rejection with terms such as “sickened,” “revolting,” “vile,” and, very often, “disgusted.” It is also worth noting that in cases, people directed their displeasure at President Trump, personally.

Examples:

"Trump is a xenophobic, delusional, and narcissistic danger to the world." – Canadian (anger) “Most unhappy - this will worsen relationships between Muslims and Christians.” – Australian (sadness) "It's disgusting. You can't blame a whole race for the acts of some extremists! How many white people have shot up schools and such? Isn't that an act of terror? Ban guns instead. He's a vile little man.” –Australian (disgust)

UK comments contain the highest levels of fear/anxiety:

"I am outraged. A despicable act of racism and a real worry for what political moves may happen next." – UK (fear/anxiety)

That said, it is also important to point out that there is a sizeable group in each country who express soaring agreement to the level of joy:

“Great move! He should stop all people that promote beating of women” – Australian (joy) “Sounds bloody good would be ok for Australia too!” – Australian (joy) “EXCELLENT. Good to see a politician stick by his word” – UK (joy) “About time, I feel like it's a great idea, the United States needs to help their own people before others. If there is an ongoing war members of that country should not be allowed to migrate as the disease will spread.” – Canadian (joy)

Majority of Canadians Willing to Take Refugee Overflow Given Canada’s proximity to the U.S., and since people from Canada were the most strongly opposed to President Trump’s executive order, this raised the question of whether Canadians would then support a measure to absorb refugees that would be denied entrance to the U.S., as Prime Minister Justin Trudeau appears to support.

(Note: In a Jan. 31 late-night emergency debate, the Canadian Parliament did not increase its refugee cap of 25,000.)

 

A solid majority of Canadians would support such an action, although it’s worth noting that there is a significant difference between the numbers of Canadians who oppose the U.S. immigration moratorium (67%) and the number who indicated they would be willing to admit the refugees affected by the policy.

When asked a follow-up question on whether “Canada should accept all the refugees which are turned away by USA's Trump EO 13769,” only 45% of Canadians agreed with such a measure, 33% disagreed and 22% said they were not sure.

Final Thoughts: How This Differs from Other Polls Both the U.S. and the international versions of this study differ significantly from any other polls on this subject currently circulating in the media because they required respondents to answer the question in a text comment box in their own words, instead of just selecting from options on an “agree/disagree” Likert scale.

As a result, we were able to not only quantify support and opposition around this controversial subject, but also to gauge respondents’ emotional stake in the matter and to better understand the “why” underlying their positions.

While text analysis allows us to treat qualitative/unstructured data quantitatively, it’s important to remember that including a few quotes in any analysis can help profile and tell a richer story about your data and analysis.

We also used a substantially larger population sample for each of the countries surveyed than any of the conventional polls I’ve seen cited in the media. Because of our triangulated approach and the size of the sample, these findings are in my opinion the most accurate numbers currently available on this subject.

I welcome your thoughts!

@TomHCAnderson - @OdinText

About Tom H. C. Anderson Tom H. C. Anderson is the founder and managing partner of OdinText, a venture-backed firm based in Stamford, CT whose eponymous, patented SAS platform is used by Fortune 500 companies like Disney, Coca-Cola and Shell Oil to mine insights from complex, unstructured and mixed data. A recognized authority and pioneer in the field of text analytics with more than two decades of experience in market research, Anderson is the recipient of numerous awards for innovation from industry associations such as CASRO, ESOMAR and the ARF. He was named one of the "Four under 40" market research leaders by the American Marketing Association in 2010. He tweets under the handle @tomhcanderson.

What Americans Really Think about Trump’s Immigration Ban and Why

Text Analysis of What People Say in Their Own Words Reveals More Than Multiple-Choice Surveys It’s been just over a week since President Trump issued his controversial immigration order, and the ban continues to dominate the news and social media.

But while the fate of Executive Order 13769—“Protecting the Nation from Foreign Terrorist Entry into the United States”—is being hashed out in federal court, another fierce battle is being waged in the court of public opinion.

In a stampede to assess where the American people stand on this issue, the news networks have rolled out a parade of polls. And so, too, once again, the accuracy of polling data has been called into question by pundits on both sides of the issue.

Notably, on Monday morning the president, himself, tweeted the following:

Majority Flips Depending on the Poll

It’s easy to question the accuracy of polls when they don’t agree.

Although on the whole these polls all indicate that support is pretty evenly divided on the issue, the all-important sound bite of where the majority of Americans stand on the Trump immigration moratorium flips depending on the source:

  • NBC ran with an Ipsos/Reuters poll that found the majority of Americans (49% vs. 41%) support the ban.

 

 

  • CNN publicized results from an ORC Poll with the majority opposed to the ban (53% vs. 47%).

 

  • A widely reported Gallup poll found the majority of Americans oppose the order (55% to 42%).

 

There are a number of possible reasons for these differences, of course. It could be the way the question was framed (as suggested in this Washington Post column); it could be the timing (much has transpired and has been said between the dates these polls were taken); maybe the culprit is sample; perhaps modality played a part (some were done online, others by phone with an interviewer), etc.

My guess is that all of these factors to varying degrees account for the differences, but the one thing all of these polls share is that the instrument was quantitative.

So, I decided to see what if anything happens when we try to “unstructure” this question, which seemingly lends itself so perfectly to a multiple-choice format. How would an open-ended version of the same question compare with the results from the structured version? Would it add anything of value?

Part I: A Multiple-Choice Benchmark

The first thing we did was to run a quantitative poll as a comparator using a U.S. online nationally representative sample* of n=1,531 (a larger sample, by the way, than any of the aforementioned polls used).

In carefully considering how the question was framed in the other polls and how it’s being discussed in the media, we decided on the following wording:

“Q. How do you personally feel about Trump's latest Executive Order 13769 ‘Protecting the Nation from Foreign Terrorist Entry into the United States’ aka ‘A Muslim Ban’”?

We also went with the simplest and most straightforward closed-ended Likert scale—a standard five-point agreement scale. Below are the results:

Given a five-point scale, the most popular answer by respondents (36%) was “strongly disagree.” Interestingly, the least popular choice was “somewhat disagree” (6.6%).

Collapsing “strongly” and “somewhat” (see chart below) we found 4% more Americans (43%) disagree with Trump’s Executive Order than agree with it (39%). A sizeable number (18%) indicated they aren’t sure/don’t know.

Will It Unstructure? - A Text Analytics PollTM

Next, we asked another 1500 respondents from the same U.S. nationally online representative source* EXACTLY the same question, but instead of providing choices for them to select from, we asked them to reply in an open-ended comment box in their own words.

We ran the resulting comments through OdinText, with the following initial results:

As you can see, the results from the unstructured responses were remarkably close to those from structured question. In fact, the open-ended responses suggest Americans are slightly closer to equally divided on the issue, though slightly more disagree (a statistically significant percentage given the sample size).

This, however, is where the similarities between unstructured and structured data end.

While there is nothing more to be done with the Likert scale data, the unstructured question data analysis has just begun…

Low-Incidence Insights are Hardly Incidental

It’s worth noting here that OdinText was able to identify and quantify many important, but low-incidence insights—positive and negative— that would have been treated as outliers in a limited code-base and dismissed by human coders:

  • “Embarrassment/Shame” (0.2%)
  • “Just Temporary” (0.5%)
  • “Un-American” (0.9%)
  • “Just Certain/Specific Countries” (0.9%)
  • “Unconstitutional/Illegal” (2%)
  • “Not a Muslim Ban/Stop Calling it that” (2.9%)
  • ...

 

An Emotionally-Charged Policy

It shouldn’t come as a surprise to anyone that emotions around this particular policy run exceptionally high.

OdinText quickly quantified the emotions expressed in people’s comments, and you can see that while there certainly is a lot of anger—negative comments are spread across anger, fear/anxiety and sadness—there is also a significant amount of joy.

What the heck does “joy” entail, you ask? It means that enough people expressed unbridled enthusiasm for the policy along the lines of, “I love it!” or “It’s about time!” or “Finally, a president who makes good on his campaign promises!”

Understanding the Why Behind People’s Positions

Last, but certainly not least, asking the same question in an open-ended format where respondents can reply in their own words enables us to also understand why people feel the way they do.

We can then quantify those sentiments using text analytics and see the results in context in a way that would not have been possible using a multiple-choice format.

Here are a few examples from those who disagree with the order:

  • “Just plain wrong. It scored points with his base, but it made all Americans look heartless and xenophobic in the eyes of the world.”
  • “Absolutely and unequivocally unconstitutional. The foundation, literally the reason the first European settlers came to this land, was to escape religious persecution.”
  • “I don't like and it was poorly thought out. I understand the need for vetting, but this was an absolute mess.”
  • “I think it is an overly confident action that will do more harm than good.”
  • “I understand that Trump's intentions mean well, but his order is just discriminating. I fear that war is among us, and although I try my best to stay neutral, it's difficult to support his actions.”

Here are a few from those who agree:

  • “I feel it could have been handled better but I agree. Let’s make sure they are here documented correctly and backgrounds thoroughly checked.”
  • “I feel sometimes things need to be done to demonstrate seriousness. I do feel bad for the law abiding that it affects.”
  • “Initially I thought it was ridiculous, but after researching the facts associated with it, I'm fine with it. Trump campaigned on increasing security, so it shouldn't be a surprise. I think it is reasonable to take a period of time to standardize and enforce the vetting process.”
  • “I feel that it is not a bad idea. The only part that concerns me is taking away from living the American Dream for those that aren’t terrorists.”
  • “good but needed more explanation”
  • “OK with it - waiting to see how it pans out over the next few weeks”
  • “I think it is good, as long as it is temporary so that we can better vet those who would come to the U.S.”

And just as importantly, yet oft-overlooked those who aren’t completely sure:

  • “not my circus”
  • “While the thought is good and just for our safety, the implementation was flawed, much like communism.”

Final Thoughts: What Have we Learned?

First of all, we saw that the results in the open-ended format replicated those of the structured question. With a total sample of 3000, these results are statistically significant.

Second, we found that while emotions run high for people on both sides of this issue, comments from those who disagree with the ban tended to be more emotionally charged than from those who agreed with the ban. I would add here that some of the former group tended not to distinguish between their feelings about President Trump and the policy.

We also discovered that supporters of the ban appear to be better informed about the specifics of the order than those who oppose it. In fact, a significant number of the former group in their responses took the time to explain why referring to the order as “a Muslim ban” is inaccurate and how this misconception clouds the issue.

Lastly, we found that both supporters and detractors are concerned about the order’s implementation.

Let me know what you think. I’d be happy to dig into this data a bit more. In addition, if anyone is curious and would like to do a follow-up analysis, please contact me to discuss the raw data file.

@TomHCAnderson

Ps. Stay tuned for Part II of this study, where we’ll explore what the rest of the world thinks about the order!

*Note: Responses (n=3,000) were collected via Google Surveys. Google Surveys allow researchers to reach a validated (U.S. General Population Representative) sample by intercepting people attempting to access high-quality online content—such as news, entertainment and reference sites—or who have downloaded the Google Opinion Rewards mobile app. These users answer up to 10 questions in exchange for access to the content or Google Play credit. Google provides additional respondent information across a variety of variables including source/publisher category, gender, age, geography, urban density, income, parental status, response time as well as google calculated weighting. Results are +/- 1.79% accurate at the 95% confidence interval.

 

About Tom H. C. Anderson

Tom H. C. Anderson is the founder and managing partner of OdinText, a venture-backed firm based in Stamford, CT whose eponymous, patented SAS platform is used by Fortune 500 companies like Disney, Coca-Cola and Shell Oil to mine insights from complex, unstructured and mixed data. A recognized authority and pioneer in the field of text analytics with more than two decades of experience in market research, Anderson is the recipient of numerous awards for innovation from industry associations such as CASRO, ESOMAR and the ARF. He was named one of the "Four under 40" market research leaders by the American Marketing Association in 2010. He tweets under the handle @tomhcanderson.

OdinText Predicts What Television Shows You Will Like

How Text Analytics Rescued Me from a #ShowHole!  My wife and I recently found ourselves in the uncomfortable condition commonly known as a “show hole.” Are you familiar with this?

A show hole refers to the state of loss and aimlessness one experiences after completing—often via binge-watching—the last episode in a beloved TV series without having a successor program queued up. The term was popularized by an Amazon Fire campaign a couple years ago, and you’ll find it hashtagged all over social media these days by people desperately in need of relief.

The show hole is an interesting phenomenon that speaks to how dramatically audience consumption habits have changed with the advent of the DVR, streaming and on-demand services like Netflix, Hulu, Amazon, etc. But what’s curious to me is how such a clearly great need continues to go relatively unmet.

Of course, subscribers to on-demand services have help in the way of recommendations algorithms. Netflix, in particular, has famously invested extensively in developing predictive analytics to suggest other shows to watch.

Still, the preponderance of cries for help on social media would seem to indicate that for many people these solutions have fallen short.

Indeed, it appears that people tend to prefer recommendations from other people, which introduces a different set of problems.

The Problem with Recommendations, Ratings and Reviews from People

In my own disappointing search for what to watch next, I found #showhole aplenty on Twitter, but the platform doesn’t lend itself well to discussion, so most of those who tweet about it get left hanging. Usually it’s just a “Help! I’m in a #showhole!” message from someone after finishing a series, but hardly anyone tweets a reply with suggestions.

Note: Because Twitter isn’t well-suited to this kind of interaction, your standard social media monitoring tool—most of them rely on Twitter data—wouldn’t be effective for the type of analysis we’ll cover today.

I did, however, find a ton of recommendation activity occurring on Facebook, Reddit and a variety of other discussion boards and community-based sites, including surprising places like babycenter.com—a support community for pregnant women and new moms—replete with threads where members actually recommend series’ for other members to try next.

This Yelp-like model for getting out of a show hole has its own limitations, though. How do I know I’ll like what you like? Is it enough to assume that since we’re both new moms that we’ll enjoy the same shows? Or if we both enjoyed one program, that I’ll enjoy whatever else you’ve watched? Similarly, if I ask you on a scale of 1-10 to rate a show, how would that information be useful to me if we don’t have the same tastes? Remember also that we’re looking across genres. Our tastes in dramas might be similar, while our tastes in comedies could be worlds apart.

In short, we have all of these people providing recommendations online, but the recommendations really aren’t any more helpful to the prospective viewer than star-based ratings and reviews. I.e., the show hole sufferer is forced to audition each of these programs until he/she finds one that fits—a time-consuming and potentially frustrating process!

How Text Analytics Can Make These Recommendations Useful

As I pondered the recommendations I saw online, it occurred to me that if I could apply text analytics to identify preference patterns based on recommendations from a broad enough swath of people, I might arrive at a recommendation suited to the unique tastes of my wife and I that we could then invest time in with a high confidence level.

Happily, I discovered that when suggesting new shows to watch via social media, people tend to provide more than one recommendation, and these recommendations usually are not limited to a single genre. This means we have sufficient preference data at the individual level, which, if aggregated from enough people, could form the basis for a predictive model.

In a very short time period, I was able to scrape (collect) several thousand recommendations across a variety of sources online. It’s worth noting that just about every single network that the average American has access to was represented in this data. This is important because someone who uses HBO GO, for example, is obviously more likely to watch and recommend programming from that network than someone does not subscribe to it.

We then layered predictive analytics atop the data using OdinText to see whether text analytics could solve my show hole dilemma. Specifically, I wanted to see what other shows are most frequently co-occurring with shows that my wife and I like in these online recommendations. (OdinText has a few ways it can help in cases like this, including the co-occurrence visualization covered in a recent post on this blog by my colleague, Gosia Skorek, here.)

It’s also important to emphasize here that we accomplished this analysis without asking a single question of anyone, although this type of data could be very nicely augmented with survey data.

OdinText Rescues Tom and His Wife from Their Show Hole!

This data was more challenging than I expected, but OdinText enabled us to find a model that delivered!

Below you’ll find examples of preference clouds based on the co-occurrence of mentions harvested from several thousand recommendations across discussion boards and other social media (excluding Twitter).

Essentially, you’re seeing OdinText’s recommendation for what you should watch next based on the series you’ve just completed.

In our case, my wife and I had completed the most recent episode of “The Walking Dead” on AMC—now on hiatus through February—and, as you can see, OdinText recommended we watch “Goliath” on Amazon.

Not only had I never heard of this series, but when I looked it up I was skeptical that we’d enjoy it because my wife and I are not particularly fond of legal dramas.

It turned out that OdinText’s prediction was spot on; we’re both hooked on “Goliath”!

I'll probably check out "Drunk History" on Comedy Central next...

Attention Show Hole Sufferers: Let OdinText Get You Out!

I think this exercise demonstrates the versatility of the OdinText platform. With a little creativity, OdinText can not only provide breakthrough consumer insights, but solve problems of all stripes.

Here are a few more examples. You’ll note that quite often the suggestions cut across networks. Even though obviously someone recommending something on HBO will be more likely to have seen and to recommend other shows on that network, the model often makes suggestions that are quite surprising, cutting across networks and time. Here are just a few:

Above we have OdinText’s recommendations for anyone who likes “Luke Cage.” (I haven’t seen it and typically am not a fan of super hero stuff, but I ran it as I saw in the data that the show was very popular) “Luke Cage” fans might also like “Daredevil,” “Stranger Things” (which I did love), and “The Flash.” The first three here are all on Netflix, the last one is on CW.

You don’t have to be a premium channel streaming snob to benefit here. If you like the popular sitcom “Big Bang Theory” on CBS, you may well also like their new “2 Broke Girls”, and “Last Man Standing” or “Modern Family” on ABC.

Some of the best shows, in my opinion, are often also ironically less popular and less frequently mentioned. Two such shows are HBO’s “Deadwood,” for which OdinText recommended one good fit—Poldark,” a BBC series--and Netflix’s “Peaky Blinders,” for which OdinText suggests trying “Downton Abbey.

I was honestly so impressed with OdinText’s recommendations that I’m entertaining building a suggestion app based on this model. (And unlike Netflix, I didn’t need dozens of developers and millions of dollars to get the right answer.)

I may also refine the underlying model a bit, as well as update the underlying data in a few months when there are enough new series being mentioned to make doing so worthwhile.

In the meantime, I feel obliged to offer immediate assistance to those poor souls in the throes of a show hole today!

If you’re stuck in a show hole, post the title of your recent favorite series in the comment section of today’s blog. OdinText will tell the first 10 people who respond what to watch next. Then come back and tell us how OdinText did.

I look forward to your comments!

@TomHCanderson

Ps. See firsthand how OdinText can help you learn what really matters to your customers and predict real behavior. Contact us for a demo using your own data here!

About Tom H. C. Anderson

Tom H. C. Anderson is the founder and managing partner of OdinText, a venture-backed firm based in Stamford, CT whose eponymous, patented SAS platform is used by Fortune 500 companies like Disney, Coca-Cola and Shell Oil to mine insights from complex, unstructured and mixed data. A recognized authority and pioneer in the field of text analytics with more than two decades of experience in market research, Anderson is the recipient of numerous awards for innovation from industry associations such as CASRO, ESOMAR and the ARF. He was named one of the "Four under 40" market research leaders by the American Marketing Association in 2010. He tweets under the handle @tomhcanderson.