« Predicting the future - joining up web data and tourist visits User generated content in travel: how do we measure it? »
Learn to love qualitative research and discover the three greatest open questions you could be asking
Businesses and organisations like surveys. Surveys are easy enough to produce (but not so easy to produce well). They give you answers to things that interest you and they give you measures that are useful when ticking your own boxes.
Ordinary people, on the other hand, seem at best to tolerate surveys and show a remarkable resilience when faced with daft questions. You know the ones - they’re a bit like, “On a scale of 1-5 tell me how satisfied you are with the Sleepytown’s CrimeSukz food crime prevention initiative and its effect on you.”
Don’t get me wrong, quantitative surveys have their place and I’ll discuss where this is in a minute. But they are rarely a conversation and in my experience, if you want to rich data from a customer’s perspective, you need to get your hands dirty and get into qualitative research.
Qualitative and Quantitative Research - a quick reminder.
For those of you a little rusty on this, the next few paragraphs are a quick primer for you. Quantitative research is about numbers. ‘9 out of 10 cats prefer a warm fire to the snow’ is an example of quantitative research. Qualitative research is about attitudes, beliefs, views and motivations - the kind of things that are more difficult to reduce to numbers. ‘I’ll vote for Mayor Sandburg but I think he comes across as shifty’ is an example of qualitative insight.
Obviously both types of research compliment each other but it seems to me that people are generally more distrusting of qual than quant.
Qual tells you what you weren’t looking for. Personally, I think this is a good thing.
If I use a quantitative survey structure that asks “What was the biggest problem - the sidewalks, the shops or the train station” I will get an answer where the respondent has weighed up the relative standing of only these three options and given me an answer accordingly. However, if I ask an open question and get answers that say, “I couldn’t see anything”, “The view was terrible” and “My child missed everything because of tall people in front of her” - then I am a) revealing a previously unknown issue and b) getting an idea of its importance because it is the thing that occurs spontaneously in the conversation. In this example, we have moved from the producer’s perspective through to the customer’s perspective.
So what’s the problem with qual?
- It’s difficult to understand
There’s no way round it - understanding qual data can be difficult. For example, you have lots of people saying lots of things in lots of different ways at the same time and simultaneously often meaning something other than what they’re uttering (see my previous post for details!) My view is that putting in some effort is worth while as the outcomes are invaluable and I speak of some solution later in this post about presentation and analysis.
The issue of sample sizes is a vexing one. In some forms of research it is vital. If you are bringing a medicine to market, for example, a sample size that enables you to know that it will make 0.05% of its users violently unwell is a justified investment. If you are a political party, knowing that you are 3 points ahead in election week is also important. But a) large samples cost large money and b) you don’t need them all the time anyway.
Two focus groups of 8 people each will generally give me a deeper insight into people’s motivations about something than a survey of 500 people (the survey of 500 on the other hand will help me to identify demographic differences as well as enabling me to determine, for example, clusters, correlations and other exotic items).
Solutions and Presentation
- Don’t be afraid of open ended questions
If you are doing a quantitative survey, there’s nothing to stop you including open questions and coding the responses. It’s not going to be as quick as totting up a ‘which do you prefer?’ kind of survey but, if you sift through the open responses, you will generally find that you can reduce them down into a manageable number of categories. The example above with people saying things like “I couldn’t see anything” and “The view was terrible” could be coded in this fashion as ‘viewing’ or possibly ‘crowding’ issues (depending on a probably known context).
This can then be condensed and treated as quantitative data but with the advantage that you have not decided the paramenters before starting. The following (fictitious) table of customer complaints gives an example of this and used the same data as the later example further down this post. The message in this instance from open field data is that a) there are three distinct areas of improvement needed and b) an indication of the priority they assume for the customer.
| Area of complaint |
% of complaints |
| Staff |
46% |
| Toilets |
22% |
| Viewing facilities |
32% |
- Good presentation is important too
The kind of table above is good for management and bean-counters - but what about for people who either don’t have the time to read that stuff of are sceptical of the interpretation you are putting on it?
Using the same data used for the table above, I’ve transferred the customer comments (with light editing for sense) into a visual word cloud. This allows the same information to be gathered at a glance.
What should you be asking?
So, I’m a fan of asking open ended questions and then playing with the results for deeper insight. But what are good questions to ask? Well, obviously, the more open ended the better. But I’m on the hunt now to work out what the standard ‘killer’ questions could be for the tourism and travel industry.
This is ground that we’ve covered in this blog before for the online world and the issue has also been covered in depth by Jim Sterne of the WAA as well as Avinash Kaushik (see Avinash’s The Three Greatest Survey Questions Ever. In terms of the online experience the questions are as follows:
- What is the purpose of your visit to our website today?
- Were you able to complete your task today?
- If you were not able to complete your task today, why not?
There are two things to note here. Firstly, it is a mixture of open and closed questions. Secondly, they are questions that are, to my mind, explicitly transactional. In other words, one visits a website usually for a clear purpose and you know whether you mnaged to do that or not. The purpose of a tourism experience could also be reduced to an almost transactional simplicity but I am not sure that people really think like that in this context so, if we are to use this kind of structure in offline travel and tourism, then we need to adapt it a little.
Before I dive into what I think might be workable alternatives for the tourism and travel industry, I’ll explain why I like this ‘back to basic’s’ approach. It’s because the answers tell you which issues matter. It gives you an insight into your customer’s use of language. It’s specific without being proscriptive. It can be used for marketing as well as feedback.
OK, it might not tell you that they were only 3.4% satisfied by the product positioning statement but, frankly, who cares when you can know that they are all badmouthing your business because the aircon kept them awake all night and the sheets smell of smoke?
So, here’s my suggestion for the three greatest tourism survey questions ever.
- Why did you stay/visit here?
- Were you satisfied?
- If not, why not?
Actually, ‘the three greatest tourism survey questions ever‘ sounds definitive when it’s not -this is a blog and so let’s see if we really can refine the greatest three travel and tourism survey questions ever together.
This entry was posted
on Monday, August 18th, 2008 at 9:26 pm and is filed under Business research, Data, Opinion, Research tools, Tourism market research.
You can follow any responses to this entry through the RSS 2.0 feed.
You can leave a response, or trackback from your own site.
19th August, 2008 at 10:28 am
Hello Stephen!
This is a great post!
I found your tourism survey questions absolutely right,and I believe they should also represent the starting point for a web 2.0 analysis (namely, UGC websites…you know I’m obsessed by them=) ) as well.
With this structure in mind, the research would be dealt with in a more focused and meaningful way.
Thank you!
Have a great day,
Sara
19th August, 2008 at 10:40 am
Hi Sara
Great to hear from you - the weather has taken a turn for the worse since you left!
I had the Web 2.0 experience in mind when I was writing this and I think that Tripadvisor etc are great examples of qual data (which can be modeled to become quant) giving you workable insight.
I think it’s a question of language but I almost feel that although Web 2.0 is a web phenomena, the questions above work better than the previous transactional ones because it is referring to an offline experience - what do you think?
19th August, 2008 at 1:48 pm
I totally agree with you Stephen! the opportunity provided by web 2.0 comments lies exactly inside what you said! they are strictly linked to the real and offline experience and as such they require a different approach from the transactional one!
the website-only experience, even if designed in the most interactive and user-oriented way ever, is (and always will be) stuck to a “virtual perception” from the visitor point of view.
I’m sorry to hear about the weather…on the contrary here since I’m back it’s getting hotter and hotter…
Since it sounds like the Butterfly Effect, at this point I’m not sure if I’ll take a journey ever again =)
Have a great afternoon!
19th August, 2008 at 7:19 pm
You’re right about the sample being tricky. As a professional moderator I know I’m getting a “convenient” sample but the people should be users or purchasers or voters or come from the target population. It really boils down to three things: why do people like you, why do people hate you, and how can you get more people to like you? It’s about understanding the meanings and motivations that drive behaviors. You can measure the behaviors later once you have a handle on what they mean.
19th August, 2008 at 7:54 pm
Hi Tony
I absolutely agree. I think that client can be suspicious of focus groups because they either think they are some silly management fad or because they simply don’t feel that the numbers are rigorous enough (and thereby missing the point of the exercise).
Absolutely spot on as well about samples drawn from target populations - we can all speak about things generally but if I want to know what high-end travelers think about a range of destinations, then getting high-end travelers is vital to the research. From that point on, I can then start to test how far the insight apply to a larger population.
Just out of interest - do you see your skills being used increasingly in an online way (however you want to define that) or do you remain resolutely offline?
19th August, 2008 at 11:52 pm
I haven’t used my skills, as you call them, online yet; but I am increasingly open to considering that. I may be an old guy but I am increasingly tech involved because there is so much communication going there.
20th August, 2008 at 9:10 am
Tim Leighton-Boyce
You’re so right about the value of this kind of analysis, even if it is hard work. I spend several hours a week on the coding and reporting of the free text comments from various ecommerce sites.
One of the things which convinces clients of the benefits of this is the value of the output.
For example, if about 10% of the comments are suggestions for new products or services then part of the routine becomes producing a specific summary for the product development team.
Customers and visitors can be remarkably open with their views on what they expect that you ’should’ be offering. That information is a goldmine for actions which can help you stand out from your rivals.
20th August, 2008 at 9:34 am
Tony: In my experience, some of the abilities you bring to a focus group are exactly the kind of abilities that can then be applied to analyzing other forms of qual research and I think the advantage of having a background in it is that you already have a mind that recognizes what’s significant and what’s not in respondents’ linguistic ambiguity.
Tim: I’m guessing one of the arts of what you do is to differentiate between the usual and the seemingly usual. For example, I assume that most of the comments you get are variants on already known themes but every now and again, there will be something that is just different and needs pulling out. I’m not sure that technology have reached this human insight level yet in open text fields!
20th August, 2008 at 6:29 pm
I think the key for me in analyzing what is said is two-fold - first, I try coming at respondents from a number of different angles including trying to bias them until I’m pretty sure they are telling me what they really think and do; then having a conceptual toolkit in order to find constructs that make sense out of what they say. This is what I call plain and fancy thinking at reasonable rates. Which is the next development on my blog. Too often people report what was said, not what it means to the client. Much of the exercise when it’s most effective isn’t really about the respondents but whether the respondents can educate me into understanding them. My job is to help them express their wants and articulate their needs. So I’m still not sure the new technologies have the degree of complex communication that an in-person focus group provides. But I remain open.
Good post , Stephen.
Tim; you are right about the benefits. I’ve seen clients develop conceptual frameworks with judgmental confidence that allows them to leap ahead and deal directly, now, with the problem. Clients are people, too.
20th August, 2008 at 7:03 pm
Thanks Tony.
I think that one of the big limitations with brief online comments is as you suggest - the lack of complexity. I sometimes wonder whether this is also a lack of context in that a short directed statement on a 2.0 website gives not a clue about the other significant prejudices and influences in that persons life. In other words, you can’t discover the multiple other touchpoints in their lives that make them complete that you can uncover in a focus group.
So how do you get round this? I guess if you’re Google, you should be able to view this persons life in very sharp detail and if you are a company with multiple touchpoints (eg one where you access different products or services depending on mood) then I guess datamining is another way of doing it (I loved this story in Wired way back that touched on this subject) - but I suspect that, for most businesses, the focus group is still the best way forward.
21st August, 2008 at 2:16 am
I know there are difficulties with online groupings but the marketing research industry has a dirty little secret and that is that telephone surveying is becoming less and less reliable because of changing technologies and phone usage and answering patterns.
So I’m going to keep nudging forward on this because I think it will become necessary on some kind of way maybe we haven’t even clearly imagined yet.
21st August, 2008 at 7:57 am
At a market research conference I attended about a year ago in the UK, there were dire prognostications about the impending fate of the online survey too. Basically, it’s going to become harder (and therefore) more expensive to get good online respondents/panels so I think there might be the same problem as with telephone interviews but for different reasons.
25th August, 2008 at 2:20 pm
Hi,
Thank you for a very interesting post. Just I would like to bring into readers’ attention two v. important rules: if you run an online survey, keep it as short as possible and avoid too many open questions as you don’t get a great response rate for this (these) particular question(s).
Myself, I like a research approach that combines qual and quant methodologies but it really depends on a research problem. If we would like to define problems, get insight into a specific issue and listen to our clients, I would strongly reccommend qual research and employ a researcher/ research agency.
Thanks
Ela
25th August, 2008 at 2:48 pm
Hi Ela
I would agree and disagree (sort of) with some of your points!
I completely agree that surveys that are too long will get low response rates (and unless you are going down the line of employing a panel broker, this is an important consideration).
I kind kind of agree about too many open eneded questions but I’m always surprised at how tolerant respondents can be of them - perhaps I’ve just been lucky. I guess, as you say, its a question of balance and too many will undoubtedly lead to a lower response rate.
I would also agree with what you say about the mixed methodology approach and wish that more businesses could take this approach!
28th August, 2008 at 9:49 am
Its better for everyone to understand well any question facing before answering anything because the rightway which am I used is to think first what does that question from customer mean.
Leave a Reply
