We work a lot with tourism and travel providers operating customer satisfaction feedback systems to help improve their services and offerings. But I’ve met a few people recently who have expressed scepticism about customer rating systems generally and it strikes me that this scepticism could be the result of not looking at the data in a more rounded context. Or due to receiving data derived from flawed methodologies.
The scepticism was expressed along the lines that these kind of things always show that 90% of customers are satisfied. The implication of this is that rating systems aren’t really telling you the full story. So, while we’ve previously written here and here about using comment cards, these recent comments show that there is still a little more ground to cover in this area.
I can understand the view that customer rating systems are inadequate – but this typically occurs only if you are looking at the data derived from the customer in isolation. As we wrote in one of the previous posts, “comment cards are just one of a suite of businesses information sources”. In other words, you shouldn’t rely on comment cards alone for customer feedback in its broadest sense. (And with such rich data all around you, why would you want to ignore the other sources?). But let’s start by looking at this “90% satisfaction guaranteed” issue a little closer as I feel that a rating like this is not as pointless as critics suggest.
To my mind it’s all about context. A 90% satisfaction rating expressed as a snapshot of customer sentiment can be fairly meaningless. However, a 90% satisfaction rating for an activity compared to (for example) the rating for a different activity, a different period or even a different location does start to have some meaning.
It’s about trends, not absolute scores. It’s about comparisons, not absolute ratings. It’s about context.
But let’s have a look at this using some real data.
Context One – data over time
The following charts are drawn from ‘real life’ but have been anonymised.
Starting with the one on the left (click on it to open a larger version in a new window), the orange line represents a lower control limit (one standard deviation from the average downwards meaning that 68% of all monthly results ever fall within this range – and if you are interested in why I’ve used only one standard deviation, see the second comment at the end of this post). The average is the grey line in the middle. The blue straight line represents an upper control limit (again one standard deviation but upwards). I’ll explain the purposes of the control limits in a moment. There is also a dark grey line which is the trend of the scores.
In this first graph we see a green line charting the percentage of people who completed a comment card for a particular aspect of their experience and who said that they were satisfied. Looking at this line, we can see that indeed it hovers around the 90% mark but that there is some variation. So what can start to take from this information?
Firstly, you will expect some degree of variation when analysing data one month to the next – it’s just natural. But there are times when a change is ‘unnatural’ and this is when control limits come into play as they alert you to when something has fallen outside of the normal corridor of performance. And these control limits can only be derived from looking at this data in a historical context as this gives you the most realistic guide to what is normal and what isn’t.
Secondly, looking at the trend line you will notice that, if anything, it has dipped a little. It’s probably nothing to be worried about. But if, for example, the line represented a customer service rating and, despite months of internal training, around one in ten of your customers were still leaving feeling that they had got substandard service. Wouldn’t that be a concern to you?
Context Two – data compared
In our second example on the right (click to enlarge), there is now a second line of data about a different service. This was rated at the same time as the first service and by the same respondents.
Firstly, it should be noted that these lines are not moving in lockstep (they actually have a correlation coefficient of around 0.25 indicating a practically non-existent relationship). The trend lines further indicate that the levels of satisfaction are moving in opposite directions and so we have a clear indication that, despite the high ratings for both lines, the responses are nevertheless suggesting that there are differing levels of satisfaction with them.
Now we’re starting to get towards something useful. We can start to ask what is going on to make people less satisfied with service A than service B over time. It is even possible to start to test operational changes to look for a positive uplift.
You MUST be happy!
The context in which the customer feedback was taken can also affect satisfaction levels (although the data I’ve worked with suggests that aggregated satisfaction levels tend to be quite similar). For example, I analysed the results of feedback from one destination where the respondents were required to hand the completed score cards straight to the accommodation provider collecting the forms. Unsurprisingly, 85% of people claimed to be elated by their recent accommodation and 5% dared to only be satisfied. In a context where the data was collected more anonymously, this split would probably be something more like 55% and 35%. In both cases, we could argue that 90% were satisfied although the second example is probably closer into the truth.
In a situation where you do have frequency data for all the scores (ie counts of how many excellents compared to how many satisfieds), it is worth looking at it in some more detail to get sense of how sentiment is shifting.
For example, are there more ‘good’ than ‘excellent’ scores? for most items but for a few that situation is reversed? This might indicate that while 90% are satisfied (’satisfied’ being, as noted earlier, the ‘goods’ and ‘excellents’ summed) the balance of satisfaction lies at the lower end than the upper for most items. And that those that buck this trend are worthy of note.
But what is satisfaction anyway?
But there are still important questions floating around in the background here and they probably all flow out of the main one of, “what does ’satisfaction’ mean?”
What I mean by this is satisfaction indicate something good, all right or possibly inadequate. For example, the data behind the charts above is coded in such a way that the rating delivered by a respondent is give a numeric value (eg bad = 1, adequate = 2 etc). From this it is possible to calculate that your visitors were 4.2 out of 5 happy this month. Unfortunately, such an approach can also demonstrate that your visitors were 1.4 out of 3 female, something that is just plain silly.
So, the approach we have taken for the purposes of top line reporting is simply to allocate results to discrete bands – if, for an example, the score is a 1 or 2, then it shows that the customer was dissatisfied, and anything above that suggests satisfaction. This means that you get an easy overview of the level of satisfaction.
But, you might say, I’m not interested in people being satisfied, I want them to be elated! A noble goal to be sure but I’m not sure just how elated one can be at the process of buying a coffee or noting that the toilets were clean. There are some things that just don’t excite people that much to cause them to rate them highly in feedback forms! They’re only notable when they go wrong!
But I guess that this is all really confirming something we’re said in the past – if you are measurng customer satisfaction but only skimming the data then you are potentially wasting your time. Only through a more indepth and intelligent use of it can start to yield up the nuggets useful for your business.
Filed by Stephen (07/04/09)
This entry was posted on Tuesday, April 7th, 2009 at 8:08 pm and is filed under Business research, Data, Research tools. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.