Thoughts from the San Francisco eMetrics Marketing Optimization Summit
With my jet lag subsiding a little and my brain still buzzing with ideas overload, I thought I’d share with you some of the insights that I took out of last week’s eMetrics Summit, the world’s foremost gathering of people working in online measurement and optimisation.
This was the year that qualitative research came to the San Francisco conference - a clear signal that businesses are beginning to try and understand the hearts and minds of their customers as they optimize their online channel, as well as their click tracks.
For me, this was an exciting and overdue development, as I think it signals that the online sector is maturing to the point that it is starting to look at 360 degree view of the customer experience - and the business implications of this.
Tom Davenport, the conference’s first keynote speaker and author of “Competing on Analytics: The New Science of Winning” , demonstrated how the highest performing enterprises are building their competitive strategies around data-driven insights. He cited the travel and leisure examples of Marriott and Harrah’s Casinos who are using analytics, data modelling and deep research to identify their most profitable customers, build innovative pricing models, manage customer experience and identify the true drivers of financial performance.
And as conference chair Jim Sterne explained in his wrap up, we’re passing the point of having to persuade the leading businesses why to measure their effectiveness online, because these firms are now using analytics more smartly than ever before and are “looking over the horizon at real competitive advantage”.
My top three picks
My top three practical insights from eMetrics were all qualitative in flavour and all different, but obtainable, examples of how so companies are tackling those fuzzy questions of “Why?” and “How do people think/feel..?”
1. Jakob Nielsen, paying attention to user behaviour in the moment
It was terrific to finally see usability guru Jakob Nielsen speak. I’m a great fan of his articles (you can read more at useit.com) and he inspired the user research approach we use in Highland Business Research (as described in this earlier Tracking Tourism post).
In his presentation Jakob made great use of video of accompanied surfs/user testing interviews to demonstrate the frustration and confusion people encounter as they negotiate their way through websites. He highlighted the importance when conducting user research of paying more attention to what visitors are doing, than what they’re saying - as users missed error messages, failed to find key information and physically recoiled from unexpected music and unskippable videos.
He made the observation that people may think they’ve successfully completed their task, even when they’ve failed (for example, by paying the wrong bill, failing to complete a transaction, missing an error message) - which is why seeing people’s behaviour in the moment has more practical value than self-reported data.
Another critical point from his presentation was the important of starting your visitor insight/user testing process with no preconceptions about the nature of the problems - because “the thing where people think the problem is, is often not where it is at all”. Instead of diving into what you think the problem is, simply get users to undertake the top five tasks on your site.
Just how relevant is this to travel and tourism businesses? Personally, if I could only do one type of online research with clients websites, I’d choose these accompanied surfs that Jakob demonstrated in his videos. The following quote is an excerpt from a research session in which a user booked a flight using his preferred airline, then some competitor sites - I hope you see what I mean:
“Southwest airlines? Best website ever. Other airlines just don’t get it…I quit taking other flights from other airlines because these guys get it right.
[Goes to another airline site]
Do I want to choose on price, flexible dates, schedule? They can just $*!*? off with that… They want to know where San Diego is? That’s just $*!*?… There are too many choices, I haven’t used this airlines site in years but it is still remarkably poor - who has the time for that? I won’t fly this airline, its web site is junk!”
Jakob’s presentation was packed with clear evidence that people aren’t going to work round a bad website, they’re simply going to take their business elsewhere!
2. Ebay, making themselves at home in your home
Elissa Darnell, Director of User Experience Research at ebay, did a great presentation on how they use a blend of offline and online research techniques to go back to the basics of who the user is, what they do and how ebay can deliver them the best possible experience.
Elissa talked in depth about a number of the techniques they use to get closer to customers and optimize their site experience accordingly, but I’ll just highlight two:
Follow homes - just like some offline firms have done for years, ebay are following their customers into their homes to observe them using the site in their normal way and in their normal environment (with all the distractions that come with that). Using a video camera and taking notes, they gain valuable directional, qualitative information that they can combine with their number based information sources.
Its not just the research team that take part, staff from right across the company - from CFO & CEO to developers and service reps - are taken into their customer’s own context, to learn for themselves that critical lesson of “we are not our customer.” The insights they glean from this regular follow home process means they learn things about their user’s needs and behaviours that they would not otherwise even think to ask about. They also gain direction on where they made need to focus further testing and development.
Cheap paper based user testing - before pages are even developed, ebay conduct quick and dirty user testing based on rough paper mock-ups of what the page might be like. The tester is asked to use their finger as the mouse and indicate where they might go on the page and what information they might expect to find as a result.
They can quickly go through rapid generations of the paper tests, quickly and cheaply, before ever spending money on software engineering. Jakob Nielson described the cost savings of this approach as being around 100 times cheaper than testing actual pages. So clearly ebay are using tactical pre-development testing make sure they get maximum user focus for their money.
3. Pay Pal, quantifying the qualitative
Pay Pal, the online payments provider, receives thousands of open comments each week from its user feedback pages. More than one person can manually react to and keep in their head at any time.
So their challenge (one I think many tourism businesses can relate to) is how do you quantify and semi-automate the process of analysing these open comments, so that you wring maximum value out of the feedback?
Step one, is they have applied a Google search engine tool to the data and made it available to internal staff to search through as they wish. This means that managers can at least get an overview of what is being said about specific interests (and in what volume) on a regular basis. The advantage is no training burden and no resourcing of the data analysis.
Step two, they have also semi-automated the process of categorising their comments, by bundling like comment types (through an in-house natural n-gram language statistical process). Once bundled into like themes (without needing to pre-develop a naming structure) they can get the comments out to all the right people to read, on a two weekly basis.
Step 3, now they are working out a system that automatically triggers alerts as new bundles of comment types emerge, or certain types of comments exceed their acceptable threshold.
It may sound complicated, but essentially with the help of a good statistician, they have built a simple system in-house that achieves most of their needs, but avoids using a highly expensive Meaning Based Computing system like Autonomy.
And finally…
As a final conference highlight (at least from my perspective!) - both Stephen & I were pleased to present at eMetrics this year (a kind of Highland Business Research double bill).
I was also honoured to be elected to join Web Analytics Association board of directors. So thank you very much to those WAA members that voted for me and I look forward to helping drive the organisation forward for the benefit of members and the web analytics industry worldwide.
This entry was posted on Sunday, May 11th, 2008 at 6:39 pm and is filed under Conference learnings, Online customer behaviour, Web analytics and web measurement. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.
