Competitor usability testing – is it useful?
This is a topic that I’ve discussed with colleagues in the past, but never taken the time to write down.. can you (and should you) do effective and useful usability testing on competitor websites?
In my opinion, this is not a good use of test and analysis time.
There’s nothing to stop you asking people what they like or don’t like about other sites, or even performing an evaluation based on simple usability heuristics, but performing scripted tests on competitor sites is not going to give you information anywhere near as valuable as testing on your own site. And even when you uncover problems, you’re not able to solve them in the context of that site.
I was reminded of this when I took the Afrigator guys to task for the poor user experience on their login page in my post about federated identity.
I was politely reminded in my comments by Stii and Justin that they have a very technical userbase, and there were technical constraints in terms of performance and UI that were imposed by RPX, the supplier of the technology they are using to manage these logins.
There are 3 points here:
- If you have limited usability testing resources (everybody has limited resources – test subjects, facilitators, time), you should focus all you have on running iterations of tests on your own websites, fixing and re-testing multiple times
- Without being an insider who built the site under test, you have no understanding of the design parameters and constraints experienced by the development team
- There may be cultural effects, or preferences of a particular subculture that you do not understand
The first point is pretty obvious… if you can only afford the time and effort to run 15 facilitated usability test sessions, you will get a lot more value out of running 3 rounds of testing with 5 subjects on your own site than you will from running 1 round of tests on your own site, and testing a bunch of sites.
You’re not able to fix the problems on those sites, so you will never be able to test whether you can improve the experience for users.
The second point is that you don’t know what constraints might be imposed through technology choices, legacy systems or performance issues. In the case of the RPX supplied login system on Afrigator, performance concerns and the limitations of the free version of their service had an impact on the interface.
At 24.com we’ve experienced this – we’ve tested competitor sites, and discovered things like “users struggle to find the search box”. What we don’t know is why the site designers chose the UI they did… maybe the quality of their search results is poor.. maybe they prefer users to browse rather than search… without understanding the context for the UI decisions that were taken it’s hard to criticise them in a useful way.
The last point is about a different type of context – that of a particular culture or subculture. A great example in two parts comes from Taobao, the most popular C2C trading site in China.
The business and site was discussed during the recent Naspers Harvard program I was fortunate enough to attend through 24.com (the case study can be found here through official channels but can probably be discovered elsewhere through a Google search)
Taobao beat eBay in the Chinese market for a number of reasons related to their deep understanding of their local users including their design and I’m not going to go into them in any detail here.
There are some interesting elements to their interface design that may have confused any usability testing undertaking by eBay in attempting to understand their competitor – the first is that the page is very busy, text-heavy, and tightly-packed. Classic interface design for the web says that you need to make good use of whitespace and use a heirarchy of different font styles. As you can see, Taobao flaunts these conventions, and manages to satisfy their users with unusual interface elements like Men/Women tabs modelled on the divisions within Chinese department stores.
The second piece of UI that I find fascinating is the reduced emphasis on customer ratings (a key piece of eBay’s offering), and an indication instead of when a Taobao user is online and ready to chat.
In the image above, the first seller is online and ready to talk to prospective buyers, while the second is not available. Taobao gives it’s users tools to build what it calls “swift trust”, and without understanding why these tools are available or why familiar alternatives are missing we can’t perform a meaningful analysis of their site.
Without “walking a mile in their shoes”, it’s not possible to truly understand why particular decisions were made by competitors when building their sites.
Related articles by Zemanta
- The Myth of Usability Testing (alistapart.com)
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_c.png?x-id=c1948d5b-84ec-4c00-a903-4a50ecddc24e)
Related posts:
Hi Tim
I disagree with your statement that running usability tests on competitors are not useful. I have done plenty of these kinds of tests and it comes into it own when you are testing a piece of functionality the team want to introduce into the site and a competitor is doing it already. Testing whether customers have adopted the functionality is great and gives a lot of insight into how your own site could be affected by the additional functionality.
We constantly faced by marketers that see pieces of functionality on other sites and request the inclusion of these, a competitor usability test will provide the necessary input on whether its valued or not by your customers.
A good example of this was the introduction of tabs into the UK market: we were asked to consider the changing of our primary navigation to tabs as per all our other competitors, we went into a competitor test only to see none of the customers clicked or even noticed the tabs on our competitors at all, we ran a series of tests and found consistent results. We did not include this functionality and found 6 months later all our competitors switching their tabs off, when asked they said they found they had no click-throughs on the tab navigation. Considerable amount of money saved not introducing the functionality that we just would have turned off again.
Most E-commerce sites internationally test their competitors on a regular basis against their own site to ensure the functionality offered to their customers are still valued and gives measurable comparisons in terms of a market landscape for customer experience and usability.
I have only had positive and valued input in doing competitor usability tests, however you always need to include your site into the mix of competitors to measure your own site’s effectiveness against your competitors, if you are tight on budget testing your own sites functionality should always take preference.
If budget allows you should test your site against competitors at least once a year to get a competitor landscape and be able to see where you sit in relation to your competitors as far as offering a usable site. However test against real competitors and not just an international mix and test with your own customers and not just a random selection to ensure you get the best results that can be used within your business.
These tests need to be well scripted to ensure consistent questioning and results. Apple for apple comparison is key to get good solid findings from such a test, but the rewards for running such a test far outweigh not running it.
Thanks for the insight Helga… I don’t place as much stock in competitor usability testing as you do, but we’re not taking opposing views on the subject – my opinion is that more value is gained from multiple tests on own sites rather than tests on competitors. The scenario you are describing (testing particular UI) could have been done on a competitor site or a paper-prototype or a rough version of the site under development. The fact that it was a competitor site doesn’t seem to be that relevant?