Ever forgotten to have lunch or take a break while working? Me too. Sometimes, you get so absorbed in what you’re doing that you forget to take a look outside. And that can be a shame. Whether your expertise is in web design, industrial design or even architecture, never underestimate the power of leaving the office for a while and getting to know the people that actually use your product. It’s an incredibly valuable tool than you can use to improve your creations. But don’t only trust me – my good friend Einstein had a similar insight: “The significant problems we have cannot be solved with the thinking used to create them.”
As a UX designer at Wix, a big part of my role involves understanding users’ needs and wants. To do this, we employ usability testing. Here is a case study of a recent test we carried out on the Wix Mobile app. We needed to check a specific feature, that allows our users to automatically create social posts promoting their business online. At the time, it had been live for three months and before creating a desktop version of the app, we wanted to test it out. This was the ideal moment to take a break, listen to others and improve the product. Here you’ll discover all the stages we went through, from setting our goals, to choosing who to meet and how to analyze users’ reactions.
What is usability testing?
Before we delve right in, let’s understand what we’re talking about. A usability test is a technique adopted in user-centered interaction design (like UX) to measure whether a product is meeting its intended purpose, by checking it with representative users. Design is, after all, about a lot more than just creating incredible visuals. The way I see it, the whole process is like filmmaking. It starts with “pre-production” – a critical stage in which you plan everything. Then you move onto the “on-stage production”, which is when you film and collect all of the information. The last stage is (you’ve already guessed it) the “post-production” when you go home with all the material you’ve collected and analyzed your findings.
How to get ready for a user test
Out of all the stages involved in conducting a usability test, the most crucial is the “pre-production”. The preparations you make can be a deal-breaker, seeing as during the test, you won’t really have the time or opportunity to fix any major problems. The secret to success here is: prepare, prepare, prepare. Follow these steps to ensure you’re ready to take on anything that comes your way.
1. Set your goals
If you’ve chosen to test a specific product, you probably already have an idea of which aspects on it work well and which less so. According to the knowledge you’ve already gained, define clearly the main aims of your evaluation. Knowing exactly what you’re testing will help you make decisions later on, such as which questions to ask and which method to use when observing the user. For our own test, we identified four goals: (1) whether the user can easily find the social feature in the app, (2) if they understand its essence and purposes, (3) how they feel while using the app, and (4) whether they actually need and use all the functions offered.
On top of these general aims, we had two questions we hoped the test could help us clarify: what are the users’ exact needs for their businesses, in terms of content, and why do people seem to be leaving the app when they get to the “template” page. As a general rule, don’t be afraid to tackle the most sensitive questions – you don’t want to leave any problems undealt with, because of your ego, or fear of what the answers could be.
2. Define the method
Out of the many techniques for moderating usability tests, we decided to use the ‘Emotional Reaction Method’. In this approach, you (the testers) are completely passive, simply observing the users’ initial natural emotions when using the app and making sure not to interfere with prompts or comments. This way, the users are faced with a blank canvas and their responses are the most authentic they can be. Of course, if you find that they’re getting frustrated with a certain function, you can give them a hand and then let them continue alone. We found that you can learn a lot from analyzing a person’s reactions, noting their words, tone, facial expressions and actions. During the tests, it was interesting to observe how success and frustration affected people’s behavior differently – from the widening in their eyes when they managed to publish a post, to their body’s agitation when they couldn’t find something.
3. Decide who you want to meet
Start off by defining your product’s typical user typecast. Where are they from? What is their line of business? Your users will be very diverse, depending on your type of product. The people you’ll test a food related app on will most likely be different to the users of a news app, so understanding your target audience is key. Since Wix is used by almost 120 million people worldwide, we wanted to meet as big a variety of users as possible. We approached people from a range of backgrounds, with different types of businesses, personality traits and ages. We ensured that half of the users were “advanced” (having Wix accounts for a long time), while the others were “beginners” (having recently created a website). In terms of how many people to meet – there’s no magic number. It depends on your product, but generally the best practice is to begin by meeting roughly ten and taking it from there. If you find that all of them give you the same feedback, then you’ve probably tested on enough people. Alternatively, if each person has a different opinion, you should expand your testing.
4. Choose a location
Once you’ve decided which users to meet, you can pick a suitable location. Ideally, find a quiet place, with as little distractions as possible – you want to create a distraction-free environment in which everyone taking part will stay focused throughout. You should also go for a neutral location, where neither you or the users have any commitments, such as tending to pets at home, or answering to other colleagues in your own office. For our test, we decided to choose the Wix Lounge in New York, where we could easily contact dedicated, cooperative users that probably wouldn’t cancel last minute. The downside was that we didn’t get the most diverse audience, with all of the users being New Yorkers.
5. Plan the set
Just like a film set, you should arrange everything in advance. The more prepared you are, the easier it will be on the day. Decide on the crew – which team members should take part in the test? You should always involve a UX designer and a product manager, as they are the ones that created the product and know the processes it’s gone through from start to finish. List what you’ll need to bring, define each person’s role and plan the seating arrangements. It may sound extreme, but we found that sketching a diagram really clarifies things on the day. There should be a “tester” that sits next to the user while they’re using the app, and an “observer” sitting outside the room taking notes.
6. Pick the right technology for your needs
Your technological needs will vary depending on which product you’re testing. Either way, you’ll want to record every single action within the session, including the user’s face, their words, their behavior and, don’t forget, the product screen – so that you can later connect the user’s responses to their actions on the product itself. Ideally, you want to be able to focus on observing the user, without having to take notes. To help us with this, we used tools, such as Photobooth to record the user’s face and voice, and Slack to broadcast everything to the “observers” outside the room. This enabled them to be part of the session without interfering. Check out this list for a full range of handy tools for usability tests.
7. Create a schedule
When creating a schedule, the most important thing is to be realistic with the timings. Treat yourself to some spare time – it will enable you to deal with surprises and will give you enough time for breaks between sessions (seriously, you’ll thank me later!). Decide on the length of each meeting, book your location and finalize the times and dates with your chosen users.
8. Write the script
So, you’ve got up to this point and it’s almost time to meet the users. But what do you actually say to them at the moment of truth? Start by breaking the ice with some informal chit-chat, getting to know the user and their business. Next, reduce tension by presenting yourself, ensuring they know that you’re testing the product (and not them) – there are no right or wrong answers. Decide which task you want them to perform and consider the possible scenarios they could encounter. To test our feature, we chose a clear ‘go sentence’: “Find a way to promote your business within the Wix Mobile app”. We didn’t want to lead them in any way by giving hints, so we avoided certain terms, like “social post”.
9. Carry out dry runs
Just like playing the piano, “practice makes perfect” works here too. Conducting dry runs will not only give you confidence, but will also clarify what equipment you need to bring and which programs to download. Before the real deal, we practiced on Wix users in our Tel Aviv offices and also tested the technology multiple times.
Lights, camera, action!
Congratulations! You’re now ready to conduct the actual sessions, fully prepared for anything that may come your way. Or not. We were met with some real surprises – from users talking for a lot longer than planned, to spending over half of the session showing us pictures of their dog (this is no exaggeration). The best tip I can give here is to go with the flow and expect the unexpected. The whole process is a lot more tiring than you would think (one of the “observers” on our team almost fell asleep at one point!), so prepare yourself for some hard, but extremely rewarding work.
At the end of each meeting, sit with your team straight away. Don’t even take a moment to go to the restroom – trust me, even the shortest of breaks can disrupt your train of thought and lead to you forgetting crucial details. Conduct a summary on the spot, writing down everything from the session and numbering the issues you found according to their order of importance. We created a spreadsheet and divided our findings into categories, like ‘missing feature’, ‘happy moment’ and ‘design presets’. This helped us order the extensive information and essentially made our lives a lot easier when we got back to examine the results.
Analyze your findings
As soon as you return to your office, work on the post-production, while the experience is still fresh in your minds. Start by summarizing your findings. Once you’ve created some order, analyze the information and come up with action items for your team. The results we found were extremely valuable in allowing us to pinpoint how to improve the product. Firstly, we found that many people weren’t aware that the app existed, so we had to increase its discoverability. We also understood that we should enable users to delete components and schedule posts, offering a more comfortable and professional experience. Looking back, we understood that we had covered all the goals we appointed ourselves at the beginning. You should check whether you managed to reveal all the initial mysteries that you identified in the first stage so that if not, you can go back and test them again.
We also found that it’s important for the whole team to feel a part of the session, including those that weren’t involved in the test itself. You should try to convey to other team members the full experience you went through so that they’ll also have a complete understanding of the test’s implications on your product.
Through meeting our users face to face and conducting the test, we not only obtained essential information but also gained a boost of motivation to improve the product. You just can’t compare this kind of interaction with a phone call or survey. It also lets the users know that we care and want to hear their opinions. It’s truly a win-win situation.
