Not All Research Data Is Created Equal
The way you plan and conduct your UX research and gather quantitative or qualitative data plays a huge role in the quality of that data and determines how much you can rely on your data without taking huge risks.
You should focus on two things: First, what are your learning objectives and what data do you need to achieve them? Second, how can you gather the necessary data in a way that ensures it’s valid and reliable for use in decision making?
What You Need to Learn
What do you want to learn? What data do you need? What kind of data do you need to be confident in making your decisions? Define the key questions you need to answer. Skip all the vanity metrics that tell you almost nothing and, instead, focus on things that matter.
How to Gather Data
Once you’ve defined what you need to learn, you can plan your research. This is where both internal and external validity become important factors. Doing research in a scientific way can help you to avoid most research pitfalls and ensure that you gather valid data.
Data is one of your organization’s most valuable assets. Treat your data with respect!
Conduct testing to learn, not to prove you are right. Avoid jumping to conclusions—for example, if you get some encouraging results from a single data source. While this would be a good indication, you should not be hasty in drawing conclusions. Don’t forget to triangulate your data. Triangulation helps you reduce risk and achieve your desired confidence level. Calculate a confidence interval that shows how confident you can be that a certain result would occur.
If you want to know more about how you can use multiple data sources to improve your decision making, you should definitely take a look at Tricia Wang’s presentation about how relying on just big data and ignoring qualitative insights killed one of the most successful companies—Nokia. If you focus only on collecting quantitative data and don’t speak with people to gather qualitative data, you’ll get a skewed view of the world.
Never forget that correlation does not equal causation!
UX Research Is Not Novel
User Experience did not invent conducting research with people, about people. Social scientists paved the way with battle-tested methods that meet most, if not all of the needs of UX professionals. We can adapt their methods as necessary.
Be Scientific When Conducting UX Research
The scientific method, shown in Figure 1, is a powerful approach. You should use it for your UX research. This would enable you to overcome your own cognitive biases, improve the quality of your research, and collect clean, useful data. Dr. Nick Fine has given a spot-on talk about how to practice scientific design. Check it out!
Treat Requirements as Hypotheses
Change your mindset and handle requirements as assumptions. Then turn them into hypotheses that you can test. To do so, you must determine your organization’s desired outcomes. John Pagonis, in his talk “Evidence-Based Product Backlogs,” explains how to use user research to prioritize and cleanup your backlogs. Laura Klein’s talk “Identify and Validate Your Riskiest Assumptions” describes how to identify and test your riskiest assumptions early on in your projects. Both are spot-on. I definitely recommend your watching these videos
Test whether your hypotheses are valid. The sooner you can gather enough evidence, the better. If your hypotheses prove to be valid, great! You can start building your product. If your early testing indicates that you should reject a hypothesis, you’ve just saved your organization a lot of money! Think about it: You didn’t have to throw away any code. Your team’s morale didn’t take a hit. You’ve spent no money on marketing or promotion. No one was screaming about negative business outcomes in your office—or your Zoom meetings.
The Importance of Documentation
When you document your UX research—from research planning to execution, including data collection and analysis—you and anyone else in your organization can check the validity of your findings. If you make this common practice, it becomes way easier to find and exclude bad data from your decision-making process. Plus, this makes it much harder for other people to come to the table with so-called evidence—data they’ve made up or that is inconclusive. In their article “How to Handle Bad Data,” James Lewis and Jeff Sauro describe how to identify and handle bad data, including information about how you might have ended up with bad data and what you can do to mitigate this problem.
We need to help our companies become data-driven organizations, then keep up the momentum. But we also need to take a scientific approach and be vigilant about how we collect and treat our data. UX professionals must avoid making mistakes that would diminish our credibility and create barriers to the adoption of evidence-based, data-driven ways of working. We need to educate ourselves, our teammates, and our stakeholders about correct research practices and their proven advantages. As Sotiris Sotiropoulos has said, “You gotta fight for your right to research.”
Of course, I understand that not everyone can conduct rigorous UX research from the outset. It’s a learning process. But take one step at a time, sustain your momentum, and keep moving forward. You must adapt your UX research practices to your specific context, needs, and constraints. Do your best to provide useful research results to your team.
If you’re just getting started with UX research, educate yourself so you’ll understand what sloppy research is, be able to identify it, and avoid practicing it. You can’t beware of potential problems with research practices if you don’t know they exist. Integrate your new knowledge into your work. Start with your next project. Once you try conducting UX research, make the necessary adjustments. Design solutions based on the data that you’ve gathered. Do not get discouraged. Stay on track and keep moving forward.
Share your knowledge about UX research with your team and stakeholders. Creating a shared understanding should always be one of your goals.
Please share your thoughts on this topic in the comments. I’d love to start a conversation in the UX community and raise awareness of risks, opportunities, and best practices.