Blink UX
Hardware Usability Benchmarking UX Research
UX Research: Benchmarking Study for a Fortune-500 Client
UX Research: Benchmarking Study for a Fortune-500 Client
Elevating user experience with wearable devices through series of benchmarking usability tests.
MY ROLE
UX Researcher
Testing Session Moderator
2 Months Total
5 UX Researchers
Qualtrics, Figma, Feedback Panel, Slack
1 Project Manager
1 Director of UX Researcher
2 Research Assistants
May-June & Nov-Dec 2024
DURATION
TEAM
TOOLS
OVERVIEW
I was hired through Blink UX to consult for a Fortune 500 Company leading the industry in wearable technologies. Blink UX is a San Diego local design consulting agency who has had clients including but not limited to Microsoft, NASA, Amazon, Google & Meta. As a UX Researcher at Blink, I was involved in a total of 3 projects from May~December 2024.
Project Type
Benchmarking Study for a Leading Fortune 500 Company
I contributed to the first in a series of benchmarking studies designed to span over five years, with the goal of continuously enhancing the user experience of the client’s products.
Each benchmarking study builds on the previous one, employing consistent research methodologies and analyzing data to track progress. This approach enables the identification of both improvements and areas that require further attention, providing actionable insights to refine the product's performance and user satisfaction over time.
How did we do our Benchmarking Study?
Let’s look at a fun example: Building a Snowman!
Project Overview
Improving UX in Innovative Virtual Reality Wearable Devices
We are tasked to measure specific functionalities and features in the client’s product. Through our research, the clients has some clear goals and deliverables laid out for us.
Goals
Identify pain points and bugs in the technology and product.
Understand the underlying sentiment correlation with frustrations and pain points identified in the study.
Results: Collected and synthesized a total of 150,000 datasets across three projects to provide foundation of the benchmarking study.
Insights Delivered:
Correlation Between User Pain Points & Emotion Scales: Measured user frustration, satisfaction, and frustration levels across product interactions.
Effectiveness of Product Feature Changes: highlight feature changes that improved user experience and vice versa.
Feature Performance Comparisons: Compare older and newer version of features, showing clear trends of UX improvements and declines
Popularity Metrics: Generate prioritized view of user preference based on most and least product features.
Deliverables
Report with qualitative and quantitative
findings
Prioritized task issues list
A cleaned data set
Client Stakeholders
Research Director & Lead Researcher
Research Timeline
The Research Timeline for Benchmarking Studies
We are tasked to measure specific functionalities and features in the client’s product. Through our research, the clients has some clear goals and deliverables laid out for us.
Timeline
Planning
Training
Data Collection
Data Synthesis
(5 Days)
(2 Days)
(5-7 Days)
(5-10 Days)
Communicate with stakeholders, Finalizing on tasks to measure
Creating Qualtrics, Finalizing task flow
Writing up lab notes/ research guidelines
Run pilot testings with stakeholders and participant
Ran 4 two-hour-long usability testings per day, over a span of a month
Compile the Issues List for data synthesis
Identify trend and patterns for product usability and user frustrations
Generate data comparison reports in relation to previous data benchmarking projects
My Role
Planning Out The Testing Task Flow
Series of tasks are going to be tested during the sessions, so we need before we actually start the participant testings, we want to make sure everything is laid out in a logically chronological order.
That required a lot of user journey mapping pre-sessions, and accompanied by our internal pilot testings and external pilot testings.
Process of Planning Testing Flow
Planning Task --> Predicted User Journey Mapping --> Internal Pilot Testing -->
1x External Pilot Testing
Collecting User Inputs and Data
As a research moderator, I conducted 1-on-1 usability studies in a lab setting. During each session, I used a Qualtrics data collection form, followed research procedure guidelines, and observed participants through a computer monitor to analyze their interactions with the product.
Key Observations:
User Journey Flow: Tracking how users navigate and complete tasks.
Disruptive Activities: Identifying bugs, user confusion, or unexpected behaviors.
Real-Time Documentation: Logging findings in Qualtrics to ensure data integrity and prevent misleading insights.
Data Documentation & Synthesis
We collect over 150,000 datasets per study, which is a large amount that can easily become really complicated and confusing when we were trying to make sense of the data.
That is why we decide to dissect data documentation and synthesis into smaller digestible dosage. After every session that we ran, we document hardware issues that we encountered into the Issues List.
Compiling the Issues List: Issue Tracking & Reporting
Logging Device Issues: Recording when and where problems occur and how they impact the data.
Purpose:
Providing quick reports to clients to request bug fixes for device hardware.
Ensuring usability reports reflect only UX-related insights, not hardware failures.
Preparing for Figma Affinity Mapping by distinguishing UX pain points from technical glitches.
This structured approach allowed us to focus on design-driven usability improvements, ensuring a clear separation between UX challenges and hardware malfunctions.
Identifying UX Themes: Figma Affinity Mapping
Using data collected in Qualtrics and insights from the Issue List, we compiled a comprehensive Figma Affinity Map to identify recurring themes and patterns in preparation for the client data report.
Purpose:
Transform low-level user issues into grouped insights.
Identify high-level UX themes that highlight key pain points.
Provide clients with actionable steps to improve the user experience.
This process helped distill raw usability data into clear, strategic recommendations for product enhancement.
Issues Tracking List
Identify recurring themes and patterns, looking at only UX product design issues
Purpose:
Low-level user issues-->Grouped Insights
Prepare for final client data benchmark report. Recommend actionable steps to improve product
Records the location and timing of device’s hardware bugs and issues
Purpose:
Pinpoint outlier data caused by
device hardware bugs
Easy to traceback during affinity mapping creation process
Affinity Mapping
Data Reporting - Client Presentation & Benchmarking
The fact that the benchmarking project’s purpose is to compare with previous projects that had used the similar testing methods and testing metrics on similar tasks, there needed to be sufficient explanation and data evidence backing up drops and ups of task performances.
Our team lead is in charge of directly communicating with the client stakeholders reporting the results of our testings. Whenever they need more clarifications on the number dipping or rising, I am tasked to provide evidence-backed explanation on why a certain task faced significantly lower scores compared to the previous round of testing.
On the report, the performance of each task is indicated through color coding: Green (good), Orange (moderate), Red (poor) for quick readability.
Impact of Our Work
Establishing for 5-Year-Benchmarking Projects
Our projects serve as the foundation for a five-year initiative focused on data collection and usability testing, tracking the product’s evolution through ongoing updates. As the first phase of this long-term benchmarking effort, our work establishes a critical framework by identifying technical challenges, refining testing methodologies, and ensuring the reliability of performance metrics for future evaluations.
Identified
Technical
Challenges
Refined
Testing Methodologies
Ensured
Reliability of Performance Metrics
Our Work Marked The Start of a 5-Year Benchmarking Initiative
Impact:
For the next five years, the client will leverage our project's data to drive product changes and new feature development, guaranteeing a consistent approach across their projects.
