Case Study
How user research influenced goal setting, feature prioritization, and design iteration to ship a measurably better search/SEO experience for IBM Cloud Docs.
Situation
Objective: Help users find the right documentation by returning fewer and better search results.
Users were unhappy with the current search which yielded too many results and users were unable to refine search to narrow results, causing users to leave IBM Cloud Docs to search on google.
Tasks
As the Lead User Researcher embedded on this and one other workstream, I was responsible for planning and conducting research.
- Understand pain points and unmet needs with existing search
- Aid with roadmap prioritization of new search features
I led Design Thinking workshop facilitation, grounding decision making with user insights.
Actions
We had 6 weeks to gather user insights in preperation for our prioritization workshop.
- Research rollup
- SME interviews & domain research
- User interviews
- Ideation workshop
- Kano study
- Prioritization workshop
- Rapid Iterative Testing & Evaluation
- Behavioral Analyics
Results
Research insights enabled collaborators to confidently align around a 3-Q roadmap to deliver an improved search experience.
- 2x in-app searches (fewer users leaving Docs)
- Significant increase to IBM Cloud Docs search click through rates
- Nearly 50% user engagement in the new filter after getting their initial search results
- Patent for new filtering capability
Research rollup of existing user feedback to identify issues with search.
- 12 in-app user feedback comments (Usabilla)
- 9 NPS responses (Medallia)
- 4 customer issues shared by CSMs (Slack)
Too many search results
“Searched the documentation for "kubernetes pricing". There are 792 (!!!) search results."
Unable to refine searches
“I'm looking for something specific, but the more terms I enter in the search box trying to narrow down the results, the more results I actually get."
Search results do not display enough information
“It’s incredibly difficult to scan search results for relevant content. Titles and a broken sentence or two really aren't enough.”
SME interviews & domain research to understand the upcoming changes in search capability and performance.
- Marketing’s Program Director of Digital Strategy to inquire about the search and SEO data driving our strategy (e.g. search inputs, SEO performance, page visits).
- Senior Software Engineer to understand the search implementation and how content attributes (e.g. titles, headers, and keywords) would and could impact search results.
Behavioral analytics related to project KPIs:
- % (most) of B2B journeys begin with an online search
- We see 2x greater Engagement rate from organic traffic vs paid media
- % (far too many) of pages get no organic Google search traffic (we create too many pages)
Secondary research on search & SEO best practices.
- 8 external articles
- 2 prior studies by IBM Researchers
- Explore search on 3 competitor docs
User interviews to learn more about existing pain points and unmet search needs.
- 30-minute remote interviews
- 9 participants from US, EU, and Asia
- Interviewees walked through a recent search experience
Gained a more contextual understanding of user behaviors observed in our analytics:
- 100% of participants ‘google it’ before searching our docs, validating the need to improve SEO (a project KPI).
- The most common search behavior is to ‘drill-down’ using keywords, beginning with a product name.
- Participants complained of too much documentation and that results return the same document multiple times. They workaround this by navigating to a sub collection, then performing a scoped search.
- Low page views of translated content is more a matter of trust than preferred language.
“At least in Europe, we try to access only the English content because we trust it more.”
Gap in our content strategy taxonomy: API documentation
If APIs are available, they are often buried within a subcollection. Participants were in favor of an API hub.
Users unable to navigate in and out of a scoped search
Participants were unable to return to search the homepage after searching within a sub collection.
Search shortcuts (aka boolean) are buried
Instructions for search shortcuts is only displayed when a search yields zero results. This information should be presented upfront to the user so they are aware of the functionality before hitting a dead end.
Delayed return of search results appears as a dead end for screenreader users
“Visually impaired people complain to us a lot. The screen reader says the page is done if they wait long enough, suddenly [results] are there. No visibility into the loading bar.”
Ideation workshop to brainstorm search features.
- Goal: Generate a breadth of search capabilities based on user needs.
- 11 Designers from 6 different squads
- 50+ concepts for search capabilties
2-hour Agenda
Review summary of user feedback and domain research
Identify user needs
Brainstorm and sketch big ideas
Kano study to aid in feature prioritization.
Kano is one of many prioritization methodologies. It was appropriate to run at this time because we had already completed discovery research and it was time to make roadmap committments. This study collected user sentiment on a revised feature list from the ideation workshop to aid the team's prioritization.
- 26 features
- Survey live for 2.5 weeks
- 60 survey responses (26 customers, 34 internal users)
- Wrote specific and granular features to clearly interpret what users liked or disliked
- Piloted survey to test clarity of feature descriptions
- Randomized order and used consistent wireframe fidelity to limit variables influencing participant ratings
- Added an optional comment field for each feature so participants could provide explanations
- Included firmographic questions to allow for segmentation
Plan: Report results in our workshop to socialize findings and democratize interpretation of survey responses.
Workshop to socialize research findings and align on a feature roadmap.
- 24 workshop attendees from 7 different squads
- Hybrid in-person/remote
- 19 features committed to a 3-quarter project roadmap.
Day 1: Share and discuss findings
Competitor Review readout
Kano readout and discussion
Day 2: Align around action items
Warm-up + research reflection
Impact / feasibility grid activity
Dependency discussion
Roadmap prioritization
Brainstorm KPIs
Workshop feedback
The research data that you shared helped me with context about the space and perceived problems. In addition, the qualitative data (the comments) helped put the average quantitative data in perspective. As we went through the small group exercises, we frequently reviewed the research that you provided to help us recall the survey results.
Head of Content Strategy, Watson Innovations
Kicking off the workshop with research was a great way to establish a firm grounding in the user feedback. The deck was referenced and used throughout the workshop to great success. The juxtaposition of quantitative and qualitative data was balanced and helpful when weighing the feedback as the group prioritized … Can't wait to see more of these Kano studies helping us prioritize our features! Would love for you to share this research method with the research team and larger design team.
Cloud Platform Design Manager
Having research at the forefront of the workshop provided not only valuable insights to our engineering and content development stakeholders but the data provided a reality check on our internal opinions about end users. Overall, a big success for our team to align on a roadmap where everyone's voice was represented.
Cloud Platform UX Designer
Applying the RITE method to take an agile approach to flagging and addressing usability issues.
RITE stands for Rapid Iterative Testing and Evaluation. It's a partnership between Design and Research where usability improvements are made as soon as they are raised by research participants.
- Mid-fi wireframes to implement changes quickly
- 4 iterations in 2 days to resolve 13 issues
- Moving search operators below the search bar from beside improves discoverability.
- Search results must have the doc’s title, a few lines of text, and date of last update.
- Participant's prefer to filter by product or service, just as in the IBM Cloud Catalog.
How the filter evolved in testing inspired our patented internal tagging system.
A new Docs Search is released!
Patent
A patent for the pattern of how you apply and remove tags to perform a search scoped to a doc subcollection.
Behavioral analytics to measure results and identify new issues.
An Amplitude Dashboard with 27 charts.
- Compared in-app searches before/after release
- Monitored total page views before/after release
- Built funnels to evaluate click through rates
- Tracked interactions around the new filter
An increase in the use of search operators
Significant drop-off in traffic to FAQ page
Metrics gathered:
2x in-app searches (fewer users leaving Docs)
Significant increase to IBM Cloud Docs search click through rates
Nearly 50% user engagement in the new filter after getting their initial search results
Improved SEO with Google now being the #1 referrer to IBM Cloud Docs
Reflection
Project takeaways
Why I'm sharing this work
Introduced the Kano method to the org who then requested I lead org-wide studies for quarterly roadmap prioritization.
What worked well
Aligning on a research roadmap to move quickly
Socializing the interpretation of Kano results
Segmenting IBMers and customers to demonstrate the importance of researching with end-users
Using a DT workshop to align on a roadmap
What I would do differently
Run a feature prioritization study with a larger sample for the 2nd and 3rd releases
Include statistical significance in the Kano presentation slides
Better documentation of design iterations during RITE testing
Thanks for reviewing a case study!
I'm happy to speak in more detail on this or other project work. Please reach out so we can continue the conversation.
Download Resume Connect on LinkedIn