Case Study
How user research influenced goal setting, feature prioritization, and design iteration to ship a measurably better search/SEO experience for IBM Cloud Docs.
Challenge
IBM Cloud Docs users were frustrated:
- Too many irrelevant search results
- No way to refine or filter
- Users abandoned IBM Docs to search on Google
Search behavior crossed 7 product squads, but there was no shared strategy to improve it.
Actions
In 6 weeks, I led research to align 7 product squads on a shared strategy that measurably improved an under performing search experience.
- Synthesized feedback, analytics, and SEO data to anchor teams in evidence.
- Conducted user interviews to validate behaviors and clarify unmet needs.
- Prioritized 26 potential features using the Kano method and facilitated decision-making based on findings.
- Used RITE testing to rapidly iterate on the new filter design — which evolved into a patented tagging system.
- Behavioral Analytics to monitor KPIs.
Results
- Aligned all squads on a 3-quarter roadmap of prioritized search improvements
- 2x in-app searches (fewer users leaving Docs)
- Nearly 50% user engagement with new filter features
- Significant increase to IBM Cloud Docs search click-through rates
- Awarded a patent for new filtering capability
After delivering strong results on search, I was asked to take on a broader challenge: Align multiple IBM organizations to deliver a unified content experience— from marketing pages through documentation — something that had been tried, but never achieved.
Process
Details on the actions that led to results:
Research rollup of existing user feedback to identify issues with search.
- 12 in-app user feedback comments (Usabilla)
- 9 NPS responses (Medallia)
- 4 customer issues shared by CSMs (Slack)
Too many search results
“Searched the documentation for "kubernetes pricing". There are 792 (!!!) search results."
Unable to refine searches
“I'm looking for something specific, but the more terms I enter in the search box trying to narrow down the results, the more results I actually get."
Search results do not display enough information
“It’s incredibly difficult to scan search results for relevant content. Titles and a broken sentence or two really aren't enough.”
SME interviews & domain research to understand the upcoming changes in search capability and performance.
- Marketing’s Program Director of Digital Strategy to inquire about the search and SEO data driving our strategy (e.g. search inputs, SEO performance, page visits).
- Senior Software Engineer to understand the search implementation and how content attributes (e.g. titles, headers, and keywords) would and could impact search results.
Behavioral analytics related to project KPIs:
- % (most) of B2B journeys begin with an online search.
- 2x greater engagement rate from organic traffic vs paid media.
- % (far too many) of pages get no organic Google search traffic because we create too many pages.
Secondary research on search & SEO best practices.
- 8 external articles
- 2 prior studies by IBM Researchers
Review competitors' docs search.
- AWS
- Google Cloud
- MS Azure
User interviews to learn more about existing pain points and unmet search needs.
- 30-minute remote interviews with 9 participants from US, EU, and Asia
- Interviewees walked through a recent search experience
Gained a more contextual understanding of user behaviors observed in our analytics:
- 100% of participants ‘Google it’ before searching our docs, validating the need to improve SEO (a project KPI).
- The most common search behavior is to ‘drill-down’ using keywords, beginning with a product name.
- Participants complained of too much documentation and that results return the same document multiple times. They workaround this by navigating to a subcollection, then performing a scoped search.
- Low page views of translated content is more a matter of trust than preferred language. Users found English docs to be more up to date.
Gap in our content strategy taxonomy: API documentation
If APIs are available, they are often buried within a subcollection. Participants were in favor of an API hub.
Users unable to navigate in and out of a scoped search
Participants were unable to return to search the homepage after searching within a subcollection.
Search shortcuts (aka boolean) are buried
Instructions for search shortcuts is only displayed when a search yields zero results. This information should be presented upfront to the user so they are aware of the functionality before hitting a dead end.
Delayed return of search results appears as a dead end for screenreader users
“Visually impaired people complain to us a lot. The screen reader says the page is done if they wait long enough, suddenly [results] are there. No visibility into the loading bar.”
Ideation workshop to brainstorm search features based on user needs.
- 11 Designers from 6 different squads
- Outcome: 50+ concepts for search capabilities
2-hour Agenda
Review summary of user feedback and domain research
Identify user needs
Brainstorm and sketch big ideas (goal=breadth)
Kano study to aid in feature prioritization.
This study collected user sentiment on 26 features from the ideation workshop. Results would be socialized in upcoming workshop to prioritize features and align on a roadmap.
- Wrote specific and granular features to clearly interpret what users liked or disliked
- Piloted survey to test clarity of feature descriptions
- Randomized order and used consistent wireframe fidelity to limit variables influencing participants' ratings
- Added an optional comment field for each feature so participants could provide explanations
- Included firmographic questions to allow for segmentation
- Gathered 60 survey responses (26 customers, 34 internal users)
Workshop to socialize research findings and align on a feature roadmap.
- 24 workshop attendees from 7 different squads (in-person and remote)
- Outcome: 19 features committed to a 3-quarter project roadmap.
Day 1: Share and discuss findings
Competitor Review readout
Kano readout and discussion
Day 2: Align around action items
Warm-up + research reflection
Impact / feasibility grid activity
Dependency discussion
Roadmap prioritization
Brainstorm KPIs
Workshop feedback
The research data that you shared helped me with context about the space and perceived problems. In addition, the qualitative data (the comments) helped put the average quantitative data in perspective. As we went through the small group exercises, we frequently reviewed the research that you provided to help us recall the survey results.
Head of Content Strategy, Watson Innovations
Kicking off the workshop with research was a great way to establish a firm grounding in the user feedback. The deck was referenced and used throughout the workshop to great success. The juxtaposition of quantitative and qualitative data was balanced and helpful when weighing the feedback as the group prioritized … Can't wait to see more of these Kano studies helping us prioritize our features! Would love for you to share this research method with the research team and larger design team.
Cloud Platform Design Manager
Having research at the forefront of the workshop provided not only valuable insights to our engineering and content development stakeholders but the data provided a reality check on our internal opinions about end users. Overall, a big success for our team to align on a roadmap where everyone's voice was represented.
Cloud Platform UX Designer
Applying the RITE method to take an agile approach to flagging and addressing usability issues.
Rapid Iterative Testing and Evaluation allows for usability improvements to be made as soon as they are raised by research participants and validated in the following user sessions.
- Mid-fi wireframes to implement changes quickly
- 8 iterations in 3 days to resolve 13 issues
- Moving information on search shortcuts below the search bar improved discoverability.
- Search results should display doc’s title, a few lines of text, and date of last update.
- Participants preferred to filter by product or service, similar to the IBM Cloud Catalog.
Test iterations on the search filter inspired our patented internal tagging system.
A new Docs Search is released!
Patent
A patent for the pattern of how you apply and remove tags to perform a search scoped to a doc subcollection.
Behavioral analytics to measure results and identify new issues.
An Amplitude Dashboard with 27 charts.
- Compared in-app searches before/after release
- Monitored total page views before/after release
- Built funnels to evaluate click-through rates
- Tracked interactions around the new filter
An increase in the use of search operators
Significant drop-off in traffic to FAQ page
Metrics gathered:
2x in-app searches (fewer users leaving Docs)
Significant increase to IBM Cloud Docs search click-through rates
Nearly 50% user engagement with new filter features
Improved SEO with Google now being the #1 referrer to IBM Cloud Docs
Reflection
Project takeaways
Why I'm sharing this work
Introduced the Kano method to the organization who then requested I lead org-wide studies for quarterly roadmap prioritization.
What worked well
Aligning on a research roadmap to move quickly
Socializing the interpretation of Kano results
Segmenting IBMers and customers to demonstrate the importance of researching with end-users
Using a DT workshop to align on a roadmap
What I would do differently
Run a feature prioritization study with a larger sample for the 2nd and 3rd releases
Include statistical significance in the Kano presentation slides
Better documentation of design iterations during RITE testing
Thanks for reviewing a case study!
I'm happy to speak in more detail on this or other project work. Please reach out so we can continue the conversation.
Download Resume Connect on LinkedIn