Case Study - Redesigning Search

I spend a lot of time coming on as specialist to support design agencies working on product development involving algorithms including machine learning, natural language processing (NLP), and generative AI (GenAI). The product interfaces range from a graphical user interface to voice-only, but always involve conversation based interactions with an end user.  In simple terms, we’re building a search oriented logic machine that can talk.

My secret sauce to doing this well is to consider my priorities in two lanes: driving business outcomes for our clients as well as studying the underlying product to identify new uses and features to seed future innovation. I became a designer in the world of management consulting and I’m a former athlete, so I relish the hunt for new ideas and supporting evidence that lead to smashing our goals and exceeding expectations.

Situation:
A regional utility software company had complaints from their customers that while the core product was valuable, customers gave the experience poor scores in their net promoter score (NPS) surveys. The application needed a facelift but at the same time, the company determined it needed to start thinking about how to include AI in order to compete. I was brought on by the agency with the project contract as the domain expert for AI design. The project plan was blurry and the roadmap needed to be rethought. I was hired to touch all of these pieces as both a designer and an educator from strategy to launch in a 10 month period.

Task:
I had the full trust of both teams from the very beginning which made a huge difference in the way this project proceeded. The interface was full of information but lacked focus and felt much too complicated for most users. The team accepted my recommendation to explore adding an agentic experience and we got to work.

Actions:
The first step of any project like this requires discovery and analysis of the language data. This allows me to understand the scope of the brain we’re building and the level of difficulty. I partnered with a small team of engineers and we quickly found that our best source of language came from the search experience and customer service calls. It didn’t take long to discover a high rate of abandonment for search queries over 5 words, immediately validating that we had a big opportunity to convert pain into success.

Next, I setup a research sprint with 20 customer interviews and some card sorting/tree exercises to help the team train to observe and analyze natural language. This allowed the team to see how the current experience failed to elicit user prompts in the search experience that mapped to their intended issue or solution. It also helped me to start grouping and ranking the problems that customers were facing. The customer service team supported this process and together we started the process of building our intent mappings so that our large language model brain could handle a variety of expressions of the intent, underlying semantic meanings, all sorts of understanding that goes well beyond simple keyword matching.

The customer service phone calls also helped us to understand and prioritize the customer journey. We identified highly trafficked problem pathways and boom! Now we knew how to make sure our product would add value— solve these top 6 problems, representing over 60% of the customer service inquiries, and we could make both customers and clients very happy. Armed with problem definitions, intent maps, and language benchmarking, we were ready to start designing the new experience.

Solid research makes solution design go smoothly. I love when that happens! We started with defining key language signposts (e.g., the intro, the close, error pathways), converting real customer search inquiries into wireframes, and live role playing our ideas. This workshop helped the team understand what it’s like to design for conversational interactions. We took care to understand the relationship between dialogue, the interface, and customer satisfaction. Even if you can articulate your issue, the new service agent needs the power to solve the problems completely.

I created the wireframes for the first 6 problem areas. The client contracted with a solid conversational AI product to integrate with and I was able to quickly configure the components and pathways myself. Once we got sign off on the interface, we moved on to training our model using the defined pathways and intent maps. I used the Wizard of Oz method to help me tune up the natural language. As we readied for the pilot, I embedded with the engineering lead to scope the integration pilot. We intentionally kept the AI features opt-in initially, focusing the first phase on generating search refinement prompts instead of full chat to de-risk the launch and preserve the technical budget for later phases. We launched the new approach as a two-week A/B test observing users from an existing pilot group of 665 users. We continuously monitored the quantitative data and used qualitative feedback to make daily iteration adjustments, ultimately achieving statistical significance for the lift in engagement metrics before scaling the rollout.

Results:
The AI integration was a major success, significantly improving customer experience ratings and notable cost reduction in customer service labor (e.g. call center staffing). By starting with our 6 problem areas, which represented 60% of customer service inquiries, increased overall self-service completion from 23% to to 72%. Our goal was to reach 50% so the client and prime contracting agency were very pleased. Within a year, the software attracted a buyout which the client was very happy with.