It started with an idea, then it became an initiative
When ChatGPT came out, I identified an opportunity early on how we could leverage LLMs to improve common pain points in our product. I sold the idea to senior leadership by visualizing for them the possibilities. The idea quickly escalated to a tiger team, which escalated to a real product team, which escalated to a company initiative.
Honestly, watching my idea take flight was one of the proudest moments in my career.
Over the course of about a year, we shipped the following 0 to 1 AI features: Segments AI, Support AI, Flows AI, Email AI, Reviews AI and more yet to come.
Below is how we took Segments AI from 1 to 2.
What is Segments AI?
Segments AI is a generative AI tool that allows users of Klaviyo (a marketing automation software) to create an audience segment simply by describing it using their own words. Also, it’s patented ;).
Listen to learn more about Segments AI:
The first iteration and what we learned
What worked
Sparkle entry point was a recognizable pattern and minimally intrusive to the current UX
Offering tool on a separate surface afforded us a beta surface to learn and refine our model
What didn’t work
Engagement with the tool was low at <5%
Despite improvements to our model, engagement remained flat
Some segments are easier to build with AI, while others are easier to build with the classic tooling
Our hypotheses
Friction between the AI tool and classic tool causes burden of choice and deters engagement
Leading with AI branding deters engagement from users who are weary of AI
Combining the AI and classic experience will enable user to create their segments more successfully
Some of our explorations
The experiment
There’s only so much that traditional user research can reveal when it comes to AI. Users need to see and feel the model. While we did show a couple users some concepts, we wanted to jump right into experimentation. To select a variation to test, we used our gut. We built fast and released the test to 1/3 of eligible accounts for 4 full weeks. We wanted to see if the new treatment improved engagement metrics without harming the overall segment creation experience.
Control: sparkle entry point
Variation: embedded/combined entry point
The results
Quantitative
Segments AI adoption increased
(4.3% to 26.8%)
Segments AI retention increased
(42.3% to 49.1%)
Qualitative
It was not immediately clear to users that you have to click the text entry box to expose traditional segmentation selectors. About half did not read the helper text explaining this.
Time to creation remained flat
(1.7m to 1.9m)
Total segments created remained flat
(5.1a to 5.2a)