Getting a Pulse on Production

We'd been running SUS surveys in our alpha program for almost two years. We trusted the signal. The question became whether we could extend that same rigor to production customers and what we'd learn when we did.

Role: Product Design Manager, iRobot

Team: Product, Design & Data Analytics

Skills: UX research, data analysis, cross-functional collaboration

---

CONTEXT

Alpha testing and usability scores gave us something real: a consistent cadence, a committed group of participants, and almost two years of trend data we could actually act on. When scores moved, we understood why. When we shipped something new, we had confidence from real users that we were moving in the right direction.

What alpha couldn't give us was visibility into production customers, people who had never opted into a feedback program and were simply trying to use the app. For that audience, our usability picture was app store ratings and Amazon reviews. Useful signals, but not the same thing as structured usability measurement.

That was the gap. Not that alpha was failing us, it wasn't, but that an entirely different population existed outside of it, and we had no equivalent window into their experience.

GETTING THE TRIGGER RIGHT

About six months ahead of a major app launch, we added a SUS survey directly into the live product to establish a baseline with real customers.

Rather than surveying users immediately after setup, we waited until they completed three cleaning jobs within their first month. That gave customers enough experience to form a real opinion, but not so much time that early friction had already been forgotten.

Responses fed directly into our analytics pipeline and surfaced in a live Mode dashboard alongside adoption, engagement, and crash data. That last part mattered. Usability wasn't living in a spreadsheet I maintained anymore. Now it refreshed every few minutes alongside adoption and reliability metrics.

Moving SUS to production also allows the alpha program to focus on what it does best. Rather than using it to keep a continuous metric alive, we could focus that program on controlled testing and specific research questions - the type of work that benefits from a smaller, known population.

THE SCORE WASN'T THE WHOLE STORY

Going in, I expected the headline to be the score itself. We'd built toward a number, and now we had one at scale.

When I looked at the distribution what stood out most was that users weren't clustering around the average the way I expected. There was a group rating the experience very highly, and a separate group rating it very poorly, with not much in between. The average was technically accurate and almost completely misleading.

That realization changed the questions we were asking. "How do we move the score up?" is a reasonable question when you're chasing a benchmark. "Why are two completely different experiences living inside the same product?" is a much more interesting one and it's the kind of question that led us to dig into the responses.

From there we could start cutting the data in ways that actually explained the split. Were certain robot platforms driving lower scores? Were new customers struggling more than long-time owners? Were specific releases introducing friction we weren't catching elsewhere? Because the data was continuous, we could chase those threads while the product was still evolving rather than waiting for the next alpha cycle. This also led us to add an open text response field so users could provide context behind their scores.

Over time, the dashboard became just as valuable as the score itself. It allowed us to slice the data by operating system, robot platform, release version, and other factors, turning SUS from a benchmark into a diagnostic tool.

A PULSE ON PRODUCTION

For the first time, we had a steady read on how the product felt to real customers. Now we had more than a snapshot from a survey cycle or a proxy signal from app store reviews. Instead, we had an ongoing measure that was simply always there.

This changed how the team related to usability data. When a release went out, we didn't have to dig through reviews hoping someone mentioned something useful. The pipeline was already running. If something shifted, we'd see it in real time.

Production SUS didn't replace what we'd built in alpha. It extended the practice somewhere new. Together they gave us something better than either could alone, a controlled environment for targeted research, and a continuous signal from the people actually using the product every day.

The most valuable outcome wasn't a number. It was knowing that usability stayed visible after release, not just before it.