How a simulator took program comprehension from 29% to 93%, and why the business almost didn't ship it.
The bank was building a genuinely valuable program. The problem was that users never quite understood it. They didn't know how to earn points, what their level unlocked, or that they could simulate their way to the next one. Internally, the challenge was different: the business and tech teams wanted to launch the first release without the simulator and ship it later due to technical complexity. We needed evidence to argue otherwise.
We ran two rounds of research with 35 participants in total. Interviews were conducted remotely, transcribed automatically, and analyzed with AI to surface patterns across sessions. The first round tested the full platform without the simulator. The finding was clear: only 29% of users connected their financial products to the points they generated. To figure out if they could reach the next level, they had to do the math manually using an equivalency table buried in the help section. The program made sense in theory, but not in practice.
That was the evidence we needed. With the simulator included in a second round, results changed: 93% of users navigated the flow successfully and for the first time understood how their financial behavior connected to their level. A third round of 10 sessions tested the simulator in detail, confirming the flow worked but surfacing three copy failures each affecting roughly a third of users.
The simulator
The simulator was not an optional feature. The argument for cutting it from the first release was technical complexity. Our argument for keeping it was the evidence: without it, the program wasn't understandable. Each product had its own rules: credit card spending, mortgage loans, consumer loans, investments, insurance, and auto-payments. We met with each product owner to understand the cases, with the tech team to validate feasibility at each iteration, with legal to make sure every equivalency and disclaimer was accurate, and with marketing to align the language with the program's promise. We absorbed all of that so users could see a single clean list. We used AI tools to rapidly prototype different approaches to the simulator's complexity before committing to a direction.
From equivalencies to answers
The problem wasn't that users didn't understand the numbers. It was that the numbers had no context. Seeing "+200 points per $1,000 spent" means nothing if you don't know how many points you need, how many you already have, or how far you are. The simulator solved that: instead of a table that required mental math, users entered their real amounts and got a direct answer: with these products, you'll reach level Prestige by end of year.
In the first round of research, users consistently assumed points were a balance to redeem. The mental model was broken before they even started. The fix was not just copy. We had to make the purpose of points visible at every step: they are progress toward a level, not a currency.
The second round revealed three specific failures, all in the language. A unit abbreviation was misread by 30% of users. A button mid-flow confused 30% because it felt like starting over rather than seeing results. And the investment minimum for earning points was only understood by 30% of users. Each fix was a single line of copy. Each one moved comprehension from 30% to near-perfect.
In the months after launch, more than a third of all program users engaged with the simulator. Of those, 86% simulated at least one product. Credit card and auto-pay were the most explored, reflecting the most common products in the user base. Thousands of users run simulations every day and come away with a clearer understanding of the program.
Understanding comes from using, not from reading.
The onboarding was never going to do the job on its own. Comprehension came from interacting with the simulator, not from explanatory text. I'd push earlier to make the simulator a primary entry point, not a secondary feature buried in the program section.
The moment after simulation is the shortest and most valuable.
Users who simulate are ready to act. We invested in the path to simulation but underinvested in what happens immediately after. Next time, that post-simulation moment gets the same design rigor as the flow itself.
Evidence beats argument, but only if you have it ready.
The simulator survived because we had data showing 29% comprehension without it. Without that number, the technical complexity argument would have won. Research isn't just for discovery. It's the currency that protects design decisions later.
