Success
The player logged in.
LOTTOLAND · AUTHENTICATION · UK
When the data said login was broken, I didn't trust the number.
We'd started rolling out a new passwordless login experience to UK players.
And according to our early FullStory data, it wasn't going particularly well.
Early reporting showed just 15.6% of players completing the measured login funnel. If that number was right, we had a pretty serious problem.
The thing was, I wasn't convinced it was.
Datadog was telling us something different. We weren't seeing a big increase in login-related customer contacts. And when we started watching actual player sessions, what FullStory called a failed login didn't always look like a failed login.
Before we started trying to improve the number, we needed to work out whether we could trust it.
10% → 60%Traffic rollout
15.6% → 64.2%Measured eligible login conversion
Product LeadWorking across Analytics + Frontend + Backend
01 · The number that didn't add up
15.6% looked bad.
But it didn't make sense.
Our initial FullStory funnel suggested a major problem between players entering their identity, receiving an OTP and successfully logging in.
The obvious response would have been to start optimising the journey. But other signals weren't telling us the same story.
Customer contacts hadn't increased at anything like the level we'd expect. Datadog and FullStory weren't aligned. And individual player sessions didn't always match the outcome being reported.
So rather than immediately trying to improve conversion, we started by questioning the measurement.
15.61%Measured conversion
Identity Entry3.5K
↓OTP Displayed2.6K
↓Login Success544
43.41sMedian time to convert
“The number looked bad. More importantly, we couldn't trust what it was telling us.”
02 · My role
Product Lead
I was the Product Lead for the new authentication experience and led the investigation alongside our Analytics Manager and frontend and backend teams.
My job wasn't to personally diagnose every event or technical issue. It was to bring the different signals together, make sure we were asking the right questions and help the team decide what we should actually fix.
Ultimately, we needed enough confidence in both the product and the data to decide whether we should keep increasing traffic.
03 · Reframing the problem
Login success turned out to be a lot less simple than I thought.
We'd been looking at login as a fairly straightforward funnel. Player enters email → receives OTP → logs in. Success or failure.
Then we started looking at actual sessions.
One player entered the wrong OTP twice before eventually logging in. Another was correctly prevented from logging in because of their account status. Another requested an OTP, went to their inbox and never came back. And in some cases, players had successfully logged in but our success event hadn't fired correctly.
All of those journeys were telling us something different. But our headline login metric wasn't.
Friction
Something genuinely got in the player's way.
Expected outcome
The product correctly prevented login.
Abandonment
The player started the journey but didn't finish it.
“That sounds obvious now. It wasn't obvious from the dashboard we started with.”
04 · How we investigated it
We followed the inconsistencies.
There wasn't one dashboard that could tell us what was happening.
FullStory
Funnels, sessions, player behaviour and abandonment.
Datadog
Authentication events and technical behaviour.
Voice of customer
Were players actually reporting login problems at the scale suggested by the data?
User testing
Reproducing journeys across mobile platforms.
Engineering
Instrumentation, sessions, navigation and OTP generation.
“The useful bit was when these sources didn't agree.”
Instead of assuming one was right, we started asking why they were different.
That's where we found most of the interesting stuff.
05 · What we found
Two problems were hiding inside one number.
Some of the problem was our data.
- Successful logins weren't always being captured.
- Some OTP events were duplicated.
- Some errors and valid outcomes were being counted as failures.
- FullStory and Datadog weren't always answering the same question.
“The original 15.6% conversion rate wasn't a number I was comfortable using to make product decisions.”
And some of it was the product.
- Invalid OTPs accounted for around 13% of journeys at one stage.
- Players could request another OTP while the previous one was still relevant.
- Back / forward navigation could inadvertently trigger another code.
- Players could request another code after only 30 seconds.
- OTP expiry wasn't communicated clearly enough.
“None of those things individually screamed ‘login is broken’. Together, they created unnecessary friction.”
06 · What we changed
We didn't redesign login.
We fixed what the evidence pointed to.
30 → 45 seconds
Give players more time before another OTP can be requested.
Reuse valid OTP
Avoid generating another code unnecessarily while an existing OTP remains valid.
Clearer email
Make OTP expiry clearer in the authentication email.
Close journey loopholes
Address navigation and reload behaviour capable of generating additional codes.
Better instrumentation
Fix the events and classification needed to properly understand the impact of the changes.
There are still bigger ideas we're exploring — including SMS authentication, smarter channel selection and reducing the OTP from six digits to four. But I didn't want us jumping to bigger solutions until we properly understood the problem we already had.
07 · Before and after evidence
So, did login improve?
Yes. But this is where I think it's important not to oversell the numbers.
15.61%Measured conversion
Identity Entry3.5K
↓OTP Displayed2.6K
↓Login Success544
43.41sMedian time to convert
“The number looked bad. More importantly, we couldn't trust what it was telling us.”
64.23%Eligible login conversion
Identity Entry3K
↓OTP Displayed2.5K
↓Login Success1.9K
17.84sMedian time to convert
“Better instrumentation and eligibility rules gave us a much clearer view of genuine login behaviour.”
I can't credibly claim
“We increased login conversion from 15.6% to 64.2%.”We didn't.
That movement came from a combination of fixing incorrect instrumentation, improving how we classified different login outcomes and making genuine improvements to the experience.
For me, that's actually the more interesting story. We went from having a number we couldn't explain to having a much clearer understanding of what was happening to players.
08 · The outcome that mattered
10% → 60%Traffic rolloutWe started with 10% of UK traffic going through the new experience.
As we fixed issues, improved the measurement and became more confident in what players were experiencing, we progressively increased that to 60%.
For me, that's the stronger outcome.
We didn't increase traffic because a dashboard went green. We increased it because we understood what was happening well enough to be comfortable putting more players through the experience.
09 · What came next
And we're not finished.
Once the data became clearer, something else stood out. Abandonment.
Some players weren't failing. They weren't hitting an error. They simply requested an OTP and didn't come back.
That's now a much more interesting problem.
Was the email taking too long?Were players getting distracted?Would SMS work better?Would a shorter OTP make a difference?
What I took from it
I started this investigation thinking we had a login conversion problem.
We did. Just not the one the original dashboard suggested.
There were data problems, product problems and completely legitimate player behaviour all being bundled into the same metric.
Separating those things changed where we spent our time and gave us the confidence to keep rolling the experience out.
“Don't improve a metric until you understand what it's actually measuring.”
Next case study · 03Native had to be better. Not just more expensive.