Professor Game
, , , ,
0

Your Login Numbers Can’t Tell If the Bar Is Right | Episode 465

  Listen on SpotifyListen on Apple PodcastsAvailable on YouTube Get the free Core Drives in the Wild guide, behavioral design applied to real products: professorgame.com/WildCD

Episode Summary

Rob breaks down adoption target difficulty, showing why login counts cannot tell you whether a goal is set right. He covers a 2026 Journal of Marketing Research study of the Australian micro-investing app Raiz Invest and a randomized test in a popular free-to-play puzzle game, drawing on both to explain why missed targets are associated with lower retention a year later. Listeners learn to measure the valuable action instead of the login and how to re-tune difficulty for each person.

About the Host

Rob Alvarez is Head of Engagement Strategy, Europe at The Octalysis Group (TOG), a leading gamification and behavioral design consultancy. A globally recognized gamification strategist and TEDx speaker, he founded and hosts Professor Game, the #1 gamification podcast, and has interviewed hundreds of global experts. He designs evidence-based engagement systems that drive motivation, loyalty, and results, and teaches LEGO® SERIOUS PLAY® and gamification at top institutions including IE Business School, EFMD, and EBS University across Europe, the Americas, and Asia.

Key Takeaways

  • A 2026 Journal of Marketing Research study of the investing app Raiz found a goal feature helped most users but left a substantial segment no better or worse off, a split linked to how hard each goal was for the person.
  • In Raiz’s email test, five message versions went to 10,000 customers each; among the 329 who adopted within two weeks, suggesting a recurring amount beat plain goal setting, while suggesting only an end goal did not.
  • Login counts did not differ across any Raiz test group, so a login-based metric would have missed what people were actually doing in the app, which is the valuable action a business needs to track.
  • In a randomized test of about 330,000 puzzle-game players, easier boards for players who had been playing less lifted win rates about 14 points and added about 10 rounds over 30 days; they used fewer boosters per round, yet total spending rose, against the company’s expectation.
  • Missed targets were associated with lower retention a year later, a horizon that a seven-week test or a sprint review would never catch, which is why Rob recommends tracking the miss rate by segment and by period.
  • Core Drive 3 (Empowerment of Creativity & Feedback) acts as the feedback loop that keeps Core Drive 2 (Development & Accomplishment) meaningful when difficulty is re-tuned for each person.

Topics Covered

Mentioned in This Episode

Both studies below are published research, not client work. Scope of the evidence: the investing app findings come from 2018 to 2019 launch data plus a 2023 email test, and the paper links missed targets to lower retention a year later as an association. It found no clear link between goal difficulty and login frequency. The puzzle game test covered only low-activity players, and total spending rose even though boosters and coins used per round fell. Treat the “every metric improves” line in the episode as a spoken summary of these results, not a finding of either paper.

Free Resources and Get in Touch

Looking forward to reading or hearing from you, Rob Full episode transcription (AI Generated)

Who set the bar on your last adoption target?

Rob Alvarez (00:00): Let me ask you a question. Who set the bar on your last adoption target? A 2026 study had the same goal feature that helped most people and yet left a substantial segment, no better off or worse. What separated them was linked to how hard the goal was for them. Why am I talking about this? I’m Rob, I’m head of engagement strategy from the Octalysis Group, the top behavioral design and gamification consultancy in the world. Rob Alvarez (00:28): And I’m also the host of the number one gamification and behavioral design podcast, Professor Game. So, what was the problem with the bar?

Three ways a bar goes wrong

Rob Alvarez (00:35): There’s three ways it can go wrong. Of course, too low, you know, completion looks fine. The valuable action that you’re looking for, what we call in Octalysis terms, the desired action, stays flat. A typical customer chose Well below the best level. and your number of logins show absolutely no link to how difficult the task is. Failure mode to misses climb and people start leaving. Misses were associated with lower retention a year later. Rob Alvarez (01:07): Failure mode three, same for everyone. Day one user versus your veteran, for example, two-year user. It is really hard to compare the level of engagement, the capacity, the capability that a user day one has built versus somebody who’s been doing the thing for over two years. And if this is something that interests you, to see how these things can go, how they can go wrong, how they can be done. Rob Alvarez (01:37): A lot better, I have a free guide called Core Drives in the Wild. All you have to do is click on the link in description, you’ll get access to it immediately through your email and you’ll get a better read on those right away.

The Raiz email test: recurring amounts beat plain goal setting

Rob Alvarez (01:50): So let’s start with the study. There was this person very recently, Raiz and some other authors, who did an email test. They sent five versions Three hundred and twenty nine adopters who adopted within barely two weeks. Suggesting them a recurring amount produced bigger gains in investing than plain goal setting. Suggesting only an end goal. Ultimate goal that they had did not logins did not differ in any of those groups.

Why login numbers miss real behavior

Rob Alvarez (02:20): So if you only look at login metrics, you’re completely missing the mark of what people are actually doing within the app. Course, you can always say, Well, this is a small self-selected group, and the test was about seven weeks, I think. And the authors, of course, present it as additional evidence to other things that have been done. However, this is a clear situation where you start understanding where a you are not just designing for getting people onboarded. Rob Alvarez (02:49): A business metric can definitely not just be login. Because that is not what you’re really being measured for within your company. You’re not really interested in how many people log in. You’re interested in what people who log in, and yes, that could be a number, what they’re actually doing within the app. That is what actually interests you. Because even more in a financial investment, this is something you need to take care of.

The puzzle game test: easier boards, higher spending

Rob Alvarez (03:14): And there’s even another test. Of course, this is in a game. It’s a popular free-to-play game puzzle. it was randomized. About 330,000 players all had passed level 20 and played fewer than 20 rounds the previous week. They treated the treated players got easier boards with more easing the less they had played. They won. Win rate rose about 14 points. Over 30 days, about 10 more rounds. played. Spending: players used fewer boosters and coins per round, but they played more and stayed. Rob Alvarez (03:50): So total spending actually rose. The company had expected the exact opposite. Both teams tested it instead of actually assuming. And they not only tested, they tested for the right thing. They were looking at the real numbers that they were interested in. Look at that. One of them is for a an app, right? Very serious business. The other one is for a game. They were both. testing for the right thing to make sure that the changes that they were looking towards implementing were geared towards achieving the right results.

Five moves for setting the bar

Rob Alvarez (04:24): So five moves for you right away. Measure the valuable action. I already mentioned this. It’s not the login. It’s not how many how many people are active in a single day. It’s really meaningfully looking at what would move the needle of your business forward. Second thing to look out for. You want to size the difficulty, not to a company-wide metric that you have. Now, with personalization, all we talk about personalization. Rob Alvarez (04:53): This is one of the few instances where making sure you understand where they’re at, what’s their baseline, so you can recommend clearer and better goals becomes increasingly important. If you treat a day one user like you would treat a veteran, you’re losing on both ends. You want to track the miss rate by segment and by period. Missed targets were actually associated with lower retention a year later. Rob Alvarez (05:22): A full year later, not when you’re measuring in the seven weeks, in the month, and your next sprint. A year later. The tracking method here is something that you want to make sure is put in the right place. And short-termism, remember, we’ve talked about this in another episode. When you go only for short-termism and you look at an only data-focused decision, you’re always going to prioritize Black Hat mechanics that turn into metrics that function in the short term, but eventually lead to user churn, which is the opposite of what you want. Rob Alvarez (05:58): You want a sustainable business that is able to bring people back day by day, by week, by month, and by year. Another lesson that we can take from here is how to retune per person, and maybe not even every single day, but per a schedule. The game that we mentioned rechecked each player’s last week of play every day. Full week. Looked at the full week. What’s the behavior in the week? Rob Alvarez (06:22): The authors of that study expect that overeasing could backfire for players who want more challenge. Which, of course, they did not test in this case. But Core Drive 3 is the feedback loop that keeps Core Drive 2 meaningful. If they are able to, through their own methods, getting that feedback loop in, whatever it is that they’re trying to get better and achieve that Core Drive 2 motivation, you want to have that very clearly addressed towards that specific type of user.

Would the person thank you if they could see the dial?

Rob Alvarez (06:57): And finally, it wouldn’t be me if I don’t mention the ethical side of all this. And when you do this, I do want to make sure you think about it from the perspective of the user. Do you think that the person who is using the app, the game, whatever you’re building, would thank you when they see how you dial up and down their difficulty level? The company was expecting a harder game to drive spending and the paper, you know, with their data. They had realized harder levels came with more spending and more churn. Easing for low activity players came with higher retention and higher spending over the 30 days measured. Rob Alvarez (07:32): Tuning the bar so people succeed with a level of effort is actually something that persons are going to thank you for if you get it right. And avoid tuning it so they just pay to skip because that is not something they will thank you for. And guess what? When you do it the right way, your numbers are gonna be better off in every single one of the metrics when you think of your users and their motivation at every single point of the way. Rob Alvarez (08:04): That is what we call human focused design versus function focused design. Keep that in mind, you’re thinking of the human behind the controller, behind the keyboard. Be f behind the mobile. When you do that, your numbers are just gonna get better.

Where to start: the Core Drives in the Wild guide

Rob Alvarez (08:18): So remember, if you want to get this right, understanding what is the level of difficulty you want to design for, this is not an easy thing to get. And I have a Core Drives in the Wild free guide to offer you to get started on this. This is not going to be the full path. This is not going to get you completely there. In Octalysis, consultants, as we are, we spend years working on this until we’re really able to cater for these needs. Rob Alvarez (08:45): We do a lot of research, a lot of data. But You can definitely get started down this path and definitely get better and understand a lot better how difficulty curves are working if you at least have a good grasp of how the Core Drives in the wild function in real corporate settings. So all you have to do is click on the link in the description. You’ll get the guide on your email through several emails and on several days. So you can digest it slowly and really take in any of the lessons that you might have. Rob Alvarez (09:13): And of course, you can always reply. Asking me any questions that you might have from that. And as always, at least for now and for today, it is time to say that it’s game over. End of transcription

Discover more from Professor Game

Subscribe to get the latest posts sent to your email.