Rob Alvarez
, , , ,
0

Why People Quit Your AI After One Mistake | Episode 468

  Listen on SpotifyListen on Apple PodcastsAvailable on YouTube Get the free Core Drives in the Wild guide, behavioral design applied to real products: professorgame.com/WildCD

Episode Summary

Rob breaks down algorithm aversion, showing why people abandon an AI tool after its first visible mistake, even when it outperforms them. He covers the 2015 and 2018 Wharton studies by Dietvorst, Simmons and Massey, where letting people adjust a model’s forecasts by as little as 2 points out of 100 lifted the share who chose it from 47% to 68%, and connects them to Core Drives 3, 4 and 8. He also explains why he keeps using the AI that helps prepare this show, despite it inventing proof every week. Listeners learn how to design a small, bounded handle for AI rollouts and how to tell a real control from a placebo button.

About the Host

Rob Alvarez is Head of Engagement Strategy, Europe at The Octalysis Group (TOG), a leading gamification and behavioral design consultancy. A globally recognized gamification strategist and TEDx speaker, he founded and hosts Professor Game, the #1 gamification podcast, and has interviewed hundreds of global experts. He designs evidence-based engagement systems that drive motivation, loyalty, and results, and teaches LEGO® SERIOUS PLAY® and gamification at top institutions including IE Business School, EFMD, and EBS University across Europe, the Americas, and Asia.

Key Takeaways

  • In Dietvorst, Simmons and Massey’s 2015 Wharton studies, people who watched a statistical model make forecasts were less likely to bet on it, even after seeing it outperform a human forecaster.
  • In the 2018 follow-up, only 32% chose a forecasting model they could not change, while 76% chose the same model when they could adjust each forecast by up to 10 percentiles.
  • The size of the handle barely mattered: 71%, 71% and 68% chose the model when allowed to adjust by 10, 5 or 2 points, against 47% when no adjustment was allowed.
  • People who could adjust the model were less shaken by its errors and later handed over completely more often, 30% versus 12%. A small handle is how people learn to let go.
  • Two rollout failures mirror each other: full autonomy dies at the first visible mistake (Core Drive 8, Loss & Avoidance), and approval on every step turns people into rubber stamps.
  • An approval step nobody ever uses to change anything works like the door-close button in most US elevators. Core Drive 3 (Empowerment of Creativity & Feedback) needs a control that really changes the outcome.

Topics Covered

  • 0:00: The AI that lies to me every week
  • 1:17: Gartner’s 40% and what their list leaves out
  • 2:16: Why we fire machines for mistakes we forgive in people
  • 4:11: Two points of control changed the numbers
  • 6:17: Calibrated trust and Core Drives 3 and 4
  • 8:55: Full autonomy vs approval on everything
  • 10:10: Approval steps as placebo buttons
  • 11:35: How to build a small, bounded handle
  • 13:16: Four questions for your own AI rollout
  • 14:06: Where the research stops
  • 14:52: What could they have done about it?

Mentioned in This Episode

Free Resources and Get in Touch

Looking forward to reading or hearing from you, Rob Full episode transcription (AI Generated)

The AI that lies to me every week

Rob Alvarez (00:00): So get this, the AI that helps me prepare for every week’s show lies to me on every single one of those times. Some examples. It makes up free resources. I do not have, and honestly, I’m not planning on making either. it even comes up with quote unquote clever names like Gamification Ethics Checklist. It also oftentimes invents some proof. This software as a service client saw a 15% bump. That’s a client I’ve never had. it does things like I do some research, so it helps me find some of that research, and then when I’m using the research, it merges two separate studies into a single one, and the stuff that’s being debated between one or the other now becomes an absolute truth and and and it’s settled by this new study that has never happened. And here’s the thing, I still use it. I’m definitely gonna use it to prepare the next episode I record. And this episode is about why I do that. And the answer is The difference between AI rollouts people keep using and the ones that people very quickly and oftentimes unceremoniously abandon.

Gartner’s 40% and what their list leaves out

Rob Alvarez (01:17): Who is this for? You’re rolling out an AI tool or an agent. The demo looks amazing. Then something happened that was wrong in front of someone. In front of the wrong crowd or the right crowd, it doesn’t matter. Gartner predicts over 40% of agentic AI projects will be canceled by the end of 2027. That’s barely a year from now. the reasons that they state for this are cost, unclear value, and weak risk control. Gartner talks about the value, about the costs, and I want to talk about the part that their list does not name. What happens in people’s head. The first time the AI is wrong. I’m Rob, I’m the head of engagement strategy in Europe for the Octalysis Group, the top, the leading gamification consultancy and behavioral design consultancy in the world. I’m also the host of the number one gamification and behavioral design podcast, Professor Game, which you’re listening to right now.

Why we fire machines for mistakes we forgive in people

Rob Alvarez (02:16): So it all starts with this study done in 2015, yes, before AI, when Dietvorst, Simmons, and Massey at Wharton people were making forecasts and could tie their bonus either to a statistical model or to a human forecaster. The people who watched the model perform first, they were less likely to choose it, even though they had just watched it be better than the human. We forgive a person for a mistake. When it’s a machine, what we want to do is fire the machine for the exact same mistake. What the researchers found is that people believe humans learn from their mistakes. And again, keep in mind this is pre-AI in any case, and catch exceptions while a machine will repeat the same error forever. And even with how much AI is progressing, this is very much still the case. And I can tell you, like I started. There are some mistakes I see the AI do over and over again, even though I state in memory and say, don’t do these things, don’t do these things in the instructions, wherever, it still keeps happening. So the most dangerous moment in your rollout with AI is that first visible mistake, especially when it happens in the meeting, in front of other people. That’s Core Drive eight. Loss and Avoidance. Nobody wants to be the person. Who is holding the tool that’s embarrassing them in front of others? And if you want to understand how Core Drives like Core Drive 8 that I just mentioned, work out in the wild, in reality, in corporate settings, you can get our guide for free by clicking on the link in the description. It’s at professorgame.com/WildCD. Of course, it’s easier to just click on it. And once you can spot Core Drive eight kicking in, you can see why one embarrassing output kills a tool that works most of the time.

Two points of control changed the numbers

Rob Alvarez (04:11): Here is, if this were storytelling, this is the twist. What if you give people some way, some some small way in which they can balance out the mistakes done by the machine? Same team ran a follow-up, right? In 2018, I think is the year where they put it out. So the task was to predict real high school students percentile on a standardized math test from nine pieces of information. Participants were told up front, they they let them know this model tends to be off by around 17.5% on average. Nobody thought it was perfect. They already knew this. If they could just take the model’s answers at face value, just as they come out, 32% of the people chose to use the model. If, however, they could adjust each answer by up to 10 points, 76% chose using the model. Then they made the handle that how much you can adjust it smaller. You can adjust by 10, you can adjust by 5, you can adjust by 2 points. So 71%, 71%. Sixty eight percent still chose the model. Two points on a scale of a hundred. That’s just moving the wheel a little bit, and that was enough. They barely used it anyways. People were allowed to move ten points, generally moved around five. People allowed two, moved a single bit, a single one. What people wanted was the possibility, that feeling of control. Two points was enough to give them that feeling that they were actually the ones in control. The part that matters most for a rollout, that’s their fourth study, was people who could adjust were less shaken when they saw the errors. They trusted the model more than themselves. And in the next round, they were more likely to hand over completely to the model. 30% versus 12%. This handle is how people get to the point where they let.

Calibrated trust and Core Drives 3 and 4

Rob Alvarez (06:17): Go. So back to the the AI now because this is pre-AI, this is pre-LLMs, all these studies that I’ve mentioned. Back to my lying AI. Why I keep actually using it. I can change everything that it hands me. I have a list of the things that I’ve found it often lying, and I check for each one of those lies. I don’t trust the AI itself. I trust me enhanced. By the AI. My trust has specifics. I know already where it is usually going to fail. And honestly, I check that in a lot of detail. Invented proof, merged studies, there’s hype. There’s many things I have to check for that I know have to check for. So I check for those really hard. But the other things I also check. Of course, I’m gonna give it a bit more leeway. I don’t check in as much detail. I’m not already, defending against what it said in these areas, but it’s calibrated. not open trust you know do whatever you want and I’ll let you get away with it and it only exists because I have the chance to meaningfully in fact in this case completely intervene here my Core Drive three is actually funny enough it’s activated Core Drive three Empowerment of Creativity and Feedback because I am creating And I am being enhanced by the AI. I am not handing over my creation. I am not handing over my feedback loops. I’m not handing over anything entirely to the AI. I still keep it that. And there’s even some light Core Drive four Ownership and Possession. Because in this case, creating outlines and creating concepts and ideas, it is something I still own. I care about it in a way. I wouldn’t care about something that you would know, which is just handed to me. And call that AI, call that a producer, call that a team, whomever. It’s not going to be the same if I am going to be intervening. I hold some of that Core Drive 4. I’m not saying a team couldn’t do a great job of that. I couldn’t trust a team or group of people. That’s not what I’m saying. But there is specific Core Drive 4, Ownership and Possession when I am actually part of what I’m receiving. In fact, I remember Yu-kai what we were having a chat about something else, and he said, Well, I I still have his comment was that he every now and then says, Well, I still have Core Drive 4. So what you’re proposing, I don’t like it. Because I have Core Drive 4 over the other idea. So there is a certain motivation that moves precisely just because of that Core Drive 4 Ownership and Possession. So you it’s not something you cannot get over, but it does definitely make you care more about it.

Full autonomy vs approval on everything

Rob Alvarez (08:55): So there are several ways in which companies can get this wrong. Failure, number one, the easy, the direct one, they tell you AI is amazing, so you say Full autonomy. The agents act by themselves. There’s nothing the user can do. And of course, when the first mistake happens, and it’s discovered by the way, because I’m sure there are many more before that one, people just completely stop. That’s a 2015 study. It’s playing out in your organization. Because you’re not giving people a lever. They cannot really handle it. It’s you hand it over to the AI and it screws up, but you’re gonna be responsible. Somebody is responsible for the cr screw up, and trust me. The AI has no responsibility over pretty much anything. Another failure, approval on everything. If you you know that you start scared by that first failure, you realize for whatever reason people said, no way. So you put a sign off on every single step. Within days, maybe weeks, people start clicking approve on everything. The agent is not really saving time anymore, and you’re not in control either. So you’re actually mastering the only way in which you can still have full autonomy. In theory, because you’re just clicking accept, but you’re also wasting time. So you’re piling together both of the mistakes.

Approval steps as placebo buttons

Rob Alvarez (10:10): If you have an approval step that never gets activated, nobody ever says no. It’s basically like a placebo button. that elevator, those studies that say that those are mostly in the US. I don’t know if it’s true in the US and or other places, but they say in most US elevators, the the close the door button does not close the door. any faster for ordinary riders. Of course if you are, have this special privileges, firefighters and so on, they can use it to meaningfully, right? But apparently if you’re the normal user, you click that button, absolutely nothing happens. You have a placebo button. It just gives you that sensation that you have some control. So this fake handle buys you maybe some calm from people until somebody notices. Then it happens all over again. You stop trusting, and you not only stop trusting the AI, you also stop trusting the handle that you’ve given them. Right? The honest version of this is small control that really is able to meaningfully change the outcome. Their surprise in the data. That we found on the studies was that small features in the twenty eighteen studies, and again this is pre AI, people were bounded in an adjustment. You can only adjust up to X percent. And with that, it did better than people that were allowed to change the answers by any amount. Bounded freedom actually was better than unlimited freedom on accuracy as well.

How to build a small, bounded handle

Rob Alvarez (11:35): So what do you want to do? You want to put a real, a bounded handle, something that people can can move around in the moment that the output becomes something. Edit, adjust within a range, reject up to one item, something that changes what happened. Keep it small. On purpose. I when I when I get the outlines from the AI that I’m using, yes, of course I can I could potentially change everything, but that also makes me start working more. And the the whole purpose of being helped by AI, if I give it the right context and so on, is not to check absolutely everything. So I keep it relatively small, right? because making it bigger, it rarely moves any adoption and the results don’t even get better either. So let the autonomy also be earned in steps. There’s another agent I run for something else. I I didn’t get it to just run alone on day one, for sure. First I let it Three actions. It went well, and five actions, it went well, then five, then ten. Each step was a gate towards the next step that I allowed it to do. And of course, errors will happen, especially with AI, so be blunt about it. Show the error. Look, this is the error that happened. In this study, people already knew that the model was wrong by around 17 points. 17 points is a lot, it’s almost a fifth. They already knew that they would this was wrong on average around that. And seven and ten were still choosing that model because they had a handle. Well, that’s what I observed. that’s with the reason I understand they had some control over it, so they were able to adjust for any mistakes that they knew the model.

Four questions for your own AI rollout

Rob Alvarez (13:16): So, what are some questions you might ask yourself? When your AI gets something wrong, what can the person do in the next 10 seconds other than, of course, just stop using it? Another question to ask yourself: does anyone ever use the approval step? If you have any approval steps to change anything? If the approval rate is close to 100%, it’s a stamp. Next question: Does your handle actually change the outcome? Or is it an elevator, close the door button? Make sure that if you are giving a handle to people, the handle actually moves the needle. And Ask yourself also who saw the first mistake. What did the system let them do about the mistake? Because if the system already has this built in, the chances are that the people are going to be more forgiving and more willing to actually use the system. Of course,

Where the research stops

Rob Alvarez (14:06): the studies I showed, some of them were on students, these are not enterprise agents. why would it transfer? Well, the idea is that here we’re understanding people, we’re not understanding. organizations and people’s brains tend to work relatively the same way. And we’re I’m trying to understand here how we react to a mistake made by a non-human agent. Right? Of course the effect has boundaries. There’s other studies, later studies, I think it’s by MIT, that also found that this aversion, this loss aversion, starts going down when the the error of the AI is relatively small. Of course compared if you say, well an a human made a mistake, I forgive them, but they’re human, they’re able to learn If the AI makes a a mistake, I don’t forgive them. But of course, if the error of the human is bigger and the AI is smaller, then this perception, of course, as you can expect, starts to change.

What could they have done about it?

Rob Alvarez (14:52): So back to basics, people keep using an imperfect AI when they think that they can fix what it gets wrong. If you don’t think you can fix it and the AI is gonna do wrong stuff, you’re not gonna use it. No surprise. If you say it and you see it that way, it’s not surprising at all. So the next time your AI gets something wrong in front of someone. All you have to do is ask one question. What could they have done about that mistake? And if you won’t understand how these Core Drives keep playing all these situations, all the motivations, and how they can be used and they are being used in corporate settings, I invite you to click on the link below and get the Core Drives in the Wild free guide, where through different cases that have happened in the podcast, situations that are related to these episodes, my analysis of them, why they work or didn’t work, you’ll get an email a day with each of these different Core Drives, and you’ll be able to use them in your own situations. And at least for now and for today, it is time to say that it’s game over. End of transcription

Discover more from Professor Game

Subscribe to get the latest posts sent to your email.