Experimentation in the New Age
EEric Singler: Hi and welcome to a new episode of B.E. GOOD, brought to you by the BVA Nudge Unit, a global consultancy specializing in the application of behavioral science for successful behavior change.
Every month, we get to speak with leaders in the field of behavioral science in order to get to know more about them, their work and its application to emerging issues.
My name is Eric Singler, founder of the BVA Nudge Unit, and with me is my colleague Jenic Mantashian.
JJenic Mantashian: Hi Eric, it’s great to be back for another episode. Today our audience is in for double the fun, as we have two amazing guests with us. Both of whom I had the pleasure of learning from directly during my time at Harvard’s Executive Education Program on Behavioral Economics.
First, we have Mike Luca. Mike is the Lee J. Styslinger III Associate Professor of Business Administration at Harvard Business School. His writing has appeared in publications including the Atlantic, Wall Street Journal, Harvard Business Review, and Slate : • Did free pens cause the Opiod Crisis ?
• Why we don't value flextime enough ?
• How to suppress your inner scrooge ?
• Shutdown shock may endure for federal workers
• How to design and analyze a business experiment
• Fixing discrimination in online marketplaces
• Algorithms need managers too
• AirBnB needs to do a lot more to fight discrimination
And we also have Max Bazerman. Max is the Jesse Isidor Straus Professor of Business Administration at the Harvard Business School. He is the author of The Power of Noticing and the co-author of Blind Spots, Negotiation Genius, and other books.
In their most recent work, Mike and Max have collaborated together to co-author the book The Power of Experiments: Decision Making in a Data-Driven World.
EMike, I had a chance to meet with you around one year ago in Paris when you invited me to be part of a panel sharing our experience regarding the application of Behavioral Science in France with a group of MBA immersive course Harvard students. It was a very insightful experience for me…and now I am very proud to have a “certificate of appreciation” from Harvard Business School which – I think - is a very smart nudge to reward your speakers.
Max, I haven’t had the opportunity to meet with you in person, but I feel like I know you because you are one of the biggest stars in the field. I have read several of your books and I am a big fan of the work you have done, especially in negotiation and ethics. So there are many reasons why Jenic and myself are very happy and honored to have the opportunity of an in-depth conversation with you in our B.E. GOOD! podcast!
MMax Bazerman: Thank you both for welcoming us to your podcast, and we're delighted to be with you, and we're delighted to be with people who have been so thorough and knowing our work.
MMike Luca: Thank you for having us. I do remember and learned a lot from our experiences over the past year and a half.
J Great, awesome. So let's get into our first question. It's not by accident that you were both guests on our podcast today and you both also authored the book, the power of experiments. Mike, can you share with us how this collaborative project came about?
MMike: This project is now a few years in the making. So to give a little bit of background, I remember when we actually kicked off this project, I had been sitting in Max's office and we were chatting about the use of experiments, but actually we weren't thinking about it as a research tool so much, even though both of our backgrounds are in using experiments for our research, we were thinking about the role that experiments are increasingly playing in organizations.
We started thinking about this pretty dramatic shift that you see as you have experiments playing a managerial role. And we started thinking about why experiments are so impactful in organizations and what are some things that are going well when managers are implementing them, and what are some areas where you could get more from using experiments in organizations?
We set out to write this book and really we had a few goals in mind. Just to give a little bit of a sense of what we were thinking; we were wondering if we could help to give some more frameworks for why people should be running experiments, and for those who are running experiments, how you could get more out of them.
J Great. Well, I'm looking forward to hearing about some of those. And before we get there, Max, I have a question for you. I was a little curious; what's the process like co-authoring the same book? Because I know with myself, I find it very difficult to write by myself. I can't even imagine sharing this effort with a second person.
MMax: It's intriguing on lots of levels. Going backward, Mike and I spent a lot of time working together on the teaching front regarding experiments. So we jointly developed a course that trained our students in behavioral economics and experiments. And we travelled with them to work on projects in the UK and the Netherlands. Mike has more recently been in Paris with that course.
We thought a lot about the question of what do people need to know about the topic of experiments as they move into real-world organizations. By background, I'm a laboratory experimentalist, so most of my publications are based on fairly abstract laboratory studies.
I've been absolutely fascinated to see experiments in the last 10 to 15 years, move out of the lab and have a tremendous influence in the real world.
I'm surrounded by colleagues who are younger than me, and that includes Mike, but I'm also thinking about people like John Beshears and Todd Rogers and Katy Milkman and others who are really at the forefront of the real world study of behavioral economics through the use of experimentation.
So when Mike came to me and said, what do you think about the idea of transforming what we've been doing from a teaching perspective into a book perspective, it sounded like a terrific opportunity. In terms of the process, I view writing a book is an opportunity to learn. So when I'm done with a book, I know so much more than when I start.
So for me, writing a book is not simply the process of telling the reader stuff that I've known. It's about learning about a topic that I know a moderate amount about, but I can certainly learn more from.
Mike is kind of a terrific compliment for me because he knows stuff that is so dramatically different than what I know. So again, I started in the lab and only more recently moved to the field. Mike is kind of obsessed with the field, much more than with the lab.
Most of my consulting work has been with offline companies. Mike is the world's expert on using experiments in the online economy. We know different things. And so the book that we wrote together, which I'm proud to co-author with Mike, it's not a book that I could have written on my own. So it's not a matter of 'would you have preferred to write on your own or with Mike?' For the book that we produced, I needed to be part of a team that included Mike. So, I'm delighted with the process. And at the same time, like any joint project, at times we would have different views on things and you need to work those out, but that isn't very different than many things that I do in life.
J Great. Well, that's just such a beautiful frame in terms of how you think about approaching a book in terms of it really being an opportunity to learn. So I really love that. Mike, tell us a little bit more about your obsession that Max talks about in terms of what sparked your initial interest in this area?
MMike: Let me just first echo a little bit of what Max's saying here about the background on writing the book because I do think in a lot of ways when Max and I first started talking about this, we were real compliments in doing this. I found it to be a pleasurable process, just to be able to think about things in a different way from how I'm used to thinking about things.
Max has a psych background. I have an econ background. As Max mentioned, he came at it with more experience in the lab. I had more experience in the field. So I think putting together was really thought-provoking. I think along the way, we probably both learned a lot, both from each other and also in chatting with organizations and reading up on the latest literature and thinking about some of the promise and the perils of experiments in organizations.
I also wanted to make one other connection, which is that, the class that Max was referencing, this behavioral insights immersion course, that's the class that Eric had spoken with our students in. In the class, we had essentially nudge style projects. And part of the impetus for this book was when thinking about putting behavioral economics into practice, experiments have become a core part of what that means. And when we visited Paris, we really thought that this is becoming a hub for behavioral economics in practice and had this panel where we started chatting with the leading practitioners, thinking about this from different perspectives, which included Eric. So thanks, Eric.
MMike: Onto your question about the passion for experiments and what really drove this excitement, I've been thinking about this for many years now. Maybe I'll just give an example of one experiment that got me motivated thinking about some of these issues.
For several years, I've been collaborating with companies and doing research on tech companies. The driving factor that's excited me the most about this is thinking about the design choices that platforms make and the way that these design choices are subtly influencing behavior on the platform.
So just to give one concrete example of this, for a decade now, I've been thinking about platforms like Airbnb and the choices that they make. To give an example of this, when thinking about the rise of Airbnb, they made a lot of different decisions about how the platform should be designed and how people should be interacting on it. And some of the choices they made were actually pretty different from earlier platforms.
Just as a brief aside into the history of what these types of marketplaces look like, we could rewind to the mid-1990s and platforms like eBay and Amazon started popping up. They had a lot of promise and some of the promise was to increase the efficiency of transactions. So they started allowing somebody who is sitting in Cambridge to buy a used espresso machine from somebody who is living in Berkeley. And if you visited both places, you might not be surprised that this is a kind of transaction that occurs between them.
When thinking about that, what they're really doing is facilitating arm’s length transactions, and it's a pretty impressive thing. And in addition to increasing the efficiency of the transaction, there's a second implication that people have pointed to in the early days of the internet, which is that it also has the potential to increase equity or increase the fairness of the transaction.
By getting rid of some of the markers of race or gender, you could get rid of the ability for people to discriminate as easily in transactions when things are moving from offline to online. Actually, there are some early papers in the literature that pointed to that as one potential benefit of the internet.
When I started thinking about Airbnb, I was thinking about the choices they made and how they differ from some of these platforms. And in particular, in contrast with Priceline or Expedia or early days of booking, or Amazon and eBay, they started making the photos and personal information of their users, very salient. And then they allowed the host to say, okay, look at the picture and the name of the person and decide whether or not to accept them.
And my colleagues, Ben Edelman and Dan Svirsky, and I looked at this and wondered whether this was going to facilitate discrimination, and got us thinking about- it's not really that the internet is just going to automatically reduce discrimination, but it's more that the platforms are going to have the ability to decide how much discrimination they're going to permit in the marketplaces and ecosystems they are creating.
So Ben, Dan, and I had ran this experiment in which we sent out 6,400 applications to different hosts, fairing the name and the names differed in one way, which is that they were statistically indicative of a user's race.
So some were statistically more common among African Americans, some were statistically more common among white Americans. What we found is that African-American guests were significantly more likely to be rejected than White guests in these transactions. Just to give a sense of the magnitude, they were about 16% less likely to get accepted in a stay.
So we did a lot of work trying to understand when and why discrimination was occurring. And after our research, Airbnb looked at our results and ended up making a series of product design choices that were aimed at reducing the amount of discrimination.
They are largely based on ideas from behavioral economics and market design. Some of them were based on proposals. Some of them were based on other proposals, but looking at all of them, the thing that excited me is by running experiments, you could help understand what's helping and what's hurting customers, ways to reduce discrimination.
When Airbnb went and tried to implement some of these ideas, they actually created a data science team that was centered around running experiments to more carefully evaluate the choices they were making to understand not just, 'is this affecting the number of rentals or their short-run profit?', but also, how is this influencing the amount of discrimination on the platform?
So for me, part of the excitement is going out into the field, working with organizations, thinking about problems that they're facing and trying to think about whether we could use experiments and behavioral economics to help leave users better off.
J Such a beautiful case study, and really relevant. When I think about it being June, 2020 right now, certainly it's great to see the power of experiments being used for good in terms of fighting discrimination. I'd love to hear a little bit more later in terms of other ways experiments can be used for good.
Max, I’m curious, obviously, that's quite an inspirational story. I can understand the trajectory of Mike's career into this space. How did you find yourself in a lab and what's been inspirational for you in terms of this direction?
MMax: I've been an experimentalist essentially my entire career, going back to graduate school. I was an undergraduate, Wharton major at the University of Pennsylvania. And as I moved to graduate school, I started working on experiments. So I've been an experimentalist longer than I would claim to be a behavioral insights scientist or a behavioral economist or whatever the term might be currently.
For me, the journey has gone from having the skills as an experimentalist in organizational behavior, behavioral science, finding the work of Kahneman, Tversky, Thaler pretty fascinating very early on. Working in that area, and as Eric suggested, eventually I did a lot of my work developing decision science into the world of negotiation and ethics.
More recently, again influenced by Mike and a number of colleagues, a generation younger than myself, I have really moved into what we're calling behavioral economics and the use of field experimentation. So it's been a 40 year-long journey for me that's been just terrific.
E Great. Mike, regarding the fundamentals of experimentation, you just gave us the first flavor of this, but my first question is quite simple and of course, inspired by the name of your book. Can you explain what you deem as the power of experiments to be? Why are experiments important?
MMike: It's a great question. It's something we've thought about a lot from multiple perspectives, from an organizational perspective, from an academic perspective, from a platform design perspective. And we thought about this when we were teaching the behavioral insights course. I also thought about it when designing a course for managers on experiments.
When thinking about the role that experiments play in organizations, there are a couple of reasons that experiments are valuable. One is we all have an intuition about what's going to work, but our intuition can be flawed and experiments can help to understand what works so that you could be more confident in the decision you're making moving forward.
Perhaps I'll just give a simple example of this. Just think about the behavioral insights team in the UK when they launched. Now rewind to 2010, they're creating this new unit. They're trying to figure out which tax letter is likely to increase tax collection rates of people who have delinquent payments.
They could have just asked people and said, Oh, which letter do you think is going to be more effective? And I'm guessing the people who wrote the earlier letters already thought they at least had some sense of which one might work or didn't think about-- then I think it was that useful to explore further; instead, they ran an experiment in which they tested different versions of letters.
One experiment they ran took the baseline letter they had been setting and then added a line in which they used what behavioral scientists refer to as social norms. So they said, X percent of back taxpayers have already paid their taxes. You are among the small minority who haven't yet paid and found that this led to increasing tax payment that ultimately pulled millions of pounds of taxes forward in collection.
So thinking about this is a pretty big win for an organization that already had been sending out plenty of letters, which is that this new letter now had the benefit of an experiment on them to make it more effective.
There are other reasons that organizations use experiments too.
Actually let's just stick with the tax letter for a second. One benefit of the tax letter was it allowed them to figure out which letter worked, but a second benefit, and I think this occurs a lot in startups and behavioral teams as they're emerging is that the experiment allowed them to credibly say that incorporating behavioral insights into the tax letter writing process had the potential to help collect back taxes.
So when thinking about-- now, we sort of like take as given that behavioral insights can be useful in policy settings, but at the time, part of the pushback they were getting was, does this really matter, just adding a line?
So even if they already thought they knew which one would work, it helped to provide a proof of concept for the behavioral insights team to go look elsewhere.
When we look at the landscape of the way people are running experiments, probably the biggest bucket of them is just uncertainty about what's going to work and what's not going to work, and using experiments to learn. But a second bucket is running experiments to help communicate credibly with other stakeholders, and other stakeholders could be bosses, it could be peers, it could be investors, it could be other parts of government, and using that to help guide the conversations that an organization is having.
There are other things we walked through in the book as well, but those give a flavor of the way that we were thinking about the managerial role that experiments play.
E That's really very insightful. Max, as much of our focus at the BVA Nudge Unit is on the application of Behavioral Economics and Nudge Theory, can you speak to the role of experiments in the emergence of these fields?
MMax: In simple form, experiments were a necessary condition for the development of behavioral economics. So BVA and lots of organizations exist as experts in the world of behavioral economics, but without experiments, there is no behavioral economics.
So through the seventies and even into the eighties, economists continued to believe in the rationality assumption. So as remarkable as young scholars today, you find that back in the seventies and early eighties, economists were holding on to the assumption that humans were in fact perfectly rational.
And it was the only evidence that could come out of experiments that provided conclusive evidence, that the behavioral ideas of Kahneman, Tversky, Thaler, and many others, in fact, were true. Up until that time, economists simply explained away any evidence that existed about human irrationality based on the inability of other methods to give as clean a result as you can get from experiments.
So it was precision and the lack of alternative explanations that came out of the experimental tradition that allowed the revolution in economics to develop.
So the simplest answer to your question is, without experiments, there is no behavioral economics.
E Great answer. Max, you are also an expert on ethics. So I was curious about your thoughts in this area. In your book, you reference a couple of controversial experiments that left the participant with some potential negative effects. Are there ethical codes of conduct you would urge researchers to follow be it in psychology or in business?
MMax: Stepping back a bit, when I think about ethics, I don't think about simple rules like don't lie and things like that. Lots of other people can provide that guidance. When I think about ethics, I get more of my insight from some of the great philosophers like Peter Singer and Josh Green who focused on how do we create the most good we possibly can across all sentient beings? I want to provide my small part in contributing to creating more value.
And if I want to create more value, I don't want to condone unethical behavior. And when I find people criticizing experiments on ethical grounds, I almost always am listening to them describe an experiment where something unethical occurred in one of the conditions of the experiment, and in a simple form, if there's unethical behavior in one of the conditions of the experiment, the experiment is unethical. Not because it's an experiment, but because there's unethical behavior embedded in one of the activities that the experiment helped create.
So I don't think it's experimentation that's unethical. I think that if any behavior that we would include in an experiment that we would find unethical independent of the experiment, that creates the moral problem.
When we think about the method per se, if you will even try out new ideas and you believe in thinking systematically, and if you believe that it might be a good idea to try out the new idea on a smaller group, rather than trying it on all 80 million UK citizens, then what you're describing is an experiment.
An experiment is a structured systematic way of trying out new ideas before rolling out the idea in a more general way. So the method is not only safe against the accusation of being unethical, I would say it's very unethical in many contexts to move forward with ideas without first running experiments. So experiments are shockingly ethical, but that again does not justify unethical behavior that's embedded within that experiment.
E Mike, what are your thoughts regarding ethics?
MMike: Maybe I could jump in with an example to build on what Max was saying here. In the book, we had discussed an experiment that Facebook had run. And in the experiment, what Facebook had done was to change the newsfeed. They looked for posts that were more positive or posts that were more negative, and then experimentally varied whether they're showing more positive or negative posts to users.
And the results suggested that by showing more negative results that the users posted a very small amount of negative words in their own posts.
So I'm thinking about this. Now, one lesson that you learned, they had a headline in a paper that'll come out is that there are some elements of emotions that they're calling contagious. So like I say something negative and that spills over into others that are operating on a platform.
But in our view, that second lesson that Facebook learned along the way is that people didn't really seem to like this experiment being run.
And they got a lot of backlash and the backlash essentially centered around the headlines, I think the Atlantic had one, the Times had one, telling you about whether Facebook was unethical for running an experiment in which they are 'manipulating emotions'.
Now thinking about this in Max's framework, a couple of things were going on; they were randomizing the feed and changing the posts that people see. Now, of course, Facebook does this all the time. They've run thousands of experiments a year changing newsfeeds. And the second is that they're influencing your emotions.
Now of course, every experiment they're running, whether they're measuring it or not is going to affect your emotions. So you're going to see if positive and negative posts affect the emotions of users, then anything that changes the positivity or negativity of stuff you're seeing is going to influence the emotions of people that are seeing those posts.
Now decomposing the ethical issues here, one issue is what Max was talking about which is, if you're opposed to showing negative posts or things that are going to negatively affect emotions, then that's not specifically the experiment, it is just design choices that Facebook makes that are going to create a more negative atmosphere.
A second thing going on is the measurement of data. It's kind of the fact that they're keeping track of emotions. I would say that's also linked to some of the ethical questions that come up around experiments, to the extent that companies are keeping an increasingly large amount of data about users and using it to measure what's happening in experiments, but there, the concern again is not about the experiment that's being run, but about the data that companies are keeping or whether or not they should be having these outcomes at all.
So the lesson we took from this experiment is not so much that there was something wrong with that specific experiment per se, but that the experiment had almost been trotted in a type of secrecy.
And we talked about this in the book where one thing that companies could do is just be generally more transparent about the types of things that they're doing, the types of experiments that they're running. In the case of Facebook, they could say, here's how many thousands of experiments we run per year, X percent were on news feed, Y percent were on advertising, and then just help people understand what the scope of things they are doing, what they're measuring, and instead use experiments to help facilitate discussions. And then you'll start to see a little bit more whether things are out of bounds and if so, organizations could stop doing them. And to the extent that they're inbounds, then people have some clarity about what you're doing, why you're doing them, and why in some of these cases, this is going to be better for different stakeholders.
J Thanks, Mike. It sounds like we could keep going into ethics for a whole another podcast if we wanted to. It's such an interesting topic with so many different perspectives and insights.
I'm curious; you mentioned Airbnb and Facebook, that's the private tech companies. And you also talked about the tax study with the government. And both these sectors, the government and private tech have really been the early embracers of behavioral experiments as is what you've indicated, and in my opinion, kind of an odd pairing. I'm curious, how do you explain the manifestation of the shared label in terms of their culture or other aspects that are similar, that they share?
MMike: It's a great question that these are two parts of the world that people don't typically associate with each other. Thinking about the UK tax department and Facebook; but on the other hand, they both ended up becoming early adopters of experimentation. They came at it from slightly different angles. I think they essentially both came to the same conclusion that as there's a rise of data availability, so in the tax collection, you could see what people are paying and randomize things.
In the case of Google advertisements, you could see what people are clicking on and use that to measure the effect of different advertisements. So I think there was a growth of data that's facilitated experimentation.
The second factor that's facilitated experimentation is just increasing ease of randomization that it becomes just logistically easier to randomize people into different groups so that you could then measure what the outcome is.
The third factor is the increasing acknowledgment that our intuition can be flawed and there are systematic biases. And a fourth factor is just a growth of high profile case studies of experiments that have helped to create value in organizations. I think all of these have led to a proliferation of experiments in different areas.
Now looking at some commonalities between the UK tax department and say a Facebook or a Google, they're both in situations where they have a large number of people that they're engaging with. So they have a wide user base, as in the UK the full taxpaying body, in the case of Facebook all the Facebook users. So that just logistically means that they could run an experiment and have the statistical power to meaningfully understand what the impact of different changes are.
They both have outcomes that they care about. So in the tax department, they care about whether or not people pay their taxes, in Facebook, they care about user engagement, then there's a whole set of metric buried under that.
I would say in the behavioral case, some of the case studies were coming from academic research where people had seen experiments being run, and probably more emphasis on the early case studies of nudge trials that led to large gains for governments.
In the tech sector, they came at it I would say almost more from an engineering perspective and came to the behavioral piece perhaps later on in the process. But I would say even there, there are high profile examples of successful experiments that people now point to and thinking about the role that experiments play in an organization.
J Max, earlier we were talking before the podcast a little bit about BVA Nudge Unit's experience working with companies in the private sector, and I wanted to get your perspective on this. Why do you think other sectors have historically been slower in the uptake of behavioral experiments? And do you see this trend changing?
MMax: Yes. First of all, I think that this is an interesting case where the government is way ahead of typical private sector companies. And I think that that's just kind of interesting because there are many in the private sector who look at the government as being behind the times. I think that because of David Halpern building off of the likes of Thaler and others, the government is way ahead of most of the private sector in terms of the use of experiments.
I think Mike did a good job of highlighting the reasons that both of those areas developed. And I think that the rest of the private sector will get on board and that they're losing lots of opportunities the longer they take to do so, but what's different?
One of the things that I would highlight is that both the government and the tech sector have the ability to make relatively small changes on easy to change things like the online platform or a form, and influence millions of people with this relatively small change.
So in many cases, when we see remarkable results in terms of the UK collecting taxes, we're not seeing enormous percentage changes, we're seeing small percentage changes that add up to tens of millions of dollars at a cost of approximately zero. We're getting the benefit of a medium to large impact with trivial costs and an enormous return on investment. What we don't see, and I'm now moving to your question more directly; we see far fewer experiments where we're trying to change the way humans actually interact with other humans.
We're seeing far fewer experiments in the private sector having to do with how people conduct performance evaluations. We see far fewer experiments on how people negotiate.
I had a pretty amazing experience a few years ago. Danny Kahneman was working with a very large insurance company on how to improve their decision making. The CEO had been fascinated by Thinking Fast, and Slow, which led to a fairly large consulting engagement.
And when the project took a turn toward negotiations, Danny basically brought me into the project, and we developed a fairly comprehensive program for how to change how thousands of claims agents would interact across more than a hundred countries, including a training program, but also a very different structure for how to think about the negotiation process.
And we presented this and the executives were moderately easy to be convinced that our ideas were good. And then we basically said, let's test it before we roll it out worldwide. And we made absolutely no headway whatsoever with the idea that we should slow down long enough to test first. They simply wanted to implement it on a worldwide basis and that's what happened.
I think that there are too many consultants, too many consulting firms. So I'm indicting all of us who are on this conversation potentially, who can more easily convince a client to implement rather than to test and implement. And I think that our clients should want tests and they should want better evidence, and we should be more interested in providing evidence that the insights that we're providing through our consulting work in fact are worthy of implementation.
So I think that consultants, CEOs, executives throughout the organization should test more, even though it's harder to test a new negotiation process in comparison to changing the color of a button on an online platform.
I'm not claiming that the shift will happen as smoothly, but it's worthwhile and we should see lots more of it. And I predict we will. I think the question is how quickly we're going to see moving into the private sector beyond the tech world.
J Great.
MMike: Can I just jump in on that for a second? I do think that there's already starting to see the transition. I think the managerial toolkit on experiments is evolving and developing and it's gone from purely being thought of as a statistical problem, which I think has happened in some of the early experiments that have been run to more of a managerial problem. And as there's more awareness among managers, I think that is helping to speed the shifts somewhat.
There are some examples of large organizations that have run much larger scale experiments. Just to give one example, since we're talking about consulting is there's a nice series of experiments in which consulting is given out to some organizations, but not others.
This is by Nick Bloom and co-authors. What they wanted to see is what's the returns to consulting. So they implemented baseline surveys, they rolled out some businesses and others gave some consulting, and then they tracked how their productivity changed, how their management practices changes.
And then you could triangulate 'what's the impact of consulting?'. It actually turned out that it had a big impact on productivity, but you can also do things like, what are the management practices that are changing? Then you could start to not only see 'does this work?'-- and we talk about this in the book as well, but to start to understand mechanisms. What are the things that consulting seemed to be driving behavior or helping organizations to thrive? And then it's useful not just to see if there any returns to consulting, but also how should you design on an ongoing basis, a better consulting product to offer to people?
So I think we've seen in a growing number of sectors, examples of large scale experiments being run, now a kind of follow-up work, there's been things like 'what if consulting looks like this?'. What if it's smaller-scale interventions trying to test some of the boundary conditions. So I do see lots of evidence out there is a growing shift in this area.
J Great. Thank you for the incremental insights on that topic. Mike, you already spoke a bit about the Airbnb case study and how the impetus for that was really around discrimination and that's one way that companies can use experiments. So I wanted to ask another question about your experience in terms of the private sector. Is there a favorite case study of yours where you really saw where experiments impacted the bottom line and perhaps the surprising way; just in case companies aren't convinced yet that they should do experiments? I'm just wondering if there's a great story that you could share around that type of case study.
MMike: Yes. There were a lot of experiments that I think I moved the needle in organizational decisions. I do think that the Airbnb experiment is one where the company went from measuring a subset of outcomes to a broader set of outcomes, and then more carefully thinking about how to evaluate tradeoffs and help to make sure they're offering a platform that's not only optimized to short term growth but also thinking about the inclusivity of the platform, which hopefully, in the long run, will lead to a more successful platform as well.
I'll give a couple of other examples because I do think it's important for organizations to get a sense of what the landscape looks like and where there have been returns. The tax letter is the second one. The third one would be that e-Bay had run a large scale experiment.
Now here's a company that have been experimenting in some areas for a long time, but one area where they hadn't been experimenting is on what the returns to advertisements that they were purchasing were.
So they have been advertising on Google and Bing a team of economists. So Steve Tadelis, Chris Nosko, Tom Blake, that had come in and looked at the advertisements they were running on Google and said, Hey, are we really sure that these things are having the impact we think they have? We're spending $50 million a year on Google ads. Maybe we should test it.
Now, again, I think this touches on a theme -- and the book, we try not to have kind of a, yes, no perspective to 'is this good?', 'is this bad?' But we saw the current advertising package or offering clearly was wasting a lot of money, but they saw areas where there were more promising returns.
So if they look at things that are less associated with eBay, used guitar of this type, there seem to be higher returns to advertisements, than when somebody is just searching for something like eBay shoes; that's traditionally associated with eBay. I would say that's another one, the returns to consulting is another experiment. I think we could have an entire session on the success stories of experiments.
I would say the common theme to all of these is areas where you could make a change and where there's a quantifiable outcome that you care about, where this could help you to understand what works and where you're not certain about which thing is going to have the more successful outcome.
E Max, another topic, what advice would you give to businesses and practitioners within private companies to set up a structure and culture around experimentation and behavioral insights?
MMax: You asked structure and culture, so I'm going to start backwards. I'm going to go with culture, let's say culture and norms.
What I would highlight is that companies regularly develop new ideas and they implement them, and they typically have fairly noisy information on whether the idea worked because it was a good idea or because the economy improved or some other random quirk that happened in the world.
And we need that to come from the top where the leaders in fact value the idea of testing.
And that was true in both the UK episode that Mike has talked about, as well as in the tech sector where leaders have basically endorsed the idea of why don't we test it rather than implement the best idea that we can come up with.
I think that we're a fair distance from having non-tech private sector companies worrying about the structure for implementing how we conduct hundreds of experiments a year. But when they get there, I think organizations can do a lot about that. In our book, we talk about Booking.com who basically developed a quite extensive platform to make running experiments easy for people throughout the organization.
And I think that as we think about non-tech experiments, those are going to be more complicated, but are likely to be well worthwhile. But I think we're years away from needing this structure. What we need to do now is change the culture and norms around experimentation.
E Okay, thanks. Mike, are there common mistakes that you have observed when companies engage in a more experimental culture that should be avoided?
MMike: Yes. So, you're saying for the more experimental, heavy organizations; and I think of one meta lesson of the book really is just to have a value and appreciation of the value of experiments to get organizations started with experimenting.
For companies that are experimenting a lot, I do think having a systematic approach is helpful, but if I had to point to a couple of things that are lessons that I see from the experiments we talk about in the book and more broadly, one is measuring the thing that you really care about. So I think that lots of organizations track the thing that's easiest to track, and I think having a slightly more thoughtful process for understanding what's the managerial goal and having your outcomes in the experiment match those, I think is important for organizations to be explicit about it.
A second is about understanding what question you're asking.
When thinking about this, we can think about two types of experiments; an experiment that's essentially a product evaluation or policy evaluation, where you're trying to test the impact of something as is; and then we could think about something like a mechanism experiment, where you're trying to understand why something works. And the 'why something works' you would hope to generalize to understanding how might you tweak the design of this product or policy or intervention, and also try to generalize the other problems that you're facing in an organization.
Sometimes if we think about some of the early A-B testing that happened in tech companies, I think probably there were too many pure-- what people sometimes refer to just as A-B test, to say, okay, here are 10 different designs of this thing. Let's just figure out which thing is most effective, and not enough thought about let's think carefully about the design and why things are working to build up toward a more generalizable framework for an organization.
E Max, in the book, you cover the topic of intuition. Surely there is a place for intuition in decision making when it comes to business. How would you address this question of intuition?
MMax: There are lots of executives out there who would love the message that they can trust their intuition. And there are many authors out there who are willing to take advantage of that desire and write books that encourage executives to trust their intuition. But the evidence is shockingly against it.
Does intuition have a role? Yes. It should be a far smaller role than executives might like. Consistently, in both rigorous analysis and through anecdote, we can see the limits of intuition, and virtually every experimental test of intuitive versus systematic thinking or system one versus system two thinking, system two thinking provides us with far better results in any quantitative measurable form that we could imagine.
And the fact that executives want to trust their intuition is simply a psychological barrier to moving toward more systematic ways of making decisions. And as I hinted earlier, I'm a bit older than Mike, so I think of the two most amazing potential disasters of my lifetime. I think of one, the Cuban missile crisis; And had we followed our intuition, it's unclear whether the four people on this call would be here today because the world would have been fundamentally changed had the US leaders gone with their intuition. It required systematic thinking to keep us from launching what would have turned into extended nuclear war.
More recently, we've had another challenge, it's called COVID-19. The US has a leader who trusts his intuition, and the fact that he trusts his intuition over the best science available, is certainly responsible for tens of thousands of additional deaths. So we see that this trust of intuition is not only against scientific evidence that comes from an experimental context, but it has profound influences on the most important decisions that our leaders can make.
E At BVA Nudge Unit, our mission is to apply behavioral science for good, including in the workplace. Thinking about this topic more generally beyond experimentation, where do you see the biggest untapped opportunities for this in the private sector?
MMax: I think that behavioral economics can do a great deal of good in changing how leaders think about their task. And when I think about the topic of leadership and what the literature looks like, I don't resonate with that being what leaders need to do. What I think leaders need to do is make great decisions. So we need to tap the insights from the behavioral economics field to help leaders make great decisions, but we also need to recognize that what makes leaders unique from other important professionals is that leaders not only affect their own decisions, but they affect the decisions of all the people that they lead and they can create the choice architecture that can lead those that they lead to make far wiser decisions. This focus on wiser decisions, I think is the other important dramatic change that we could see.
J Great. Thank you, Max. Mike, I wanted to also ask you about the Coronavirus pandemic. Max touched upon it already very briefly, and I wanted to get your thoughts. Given the need to move very quickly and respond to issues related to our health and economic policies, what advice can you share with leaders on this type of decision making?
MMike: This is a complicated question. I would take on a narrower version of this, which is what's the role that behavioral economics and the role that experiments are playing in some of these discussions right now? I would say both on the individual behavior front and on the business front that they're areas where bringing behavioral insights in might be able to help achieve different policy goals. So when you think about anything from encouraging social distancing or to wearing masks or on the business front, thinking about whether to stay open or closed and how to redesign your organization to make sure to keep people healthy while keeping the business productive, there are large behavioral components to all of this.
We know from the behavioral literature, what kinds of systematic mistakes could happen in these contexts. We also know some of the factors that might be able to help encourage behavior. We also know that decision making and times of stress can be especially prone to bias.
So I think it is important for leaders to be thinking about these issues. In terms of the role of the experiments, people sometimes ask us should we be giving up on experiments now, or should we be doubling down? I do think that having different types of experiments and a different focus could help organizations to think about how to react in the near term.
One example, you can think about Yelp!, a company I've worked with a lot in the past. Yelp! is an online review platform. Should they be running experiments to test the impact of different ads? Probably not, but they are rolling out a bunch of new features to think about how to help small businesses deal with COVID.
Should they be experimenting with those features and how to design them and how to best help businesses? Absolutely. It seemed like there would be a large gain understanding of how to best design them.
Also it's an issue we have thought about from a research perspective as well. I'm part of a research team where we've been thinking about small businesses and how to help them navigate the COVID related disruptions they're facing. And I've seen behavioral insights and experiments play a role in this.
To give one example of each behavioral insights, is that forecasting what the evolution of COVID is going to be, and what disruptions are going to be like for your business is a tricky problem. And we know that it's hard to make these kinds of forecasts, and we know the types of biases that might go wrong. So helping to provide better forecast and help to de-bias in that part of the process, could have large gains for organizations. In terms of experiments, in the US, there was the CARES Act, which I'm assuming everyone on this call has followed by giving funding to small businesses. So you might say, Oh, are there experiments that might be helpful? It's not too big of a program for experiments to play a role. Well, actually, before CARES Act had been rolled out, some collaborators and I, Zoe Cullen, Marianne Bertrand, Edward Glaeser and Alex Bartik had thought well, we can't run experiments with the actual policy, but one thing we could do is have a survey of small businesses and vary the information that's given about different hypothetical types of aid, and then ask some questions like 'how long do you think you're going to be able to stay in business?' 'What's the probability you're going to have to close down?' 'How many employees do you expect to have open by December of 2020?', and use that to help figure out which type of policy is likely to be most effective.
So I think on the policy front right now, it would be pretty valuable for policymakers to think about running different types of targeted trials, to get a handle on how different policies they’re considering might affect the outcomes that businesses.
J Great. Thank you. It looks like we are officially out of time, so I just want to wrap up here. Max, I hear you have a new book coming out. Maybe you can share briefly with our listeners a little bit about that, when they can expect it, and how just to stay connected to you. And Mike as well, what's the best way to stay connected to you? And any projects that you want to share with our listeners?
MMax: Thank you. Yes, I have a book coming out in September, with Harper Business, entitled 'Better, not perfect'. It's a book about how we could all think about creating more value in the world. Very consistent with a lot of the themes that we've talked about in terms of making wiser decisions and more ethical decisions so that we maximize value across all sentient beings, and rather than having the unrealistic assumption that we can move to some perfect state that's common in philosophy, my book focuses on how there are so many ways in which we can all create strategies to be better in a sustainable manner.
I'm easily reachable through my Harvard business school webpage, and my email is mbazerman@hbs.edu. Thank you very much.
MMike: I'm also reachable on my webpage at HBS , and my email is mluca@hbs.edu. I'll just take this opportunity to thank both of you for interviewing us today. This was a thought provoking discussion, and a lot of fun to think about the role that experiments are playing in organizations.
E Thanks a lot, Mike. Thanks a lot, Max. It was great to have you on board.