So, I made the book :-). That story about Pearson on pages 220-223? That was me and my team. The first time I heard of Justin was when we Tweeted something in support of A/B testing when the “controversy” was going around Twitter. I appreciated that and the many expressions of support I got from others (including Neil Heffernan who was a guest on the virtual book club).
Interestingly, the way this came to general attention was that we decided to share our findings with the education research world instead of keeping them to ourselves so we presented at the American Education Research Association conference. A reporter from EdWeek picked up on it and ran with the headline along the lines of “Pearson conducts social-psychological experiment on students.” Then a blogger from the Washington Post picked it up and we were off. I spent days doing interviews with other media outlets.
I had never before talked to Investor Relations, the team at Pearson who manages all the communications with investors, but my phone rang that week as our stock price fell. It turns out that the market overall was down so they decided it wasn’t attributable to me but they needed a response in case investors asked. One of the investment rating services reviewed it and concluded “Pearson tries to improve learning. Washington Post objects.” I exhaled. All this because we wanted to see if adding some growth mindset messaging improved performance.
As you can imagine, this has made me somewhat gun shy about A/B testing but also really interested in the public’s view of it. I was fascinated by this article in the Proceedings of the National Academies of Science studying the public’s view of A/B testing. Across 16 studies spanning 9 domains, people were more likely to favor implementation of unproven treatments for everyone than running A/B experiments to prove if it works. I don’t think its just concerns about the data and privacy, I think there is some issue or perception of equal treatment or fairness at play here. I would love to hear some discussion on Monday about how to reconcile many of our belief in this type of experiment as a way to learn about learning and improve our products with the general public’s seeming distaste for them.