Everyone Gets An “A”!
(We’re coming up to the first exam of a summer class, for which I’m a TA. We’ve been discussing the importance of study design and accidental influence.)
Professor: “Often, these kinds of things can be hard to predict. Let’s look at a real-life example: when you take exams, there’s always an empty seat between each student. But here in this classroom, you’re all packed in. So, we split you into two rooms. We do this by randomly assigning each student to room A or B. There’s no room actually called, “Room A,” you see, it’s just what we call the two rooms. As you might recognize, both those letters are grades you can get on a test, and we were worried that this might influence the outcomes of the exam. What do you think we found?”
Student #1: “Well… if you’re primed to think you’re going to do well, maybe you’re more relaxed, more confident. So, better exams in room A?”
Professor: “Good! Any other ideas?”
Student #2: “If I’m coming into an exam thinking I’m already in the worse-off group, I’m going to read all the questions carefully, double-check my work, and so on. But if I think, ‘Look at me; I already got an A,’ then maybe I’ll be sloppy.”
Professor: “Also good! Both make sense, both are intuitive. So, [Other Professor] and I got together to study this. We looked up all the grades by room assignment to see what the difference was. We isolated students to switch rooms between midterms and finals to see if they improved or worsened. We even looked into how long they knew their room assignment to see if there was a dose effect. And what do you think we found?”
(There’s pandemonium for a while, while the students argue. Finally, we put it to a vote: 42% think A did better, 14% think B, and the rest don’t think there was a difference.)
Professor: “Despite your votes, the room B students had a higher average! Now… how many of you are checking your emails right now to see who’s in what room tomorrow?”
(Most students sheepishly raise their hands. The rest are too caught up in their laptops.)
Professor: “[Student #3], which are you in tomorrow?”
Student #3: “Uh… 1102? Is that A or…”
Professor: “See, we forgot that the students are just sent the room numbers, and not our little A/B system. So, here’s my last two pieces of information: statistically, flukes do happen occasionally, and we’ve gotten rid of our A/B system entirely!”
(On exam day, I saw that the A/B column I was used to now sorts students into group “A” and group “Other A.”)