I have certain prejudices about these things that I suspect are about to come bubbling up. I’m not going to pick any examples from the book for this piece, I’m going to deal exclusively with generic and invented ones. This has two advantages. The first is that it makes it easier for you to reject what I say, which is perfectly fine with me. The second is that it makes it easier, I think, for you to apply this thinking to a whole barrel of studies. Either is fine, but I think I’m right, and would prefer the latter.
If we were making an advertisement for the latest Chevy and came up with something like I looked at the cheapest Chevy on the lot and thought I was looking at a Cadillac
, it might not be the best ad, but it would be infinitely better than I always buy the cheapest Chevy because I lose less on the resale
. While the second is probably more true than the first, the first is more surprising, more motivating. So an academic psychologist with a study to sell does not want to present it as a run of the mill, completely expected study, but as something at least mildly extraordinary. The best way to do this is to present the results as completely contrary to expectations.
A study that produces a 60/40 split isn’t a particularly earth shattering study, especially if 60/40 was the expected result. I don’t think Psychology journals expect to publish material that merely says, “Gee wiz, we were right.” They want something with spunk. They would rather have something where the psychologists thought the results would be 10/90, but they turned out to be 60/40. Now we have something.
And when the study is properly written, it focuses on the surprising 60, and generally has little to say about the 40. I’m not sure how much we learn from this, but from the studies we learn a great deal. Publish or perish has produced mountains of studies. But have the studies produced mountains of results?
I like to point out that a great many of these studies use college students. I also like to point out the politically incorrect notion that 50% of all people are stupid. Though when we ask people how smart they are, or how smart other people are, the number is always very much smaller. But for the moment, let’s say 50%. That could mean that 50 of the 60 represent people who are merely stupid. The other 10 could be people who misunderstood the question. That would mean that 80% of the smart people showed up in the 40% group. If that were true, it would obviously change our view of the results.
Let’s consider something else. What percentage of smart people, that is people who are intelligent and have things together, are going to waste their time volunteering to participate in meaningless Psychology studies? The higher the percentage of nonparticipants, the lower the overall potential for the test group. It could be that the 60 of the 60/40 represent 100% of the available stupid people. So the results would turn out not to be that 60 of 100 people prefer A, but rather that, generally speaking, stupid people prefer A to B, which is very different from saying that people
prefer A to B.
And now the 40 become rather important. Though they probably tend to avoid participation in Psychology studies, those that took the test greatly preferred B. So the results could be that intelligent people greatly prefer B, and that’s a very different result from the one that would be published as 60/40. Caltech students — remember, Mlodinow is a Professor at Caltech — tend to be very smart, but Caltech is not exactly a hotbed for Psychological studies, and students, as plentiful as they are, may not be our best choice for test subjects in the first place.
I’ve been somewhat ridiculous in this, but the question is, have I been completely wrong? If academic psychologists are trying to sell us interesting results, what are the results they are not trying to sell us, and what would the results be if they did things differently? The study in Trout Fishing in America
was without any value at all. Yet, Mlodinow picked it from a 1954 journal as something of importance. Something that teaches us (or doesn’t teach us) about the game of life. Maybe if a study presented us with a 90/10 result, the 10 would contain an important message.