The Appearances and The Things Themselves

Here's a neat puzzle by Scott:

My dermatology lecture this morning presents: one of those Two Truths and a Lie games. You choose which two you think are true and – special house rule – give explanations for why. The explanations do not require specialized medical knowledge beyond the level of a smart amateur. Answers tomorrow-ish.

1. Significantly more Americans get skin cancer on the left half of the face than on the right half.

2. People who had acne as children live on average four years longer than those who did not.

3. In very early studies, Botox has shown great promise as a treatment for depression.

My thoughts below the fold, you may want to guess first.

Initially, I reasoned thusly:

1 seems true because it's that driving thing, right? People get very slightly more sunlight on their left, and while sunlight is good for most people on net, it also slightly increases skin cancer risk. So, I'm unsurprised by this finding.

2 - no idea what mechanism would cause this. I'm stumped. Acne isn't a signal of health, and healthy behavior doesn't cause acne.

3 seems plausible because if Botox suppresses frowns, there's some evidence that the appearance of less unhappiness in the mirror will make people happier via subconscious feedback and reconciliation of conflicting signals.

So 1 and 3 are the true claims, and 2 is false.

But then I saw this comment:

2) I think is true (but sneaky/misleading use of statistics) – acne strikes late in childhood so part of the population of people who never experience acne are people who died pre-puberty and ‘did not have acne as children’, pushing down the average life expectancy.

My first thoughts:

AAAAAAAAAAAAAAAAAAAAAGH! Guh. Buh. ..........................

Translated into words, this roughly means:

I was incorrectly looking for an estimate of how probabie it is that the risk factor described by the study causes the effect measured by the study.

I should have been looking for an estimate of how probable it is that the study came out as described.

This is like the difference between estimating "Will voting for Democrats benefit me?" and "Would the President of the United States say that voting for Democrats will benefit me?" Or the difference between guessing whether evolution is true, and guessing whether your textbook will say so.

Taking the latter approach, the problem seems totally different.

The causal mechanism behind 1 still seems plausible but the effect would be very slight and I can't think of any tricks in the data that would make it more likely.

2 seems likely because of selection effects - people with acne lived long enough to get acne. Plus maybe some reporting effects - does data get gathered equally well among middle-class and poor kids?

The true causal mechanism behind 3 is "early studies." Early studies are often small and show all kinds of weird effects to be "statistically significant." This XKCD comic illustrates it pretty well, but let me know if you want me to try to write a simple explanation of statistical significance.

So Botox might not do anything to cure depression, but early studies are not unlikely to show that it does. This is why we wait for a consensus among researchers instead of doing the costly and only-slightly-better-than-random-on-average things early studies suggest.

Therefore, 2 and 3 are true, and 1 is not.

On the meta level, I did the problem in two different ways, and both times decided 3 is true. Even though each time, this was with less confidence than the other one I thought was true, when I combine my two methods I should believe 3 is true more strongly than either 1 or 2. Here is my prediction that 1 is false. Here is my prediction that 2 is false. I will be fairly surprised if 3 is the false one.



Apparently none of them were false. I don't even know how to update my process to deal with this, though I should have made my PredictionBook predictions conditional on one of the three answers being false. As it is, I lose lots of Bayes points.

I do kind of stand by the above reasoning, except for the assumption that if two of the three  are true, the other must be false. Serves me right for trusting a quiz question to be accurate. Even one written by Slate Star Codex. Next time I will not assume the sample space is partitioned by the three stated possibilities.

Oh, here is that XKCD comic:



Leave a Reply

Your email address will not be published. Required fields are marked *