I grew up in a Jewish household, so I didn't have Santa Claus to doubt - but I did have the tooth fairy.
It was hard for me to believe that a magical being I had never seen somehow knew whenever any child lost their tooth, snuck into their house unobserved without setting off the alarms, for unknown reasons took the tooth, and for even less fathomable reasons left a dollar and a note in my mom's handwriting.
On the other hand, the alternative hypothesis was no less disturbing: my parents were lying to me.
Of course I had to know which of these terrible things was true. So one night, when my parents were out (though I was still young enough to have a babysitter), I noticed that my tooth was coming out and decided that this would be...
A Perfect Opportunity for an Experiment.
I reasoned that if my parents didn't know about the tooth, they wouldn't be able to fake a tooth fairy appearance. I would find a dollar and note under my pillow if, but only if, the tooth fairy were real.
I solemnly told the babysitter, "I lost my tooth, but don't tell Mom and Dad. It's important - it's science!" Then at the end of the night I went to my bedroom, put the tooth under the pillow, and went to sleep. The next morning, I woke up and looked under my pillow. The tooth was gone, and in place there was a dollar and a note from the "tooth fairy."
This could have been the end of the story. I could have decided that I'd performed an experiment that would come out one way if the tooth fairy were real, and a different way if the tooth fairy were not. But I was more skeptical than that. I thought, "What's more likely? That a magical creature took my tooth? Or that the babysitter told my parents?"
I was furious the possibility of such an egregious violation of experimental protocol, and never trusted that babysitter in the lab again.
An Improvement in Experimental Design
The next time, I was more careful. I understood that the flaw in the previous experiment had been failure to adequately conceal the information from my parents. So the next time I lost a tooth, I told no one. As soon as I felt it coming loose in my mouth, I ducked into the bathroom, ran it under the tap to clean it, wrapped it in a tissue, stuck it in my pocket, and went about my day as if nothing had happened. That night, when no one was around to see, I put the tooth under my pillow before I went to sleep.
In the morning, I looked under the pillow. No note. No dollar. Just that tooth. I grabbed the incriminating evidence and burst into my parents bedroom, demanding to know:
"If, as you say, there is a tooth fairy, then how do you explain THIS?!"
What can we learn from this?
The basic idea of the experiment was ideal. It was testing a binary hypothesis, and was expected to perfectly distinguish between the two possibilities. However, if I had known then what I know now about rationality, I could have done better.
As soon as my first experiment produced an unexpected positive result, just by learning that fact, I knew why it had happened, and what I needed to fix in the experiment to produce strong evidence. Prior to the first experiment would have been a perfect opportunity to apply the "Internal Simulator," as CFAR calls it - imagining in advance getting each of the two possible results, and what I think afterwards - do I think the experiment worked? Do I wish I'd done something differently? - in order to give myself the opportunity to correct those errors in advance instead of performing a costly experiment (I had a limited number of baby teeth!) to find them.
Cross-posted at Less Wrong.