I got a healthy dose of criticism on Facebook for the experiment I outlined in the post My Chamomile Tea and Sleep Quality Experiment. To bring everyone up to speed, I design tests and collect data about myself. I am just one member of a growing movement called The Quantified Self. The purpose of my tests are to achieve better health and performance. Not for society, but for me.
Problems With the Test?
A few of the problems that were pointed out to me included sample size, length of test, beverage comparison, confirmation bias and placebo effect.
- Sample Size -My test with chamomile tea involved a single subject – me. Since I don’t care if chamomile tea helps others, I think the sample size of me is perfect. That is really the point of the Quantified Self. I’m testing ideas where the responses can be highly individual.
- Test Length – As for the test length, is 60 days enough? For me 60 days is about the most patience I will have with a single test. If I don’t see a measurable benefit in 2 months, then I assume that benefit is at best too marginal to continue investing resources in tracking. At that point it is better to move onto a new experiment. If I do see a benefit, then that is wonderful news. After all, that is what I’m seeking. That benefit may be short lived, but if I can see it, then it is real.
- Beverage Comparison – Another criticism was that I didn’t compare the chamomile versus other warm soothing beverages. My response is that I chose chamomile because it has best reputation for assisting with sleep. Also, my test showed it did not improve sleep quality. Had the test showed a positive outcome, I could have experimented with other warm beverages to gather comparison data. Note that my test was for chamomile tea, not all warm beverages. There very well could be a warm beverage that improves my sleep quality.
- Confirmation Bias – Since I only test ideas that I think might have a chance of showing a positive outcome, the possibility of confirmation bias does exist, especially early in the test. You want something to work, so that might influence the results. This is where having a lot of data helps. I have discovered on a few tests where data in the first 3-5 days tends to be more positive than the overall test. This is why I collect data over longer test periods.
- Placebo Effect – I wish I fell for the placebo effect. Life would be so much sweeter if I could go to a supplement store, imagine how great some pill was going to be and then have it all materialize. If I believed strongly enough that chamomile tea would improve my sleep quality and it did, then I would count that as victory. A win is a win. Unfortunately, in the year I’ve been doing testing that hasn’t happened yet.
No test is going to be perfect, but not testing is a sure fire way to lose. Spending a few minutes each morning in front of a spreadsheet for the past year has measurably improved the quality of my life. At the point when I stop receiving benefits to justify the minimal time commitment, I’ll stop collecting data. But I don’t see that happening anytime soon. I’ve only just begun.
Photo by Craig Chew-Moulding
chuck
Mar 14, 2012 — 8:41 am
that is why i hate Facebook
MAS
Mar 14, 2012 — 9:08 am
@Chuck – The audience is certainly different. They do seem to like recipes though.
Alan
Mar 14, 2012 — 12:57 pm
MAS,
I thought I would make a comment about Chamomile Tea. I remember reading somewhere that it is a member of the ragweed family. It might
not be a good idea to consume it all the time if you believe you have any sensitivity to ragweed. Of course you might never have a problem!
I ultimately did have a problem when I was comsuming Chamomile Tea
and stopped.
As to Facebook, they’re all experts, not likely.
As you said, no test is perfect, but doing nothing at all will not solve the problem. I commend you for all that you are doing.
I have benefited from many of your suggestions especially in the fitness area. It got me to try things that I might not have otherwise.
MAS
Mar 14, 2012 — 8:39 pm
@Alan – I did not know that about ragweed. Fortunately, I don’t have an outdoor seasonal allergies, but I know someone that does. That is good info.