Defending the Quantified Self

I got a healthy dose of criticism on Facebook for the experiment I outlined in the post My Chamomile Tea and Sleep Quality Experiment. To bring everyone up to speed, I design tests and collect data about myself. I am just one member of a growing movement called The Quantified Self. The purpose of my tests are to achieve better health and performance. Not for society, but for me.

Problems With the Test?

A few of the problems that were pointed out to me included sample size, length of test, beverage comparison, confirmation bias and placebo effect.

1. Sample Size -My test with chamomile tea involved a single subject – me. Since I don’t care if chamomile tea helps others, I think the sample size of me is perfect. That is really the point of the Quantified Self. I’m testing ideas where the responses can be highly individual.
2. Test Length – As for the test length, is 60 days enough? For me 60 days is about the most patience I will have with a single test. If I don’t see a measurable benefit in 2 months, then I assume that benefit is at best too marginal to continue investing resources in tracking. At that point it is better to move onto a new experiment. If I do see a benefit, then that is wonderful news. After all, that is what I’m seeking. That benefit may be short lived, but if I can see it, then it is real.
3. Beverage Comparison – Another criticism was that I didn’t compare the chamomile versus other warm soothing beverages. My response is that I chose chamomile because it has best reputation for assisting with sleep. Also, my test showed it did not improve sleep quality. Had the test showed a positive outcome, I could have experimented with other warm beverages to gather comparison data. Note that my test was for chamomile tea, not all warm beverages. There very well could be a warm beverage that improves my sleep quality.
4. Confirmation Bias – Since I only test ideas that I think might have a chance of showing a positive outcome, the possibility of confirmation bias does exist, especially early in the test. You want something to work, so that might influence the results. This is where having a lot of data helps. I have discovered on a few tests where data in the first 3-5 days tends to be more positive than the overall test. This is why I collect data over longer test periods.
5. Placebo Effect – I wish I fell for the placebo effect. Life would be so much sweeter if I could go to a supplement store, imagine how great some pill was going to be and then have it all materialize. If I believed strongly enough that chamomile tea would improve my sleep quality and it did, then I would count that as victory. A win is a win. Unfortunately, in the year I’ve been doing testing that hasn’t happened yet.

No test is going to be perfect, but not testing is a sure fire way to lose. Spending a few minutes each morning in front of a spreadsheet for the past year has measurably improved the quality of my life. At the point when I stop receiving benefits to justify the minimal time commitment, I’ll stop collecting data. But I don’t see that happening anytime soon. I’ve only just begun.

Photo by Craig Chew-Moulding