Saturday, December 4, 2010

Measure often to not trick yourself

Improvement is more that often something that is hoped to be achieved “over time”, “as a long term goal” or “continuously”. This goes for both individuals and for teams.

When trying to improve, you will need some kind of measurement to see if are moving in the right direction and if changes you make are improving the metric in the direction you desire. When dealing with subjective metrics, like some form of happiness metric or a feeling-of-fulfillment metric, this might only make sense when looked upon as a trend over a longer period of time.

There is a difference between hard facts (story points, bugs count, test coverage, stock quotes) that can be back tracked fairly easy; and softer metrics (attitudes, motivation, feelings etc) that are not as easy to recover after the fact.
I think the reason for this being so is because of the way our mind works. Our memories of passed experiences are not recorded in detail. We have fragments stored, and the rest is made up by our brain. Made up to the best of our knowledge. Our knowledge at the time we try to recall. The more time that has passed, the worse the quality of the metric.
Paraphrasing Daniel Gilbert, you could say that “The holes in our memories are filled with material from the present”.
(Psychologist Gilbert's “Stumbling on Happiness” is a very entertaining book on how the tricks our brain plays on us when we try to remember stuff from the past, are also played on us when we try predicting the future.)

However, I believe it useful measuring improvement using a softer (and more or less subjective) metric, as long as the samples are made often and analyzed over a longer period of time.

Taking samples often enough might be difficult. Retrospectives-like activities might be a good time for a team to do such activities unless your iterations are too long or, as I've noticed with some teams, retrospectives are only held after successful iterations. (I have also seen softer “satisfaction”-like metrics being taken during daily stand-up but not being recorded properly and thus of very little use.)

In his book “Drive”, Dan Pink recommends giving yourself a Flow Test á la Mihaly Csikszentmihalyi, who coined the term “flow”. Csikszentmihalyi's test persons were asked about their (subjective) mental state around eight times a day, at random intervals, with the hope of minimizing the effect of the mind “filling in the blanks”. Pink states the obvious fact that we now have technology at our fingertips (e.g. cell phones with reminder features) to enable us to perform such tests easily.
And just recently, I saw that Sarah Gray and Corey Haines are working on something they call MercuryApp that seems to be a tool to enable you to set up and take these kinds of tests for later analysis of your trends. (I signed up for an invite but haven't gotten one yet, so I am not sure how well it works.)

In financial investment circles, more than often you will hear “the trend is your friend”. If you want to measure something your mind makes up, measure often and act on the trend!

No comments:

Post a Comment