Sunday, July 15, 2012

Pitfalls of measurement

In my last post, I talked about some ways people try to influence behavior by measurement, and some of the disastrous consequences.

The more I think about it, the more convinced I am that these are not simply exceptional cases. These are the rule, not the exception.

Any time you measure performance hoping to achieve an outcome, you are more likely than not to be disappointed.

There are reasons for this, and some possible answers. I'll talk about solutions in another post, but first it's important to understand why measurement is so difficult.

The biggest problem is that people are inherently competitive. The moment you start measuring something, people are extremely good at making sure they "measure up" - whatever that means. Sometimes this shows itself as explicit and conscious behavior. Sometimes the effect is subconscious or more subtle. Sometimes it is indirect, such as through pressures created within the organizations that people belong to, as a result of organizational measures such as profit.

Whatever it is, measurement influences behavior, and often not in the way you want.

Why is that?

Here are a few reasons...

The measure is not a good indication of the outcome

Many of the things we want to achieve are difficult to measure directly, so we end up with lots of indirect measures. Some of these are reasonable, many are not.

For example, we would like to think we are being led by people most fit to govern.

The theory of democratic voting might work well in small groups, where people truly know each other, and assuming that people have the integrity to vote for those they genuinely believe would be the most suitable.

The trouble is that modern democracy is little more than a measure of popularity, and in a large world where voters have no prospect whatsoever of actually knowing the candidates, popularity has in turn become a measure of all sorts of undesirable traits such as the ability to lie convincingly, or to most effectively trash your opponent.

These traits have nothing whatever to do with fitness to govern.

The measure is appropriate, but is not the whole story

People will strive to achieve whatever is being measured. Sometimes this comes at the expense of something else important, but the other important thing suffers because there is little motivation to pay attention to it. It is not being measured.

For example, a focus on saving lives, and on life expectancy, has led to people being kept alive at all costs - whether they want to or not. In focusing exclusively on one (albeit important) measure, other things like quality of life tend to suffer. Also, health services are under pressure to spend disproportionate amounts on expensive procedures to prolong the lives of a few individuals, at the expense of less glamorous treatments that would ease the suffering of millions.

The measure can be cheated or subverted

One of the biggest problems with many measurements - it is often seen as easier to cheat than to comply. Often, the only thing stopping people is honesty - which is another measure that people respond to in varying degrees.

The point of cheating is to improve the measure by some means other than that which was intended. This varies from outright dishonesty - ballot boxes stuffed, records falsified - to more subtle tactics that fall into a grey zone, such as pressuring marginal students into not sitting exams in order to boost pass rates.

And the scariest part is...

Most of the ways in which measurements misfire are not down to dishonesty.

People act as a result of myriad - often conflicting - motivations. Measuring some aspect of performance alters that web of motivation, and it's only natural that people most feel the pressure of the measurement itself rather than the underlying intent. As a result, people tend to follow the line of least resistance, even if the results are clearly undesirable.

And, given how difficult it is to devise a truly good measure in the first place, bad things happen more often than any of us would dare to imagine.

14 comments:

not displayed said...

One of the main reasons I hated my last job was the sales targets. Each evening we would have to declare to the manager our results for the day.
You could flog yourself all day and just not get a sale while another would fluke several with little effort. This form of measuring effort ended up being very demoralising.

Jean Davis said...

Good points. While it's good to have a goal, holding ones self to standardized measurements only leads to unhappiness and disappointment. Ugh, and the political popularity show, don't get me started.

Pearson Report said...

I had an experience today where measuring the ability of the person sent to deal with a parking problem showed how little stock is put into the involvement of emotions and how quickly they can change any preconceived measurement one may have had of said person.

Good post.

Botanist said...

Mynx, sales targets, yeah, that's an example of organizational behavior that then imposes pressures on people. Depending on how forceful the company is with punishments and rewards - to "motivate" the work force - that can easily lead to nasty and unwanted behavior on the part of otherwise decent human beings.

Jean, maybe I'm just getting cynical in my old age, but I'm seeing modern politics as the biggest failing of the democratic ideal :(

Jenny, when measurement comes in, emotions and plain old common sense tend to fly out the window.

unikorna said...

I am endowed with a significant dose of self irony, which I hold dear :). That is why I never take myself too seriously...so whoever might measure my abilities in one way or another...would eventually discover, that the object of their measurement is entirely uninterested. But I am very interested in the good opinion of those I admire and respect.

Botanist said...

Glad to hear it, Unikorna :) A lot of people in the world could do with taking themselves rather less seriously!

Jennifer Hillier said...

This is a great essay (it's too good to just be a "post"!) and I agree with you. It's one of the reasons I don't miss my old day job - it was all a numbers game and nobody was ever satisfied.

Shell Flower said...

Ah,this speaks on so many levels. First I think of work and the ridiculousness of some of the benchmarks designed for robots, I mean employees. Seriously, it's all about cheating the system just for entertainment, not because you can't meet the guidelines, I swear. Kind of like in quantum physics, the observed is affected by the observation.

Botanist said...

You are too kind, Jennifer. More posts to come, when I get around to forming my thoughts. I reckon there is hope, but not until we work out how to tame the corporate beasts.

Shell, I had not considered the quantum analogy - and there's me an avid science follower too! - so thanks for that insight. I think it is entirely apt!

Susan Bodendo/Super Earthling said...

As I read this post I thought about how many blogs I’ve visited that are similar in nature. The difference is, the majority of their posts have me nodding off before I’m even half way through, while yours keep me captivated.

You’ve made some interesting and valid points here (as in your other posts), which lead me to believe you’ve got an innate understanding about human nature and how and why we act as we do.

I hope you’re thinking about writing a book someday because you most definitely have the skill and the ability to hold readers’ attention! :)

Lydia Kang said...

In healthcare, they've taken the "art" out of medicine and replaced it with a ruler. Not good.

Wonderful essay. (I agree with Jennifer--this is beyond a post!)

Botanist said...

Susan, that is high praise indeed from someone whose blog is the epitome of inventiveness and captivation! I am indeed writing books - nothing published yet :)

Thank you also, Lydia. I didn't set out to write an essay - the very word brings back cold chills, I gave up essays back in school :) However, I'm starting to get worried that many comments bemoan the horrors of the "day job". This is sad, seeing how much of our time we spend there. There has to be a better way!

Teresa Cypher said...

Really interesting post, Botanist! A lot of the things I never before gave a thought. People are inherently competitive, and I don't think there's any way around it. Thus, we keep score. Most of us probably barely notice it. Some people are blatantly obvious about it, though. It can leave a person with a sense of inadequacy when they don't as you put it, "measure up". I'll be back to read the next installment.

It looks different here. You've changed your picture up top! Looks good. How is life?

And...I must be one of the very fortunate ones--I'm blessed with a day job that I really like. But I've had a few that I didn't. :-)

Botanist said...

Hi Teresa, yes, I did change the picture at the top. I do that from time to time just to see if anyone notices :)

Related Posts Plugin for WordPress, Blogger...