There's no Accounting for It: Areas where the quest for the true story leads us to falsehood

We have confidence in numbers, not the evidence of our senses, as is often claimed. Everything we consider to be accurately observed is, in fact, accurately defined - i.e., defined in a way that makes the phenomenon of interest measurable, countable. Not until we establish the numeric lens of accuracy, or the illusion thereof, do we consider phenomena to exist beyond our thinking, our interestedness. Pause and ponder that.

You see, our confidence is misplaced. We mistake the indicator for reality. The relevance of data resides in the definition (a.k.a. "operational definition") of what is counted, and rarely are such definitions exclusive and exhaustive of phenomena, outside of some narrow scientific disciplines. 

Speaking from his experience in the area of Psychology, Jack Martin of Simon Fraser University explains that, whereas operational definitions provide investigators with initial cues to the identification and more thorough understanding of phenomena, psychologists frequently treat such definitions as if they were conclusive and exhaustive. Complex phenomena, such as human motivation and confidence, are narrowly understood in terms of a small set of predetermined factors. Understanding becomes reduced to some kind of criteriology, a labelling game. The purposes of investigation, learning and discovery, making new and deeper interconnections, are left out of this paint-by-numbers story.

From the psychology lab to the office, the situation becomes culturally entrenched. Those who report, from students to office workers, spend their days manipulating and their nights worrying about institutionalized performance measures, standardized tests, and a daily barrage of transactional data, all narrowly defined and often rather arbitrary, devoid of much concrete relevance. These measures are meant to provide insight into trends, and evidence of this or that performance, but in fact they mostly just

• eliminate context, 
• block connections, common sense and insight
• erode our capacity for reasoning 
• keep us too preoccupied to examine our purposes
• and ultimately put us on the self-sustaining hamster wheel of empty bureaucratic process

The focus on numbers may give us the secure sense that we're getting precise information, but it's a false sense of security.

See also The Logic of Quantophrenia and possibly Amy Lemay on assessing impact as reported by Asha Law at casrai.org.