There's no Accounting for It: Areas where the quest for the true story leads us to falsehood

We have confidence in numbers, not the evidence of our senses, as is often claimed. Everything we consider to be accurately observed is, in fact, accurately defined - i.e., defined in a way that makes the phenomenon of interest measurable, countable. Not until we establish the numeric lens of accuracy, or the illusion thereof, do we consider phenomena to exist beyond our thinking, our interestedness. Pause and ponder that.

You see, our confidence is misplaced. We mistake the indicator for reality. The relevance of data resides in the definition (a.k.a. "operational definition") of what is counted, and rarely are such definitions exclusive and exhaustive of phenomena, outside of some narrow scientific disciplines. 

Speaking from his experience in the area of Psychology, Jack Martin of Simon Fraser University explains that, whereas operational definitions provide investigators with initial cues to the identification and more thorough understanding of phenomena, psychologists frequently treat such definitions as if they were conclusive and exhaustive. Complex phenomena, such as human motivation and confidence, are narrowly understood in terms of a small set of predetermined factors. Understanding becomes reduced to some kind of criteriology, a labelling game. The purposes of investigation, learning and discovery, making new and deeper interconnections, are left out of this paint-by-numbers story.

From the psychology lab to the office, the situation becomes culturally entrenched. Those who report, from students to office workers, spend their days manipulating and their nights worrying about institutionalized performance measures, standardized tests, and a daily barrage of transactional data, all narrowly defined and often rather arbitrary, devoid of much concrete relevance. These measures are meant to provide insight into trends, and evidence of this or that performance, but in fact they mostly just

• eliminate context, 
• block connections, common sense and insight
• erode our capacity for reasoning 
• keep us too preoccupied to examine our purposes
• and ultimately put us on the self-sustaining hamster wheel of empty bureaucratic process

The focus on numbers may give us the secure sense that we're getting precise information, but it's a false sense of security.

See also The Logic of Quantophrenia and possibly Amy Lemay on assessing impact as reported by Asha Law at casrai.org.

The Relationship between Objectivity and Bureaucracy

In 2009, I posted a short blog entry, How Organizations Remain Bureaucratic. Here, I'd like to delve a bit further into what is pejoratively known as "bureaucracy" and tie the discussion to a broader cultural theme, one I'll call Fetishizing the Heuristic. The terminology is a bit academic, but apt, so I'll explain.

Heuristics are flexible guidelines, generally sane models, not meant to account for every contingency, but a pencilling in of a plan or set of criteria based on experience. (Maslow's Hierarchy, for example, was intended to be considered a flexible heuristic, but rarely is.)

To fetishize something means to idolize it...

So when a heuristic is idolized, the criteria embodied in a flexible guideline magically come to have a life of their own, alienated from the creative and reasonable minds that set them out. They become viewed as if they issue from a higher order of existence.

So, for example, a preliminary plan is developed that covers a time span, or evaluative criteria are set out provisionally. As soon as dates and scoring methods are assigned, it's as if these plans or criteria issued directly from an eternal source of objectivity, not from the minds of those composing them. (They become reified in processes, detached from the aim that inspired their development.)

The problem that arises is that once the guideline has been drawn up and numbers assigned to what started off as a provisional guideline, no counter-evidence from the actual world of objective reality will change the guideline. Where there is deviation, it will considered as a way of illustrating how things and people have failed to meet the criteria, not as evidence that the model needs adjusting.

Jack Martin of Simon Fraser University wrote in 1996 (in a no longer available newsletter of the American Psychological Association) lamenting how assessment criteria, never intended to be more than reasonable, flexible, useful guidelines, become irrationally narrowed and fixed once they are operationalized (assigned a numerical scale). This causes the quality and relevance of research in empirical psychology to degenerate.

Then, of course, these guidelines become embodied in institutional procedures - further alienated from the agents, the creative and reasonable minds of people in the organization, who would otherwise use their minds to question them, adjust them, and maintain their usefulness.

In practice, understanding how we can fetishize the heuristic should lead us to be more active in questioning and revising plans and criteria as necessary, understanding they issue from human needs and objectives.

The broader, more philosophical understanding is that a lot of the conceptual packaging we use to identify something as objective is faulty and needs to be revisited. We need to re-insert the awareness of our responsibility as agents into our plans and to adjust in an ongoing way to what is happening, rather than mechanizing thought in bureaucratic processes or bracketing ourselves out of existence while we follow an eternal diagram (our role in conceiving it, forgotten).

In a nutshell, don't treat a working hypothesis, guideline, process or model as if it issued from the heavens. After all, it's your model and you should have every right to correct it as more evidence becomes available - and there will always be more evidence, unless you are omniscient. 

It is a mistake to confuse methodology with objectivity

People use a variety of evaluative processes in business to ensure objectivity. But the dangerous fact is that these evaluative processes, these methodologies, even the most rigorous, are woefully insufficient to guarantee objectivity.

Why? There are three basic dangers in relying on method to ensure objectivity.

1. The normative dangers

The overarching danger attaching to evaluative models is the belief that we can use methodology as a legitimating device, even if our evaluative methodology has flaws. Furthermore, if any kind of injustice has resulted from the application of a methodology how can it even be called an injustice, if the "test" for justice is the methodology?

2. The epistemic dangers

I use staffing as a paradigmatic example because of its situation, which cross-cuts the human and the operational domains of management.

To ensure transparency and effectiveness, it is assumed that a method which permits the definition, measurement and testing of people's "competencies" will do the job.  First, who guarantees that the definition of competencies is appropriate? What sanctions the applicable criterion?  It depends on the motive - are the criteria sanctioned in terms of the business needs that the hire will meet or the efficiencies that the hiring process will meet? (If you have to think more than 5 seconds on that one, you're in trouble.)

Second, competency is not a feature of potential hires like the number of doors on a car. People are living, learning beings with potential and we can't screen for traits we haven't identified but might have found very valuable.

Competencies mean little without relationship to motivation to develop them further and opportunities to use them.

We have a collective Western habit of freeze-framing states and confusing the ability to label the frame with the ability to recognize the meaningful stuff in phenomena. Nothing living stands still, including the level of the abilities of your potential staff.

3. The ontological dangers

Empiricism is at pains to distinguish itself from teleology - the identification of ends. The underlying belief at stake is the idea that a grand purpose or design to everything is an anthropomorphic projection. This may or may not have clear import in certain areas of scientific research, but to carry that principle of wariness over into the types of planning whose purpose is to improve quality of life is ludicrous.

For example, in staffing (the most clear examples come from that domain, as I explained earlier), operationally defining merit criteria in terms of a screening methodology puts the methodological cart before the ontological horse. A competency is not an exhausively definable capacity, but a set of potentials that evolves over time. Efforts should be made to ensure fairness and objectivity, and to ensure thereby that the right person is selected for the job (i.e., the "telos" or end is that the essential criteria are connected to the needs of the job and the person's fitness for the role overall is assessed according to those ends). This connection is lost when the specific criteria become disconnected and itemized, and especially when they become too narrowly defined due to a different objective - the objective to manage the "empirical" screening process efficiently. Of course, divorced from its ends, efficiency has no meaning. This is neither fair nor effective.

I may have said this before, but it is clear that no matter how much “empirical evidence” we have, in the end it is our capacity for meaningful valuation that allows us to identify worthwhile goals and our capacity for reasoning that allows us to collect the relevant evidence in support of those and assess it. These two capacities are not separate, but deeply connected in us and contribute equally to the quality of our lives. My examples tend to be drawn from staffing because they are the most straightforward, but the underlying principles could apply to many areas public admin.

Going against the grain of the contemporary Zeitgeist, to ensure objectivity, and by extension, fairness,  in people or program evaluation, we should be sure to use our capacities for meaningful valuation and reason to the best of our abilities.

Overcoming Institutional Inertia

Change is difficult in large institutions.  In a previous post, I wrote:

I'm not even sure that it's that "people prefer the devil they know," as the old saying has it. It's just that it is dangerous to both the authors of the narrative and those subject to it to question the rules, to disturb the "collective illusion" that sustains the power structures. For better or for worse, power structures represent "the very fabric of society" and resistance to change should not be underestimated. 

There I argued that it would be technology that would inevitably drive changes. I was thinking, for example, that the Web 2.0 phenomenon that has emerged in recent years presents a platform for horizontal, collaborative, collegial and networked dialogue.  

But to get on board with these technologies to solve work related problems and issues, your heart just has to be in it. You have to care enough about doing things well, or creating an environment where people can be more effective, that you're willing to share your ideas and have them peer reviewed. Before you can do that, you have to be able to identify where changes and improvements are needed, and where people can make a difference.  But even before that, you need the hope that your individual or collective efforts can make a difference.  You have to believe that making a difference is possible.  


Employees can sidestep these risks by focusing on processes and tasks, rather than the broader purposes and the overall efficacy of these activities.   Managers can sidestep institutional roadblocks by behaving "Quantophrenically" or they can simply take roadblocks as natural and eternal aspects of the managerial landscape. In so doing, employees and managers contribute passively to institutional inertia.


These approaches are either products or results of cynicism, which is unfortunate.  Where does the root of the cynicism reside?  How can the tide be turned, now that we have such amazing opportunities to make changes?  


This may be much more about cultural ideology than about individuals. IMO, there has been massive oversimplification with rationalizing and standardizing over the history of business culture in general, and so much of a focus on means and methods (e.g., money, processes, perpetual "growth" in economic terms).  This means that we are cogs in a big economic machine that is, as Charles Taylor put it, "all dressed up with nowhere to go."  In order to make the changes we need now, we need to refocus on things that are intrinsically valuable (growing and sustaining mental, emotional and ecological well-being). The focus on means, methods and quantophrenics has in the past eclipsed discussion on the more intrinsically valuable objectives. 


I do think the tide is turning, however, and that a newer, richer understanding of the complexity of our natural and social systems is causing our more human values to surface, and new voices will increasingly be heard. 

The Logic of Quantophrenia

Here's a crash course on the reasonable and unreasonable (quantophrenic) use of indicators (business metrics):

Reasonable:

Given a reliable indicator (sign) of a concrete problem, there is a concrete problem.
If the problem is resolved concretely, the indicators should represent (track/signal) that.

I>P (If I then P)
Therefore,
~P > ~I (If not P then not I)

This is a logically valid form of reasoning. It spurs committed people to remedy the concrete problem.

A simple example for illustrative purposes: Imagine that an unexpectedly high number of employees have been on long term acting assignments in different parts of your organization. What does that indicate? Let's say you do some research and your investigation reveals that, although there are a number of bona fide developmental activities occurring, the problem is due to cumbersome staffing processes and a weakness in integrating business and HR planning, making assignments a quick remedy. So what do you do? You reason that if you streamline staffing and pay more attention to resourcing with respect to projected business requirements, over time, your numbers should diminish to levels that would be expected given developmental activities.

Quantophrenic version:

Given some accurate indicator (sign) of a concrete problem, there is a concrete problem.
If the indicators have changed for the better, the problem has been concretely resolved.

I>P
Therefore,
~I > ~P

IS NOT a valid form of reasoning (it commits what logicians call "the fallacy of denying the antecedent"), and in itself a sign of a problem; i.e., tick box management.

To illustrate, using the previous example, an unexpectedly high number of employees on long term acting assignments indicates problems with cumbersome staffing processes and a weakness in integrating business and HR planning. So what do you do? To meet your indicator target, you remove everyone from their acting assignments. Obviously, you have met your indicator target 100%, but you have not solved the concrete problem, in fact you have quite clearly exacerbated it.

This is a very simplistic example, but it illustrates the potential dangers of metrics as business drivers.

Question: what drives quantophrenia?

Conceptual Filing Cabinets are not Reality

Ever noticed that when corporate downloads an initiative to remedy an internal management problem, the effects on the ground tend to be the opposite of what was intended? Without going into details, this may explain why. Mintzberg said:

"Strategic planning is not strategic thinking. One is analysis, the other is synthesis"*

The fact is, the corporate structure tends to be organized in terms of business lines or branches, so initiatives get forumulated around compartmentalized resources.  That means analytical thinking will predominate in planning - and analytical thinking relies on compartmentalized categories – checklist items. People mistake their conceptual filing cabinets for reality and are rewarded for substituting numbers for facts. But as some astute Brit said not long ago, “You don’t fatten a pig by weighing it.”  The success of mechanistic models of the past has seduced us to seeking "simple" solutions in  complex arenas, much more complex than are tackled by the hard numbers of "hard science." 

Most activities in business are dynamically interconnected and adjustments in one area can be offset by failing to consider their impact on other areas. Thinking strategically involves connecting the dots, lateral thinking, the use of the right brain, which is basically grounded in intention, will, and commitment (caring).

The first thing required for an effective strategy is that you genuinely care.

The second thing is flexibility and openness.

Some folks believe that analytical thinking is a cut above more poetic sensitivities and that it is the sole defender objectivity and of fairness. I'd like you to consider that such a view is rather narrow, sometimes honest but simplistic, other times useful to people with a power-for-power's-sake agenda. Models of evaluation based on analytical thinking are just as prone to manipulation and bias as any poetic kind of perspective. They are also dangerous as one doesn't need brute strength or mass hysteria to convince, but one can just point to "the numbers"as an incantation. This is less likely in very precise areas such as physics, but in business, there are just no exhaustive and exclusive definitions that can be reliably used for evaulative purposes. Our definitions and indicators are handy rules of thumb, never meant to be the be-all and end-all of evaluation. (ISO - I despair.) A planner can pick a few handy items from a potentially infinite list of possible indicators.  Nevertheless, those who employ a definition, a methodology, who fail to think things through broad-mindedly with an honourable meaning, purpose and value, who yet check a list of measures, have the advantage of the appearance of objectivity. This is a pernicious effect of the "trope of science" and needs rooting out.

*link no longer available.