It is a mistake to confuse methodology with objectivity

People use a variety of evaluative processes in business to ensure objectivity. But the dangerous fact is that these evaluative processes, these methodologies, even the most rigorous, are woefully insufficient to guarantee objectivity.

Why? There are three basic dangers in relying on method to ensure objectivity.

1. The normative dangers

The overarching danger attaching to evaluative models is the belief that we can use methodology as a legitimating device, even if our evaluative methodology has flaws. Furthermore, if any kind of injustice has resulted from the application of a methodology how can it even be called an injustice, if the "test" for justice is the methodology?

2. The epistemic dangers

I use staffing as a paradigmatic example because of its situation, which cross-cuts the human and the operational domains of management.

To ensure transparency and effectiveness, it is assumed that a method which permits the definition, measurement and testing of people's "competencies" will do the job.  First, who guarantees that the definition of competencies is appropriate? What sanctions the applicable criterion?  It depends on the motive - are the criteria sanctioned in terms of the business needs that the hire will meet or the efficiencies that the hiring process will meet? (If you have to think more than 5 seconds on that one, you're in trouble.)

Second, competency is not a feature of potential hires like the number of doors on a car. People are living, learning beings with potential and we can't screen for traits we haven't identified but might have found very valuable.

Competencies mean little without relationship to motivation to develop them further and opportunities to use them.

We have a collective Western habit of freeze-framing states and confusing the ability to label the frame with the ability to recognize the meaningful stuff in phenomena. Nothing living stands still, including the level of the abilities of your potential staff.

3. The ontological dangers

Empiricism is at pains to distinguish itself from teleology - the identification of ends. The underlying belief at stake is the idea that a grand purpose or design to everything is an anthropomorphic projection. This may or may not have clear import in certain areas of scientific research, but to carry that principle of wariness over into the types of planning whose purpose is to improve quality of life is ludicrous.

For example, in staffing (the most clear examples come from that domain, as I explained earlier), operationally defining merit criteria in terms of a screening methodology puts the methodological cart before the ontological horse. A competency is not an exhausively definable capacity, but a set of potentials that evolves over time. Efforts should be made to ensure fairness and objectivity, and to ensure thereby that the right person is selected for the job (i.e., the "telos" or end is that the essential criteria are connected to the needs of the job and the person's fitness for the role overall is assessed according to those ends). This connection is lost when the specific criteria become disconnected and itemized, and especially when they become too narrowly defined due to a different objective - the objective to manage the "empirical" screening process efficiently. Of course, divorced from its ends, efficiency has no meaning. This is neither fair nor effective.

I may have said this before, but it is clear that no matter how much “empirical evidence” we have, in the end it is our capacity for meaningful valuation that allows us to identify worthwhile goals and our capacity for reasoning that allows us to collect the relevant evidence in support of those and assess it. These two capacities are not separate, but deeply connected in us and contribute equally to the quality of our lives. My examples tend to be drawn from staffing because they are the most straightforward, but the underlying principles could apply to many areas public admin.

Going against the grain of the contemporary Zeitgeist, to ensure objectivity, and by extension, fairness,  in people or program evaluation, we should be sure to use our capacities for meaningful valuation and reason to the best of our abilities.

On Risk Aversion

Is risk aversion a fear of auditors or bad press...or is it denial of the natural transformation that is occurring now towards models of complexity and resilience? It seems we prefer the illusion of control, delivered by models that are
- analytical,
- artificial,
- compartmentalized,
- categorical
- linear,
- mechanical, and
- simplistic.

To be able to achieve manageable simplicity, we would have to be able to identify and control the context, the boundaries, the parameters in which we operate. It is no longer possible to imagine that that we can. The flurry of performance measurement gone mad is probably a reaction formation. Facile metrics are a tragic and total waste of resources, and that's one of the few simple things we do really all know.

The Logic of Quantophrenia

Here's a crash course on the reasonable and unreasonable (quantophrenic) use of indicators (business metrics):

Reasonable:

Given a reliable indicator (sign) of a concrete problem, there is a concrete problem.
If the problem is resolved concretely, the indicators should represent (track/signal) that.

I>P (If I then P)
Therefore,
~P > ~I (If not P then not I)

This is a logically valid form of reasoning. It spurs committed people to remedy the concrete problem.

A simple example for illustrative purposes: Imagine that an unexpectedly high number of employees have been on long term acting assignments in different parts of your organization. What does that indicate? Let's say you do some research and your investigation reveals that, although there are a number of bona fide developmental activities occurring, the problem is due to cumbersome staffing processes and a weakness in integrating business and HR planning, making assignments a quick remedy. So what do you do? You reason that if you streamline staffing and pay more attention to resourcing with respect to projected business requirements, over time, your numbers should diminish to levels that would be expected given developmental activities.

Quantophrenic version:

Given some accurate indicator (sign) of a concrete problem, there is a concrete problem.
If the indicators have changed for the better, the problem has been concretely resolved.

I>P
Therefore,
~I > ~P

IS NOT a valid form of reasoning (it commits what logicians call "the fallacy of denying the antecedent"), and in itself a sign of a problem; i.e., tick box management.

To illustrate, using the previous example, an unexpectedly high number of employees on long term acting assignments indicates problems with cumbersome staffing processes and a weakness in integrating business and HR planning. So what do you do? To meet your indicator target, you remove everyone from their acting assignments. Obviously, you have met your indicator target 100%, but you have not solved the concrete problem, in fact you have quite clearly exacerbated it.

This is a very simplistic example, but it illustrates the potential dangers of metrics as business drivers.

Question: what drives quantophrenia?