“Navigation By Judgement” review
For a few years now my simple dream of supporting people by giving to charities has been in a dormant, squashed position because I cannot stop imagining the following scenario. A IDO (international development organisation) sets up a hospital in a poor country. It’s an incredible hospital; many locals are served and given life-saving medical treatment. The only problem is, it drains the local hospital (which already existed) of highly qualified local staff since the pay is better at the IDO’s health system. Oh, and also the capacity of the local government has not been increased at all during this initiative, such that if funding were to dry up for this new hospital, the government may not have the ability to keep this new hospital alive. Unfortunately, it turns out that a close analogy to this situation exists in reality. A USAID (United States Agency for International Development) project in Liberia almost fit the description above - simply replace “hospital” with “health system”. The building of this health system is one of the projects discussed in Dan Honig’s book “Navigation by Judgement”. When I reached this case study I was shocked to find how closely it matched the scenario I had been thinking about for years, and that it was described so baldly - as if this sort of thing is not uncommon. In this review we will get to the main points of this excellent book, but I will focus on this health system case study for purely personal reasons. This type of scenario has so fully undermined my confidence in aid that I’ve had to read quite a few books to recover myself; and there is no reason to believe this sort of thing has been stopped.
Why is this health system case study so troubling? The initial problem is that the project itself had government capacity building as one of its goals - so the project was ineffective by its own definition. However, the broader question is - why should I be so concerned that this particular goal was not achieved, if a health system was built that helps sick people? As Honig points out later in the book, for quite a while now country ownership has been considered to be important for aid sustainability. If a government and country can internalise change, this can help them improve in the long term. “Will country Z’s net position be better in ten years?” is a question that every IDO and corresponding donor country government should ask; but the implications of the question are often neglected. The most troubling feature of the story for me is that the health system was effectively in competition with the local health system. In addition, not only the results but also the methods of the USAID project were problematic. USAID rejected the chance to work with what the book portrayed as a capable and committed government (led by Nobel Peace Prize winning president Ellen Johnson Sirleaf), choosing instead to build a parallel system and work in parallel to the government. If USAID staffers had insight into how to create a successful institution, none of this insight was shared with the policy-makers who might have benefited from this incidental knowledge and gone on to transfer it to other government projects.
How did it happen? USAID’s employees are not out to sabotage the Liberian people; they actually attempted to involve the Liberian government “to the extent the project’s structure allowed.” Unfortunately, the project’s structure prohibited them from doing so to any great extent - and this is not a problem unique to the project at hand. Most USAID projects are set up in this way because the “authorising environment” of USAID (namely US Congress) operationalises a profound distrust of USAID by comprehensively restricting their actions. Honig carried out interviews with around a hundred and forty people who were involved in aid projects somewhere along the chain of influence from donor country governments, to IDOs, to field agents. One interviewee describes USAID as being “under siege” for the last thirty years of US government! As an organisation that simply is not trusted, USAID must ensure that projects undergo rigorous quantitative reporting for each project, and that the projects are carried out exactly according to directives from Congress. Moving to the present case study, we find that the project’s rigid blueprint could not tolerate any deviation from the plan. Therefore for USAID to collaborate with the Liberian government in any meaningful sense was not an option, because it might have disrupted the operations signed off on, and therefore required by, Congress. One might ask why Congress does not investigate the accumulating failures of many such projects. The case study can shed light on one dimension of this. As per most USAID projects, rigorous quantitative measures were used, and a lot of data was collected. This gave the false impression that the project was a thorough success. According to these figures, USAID achieved what they set out to do: train x people, host y women giving birth in health facilities, and publish z policies. However, these measurements masked the fact that government capacity was not improved, and the project was not sustainable. The short-sightedness of these measures is something widely acknowledged - even by employees - however, they are constrained to this way of working. They are incentivised to achieve the quantitative measures no matter what, therefore having less time to learn about how to best achieve the goals of the project (which are broader than these targets suggest). This is a distortion of best practice that cripples the project and USAID more broadly - once again, enforced by Congress.
What would a better pathway look like? According to the book and to a paper Honig wrote after his book tour, IDOs should pull every lever within reach to choose a “navigation strategy” appropriate for the situation. There is a spectrum between “navigating by judgement” and “navigating from the top” from which to choose. USAID is almost always restricted to the method of tight, top down control, or “navigating from the top.” This is not always a bad thing; in fact, it can be helpful in avoiding corruption and mistakes on the part of field agents and contractors. It is most likely the best strategy when projects fit into the following categories:
Verifiable - can be accurately assessed using quantitative data (outputs or KPIs), and thus easily controlled from afar. If it can be (accurately) measured it can be managed.
The political/social/economic environment is predictable - there are likely to be minimal unpredictable events or shocks, and the environment is “legible” (i.e. understandable for someone who is not there; as opposed to being controlled by de facto power relationships or requiring “soft information” or knowledge that can’t be expressed in figures or even words. In other words if “you had to be there” the environment is not legible.)
The LESS verifiable a project is and the LESS predictable the environment is, the LESS navigation from the top will work. It will be too difficult to assess and too inflexible to deal with shocks. In these situations “navigation by judgment”, relying on field agents’ judgements for contextual decision making, becomes increasingly necessary since it allows the organisation to flexibly respond to complex and unpredictable contexts. Although this opens up the possibility of the aforementioned risks (corruption and mistakes), the complexity of unforseen events and uncertain environments can quickly reach a level which necessitates flexible decision making on the ground - and therefore navigation by judgement is worth the risk. The relationships just discussed are empirically supported in the book using a large data set - unverifiable projects in uncertain environments are more successful if navigation by judgement is used.
How could this have applied to the health system case study? The study demonstrates what can happen when the reputation of the organisation is not tarnished with mistakes or corruption - a shiny new system is built that perfectly fulfils the brief. However, the problem that overshadows all aforementioned risks is that the brief itself is deeply misguided! On the ground, despite a well intentioned Liberian government, Liberia was a post-civil-war country. This uncertain environment interfered with USAID’s ideal plan to build the capacity of the government while constructing its excellent health system. Could USAID have done things differently? Perhaps Congress will change the way it operates to allow USAID to use more navigation by judgement. Many other IDOs have chosen this option. The book looks at four case studies with USAID projects, and four with projects run by DFID (the UK’s Department for International Development). In these case studies, DFID demonstrated a tendency towards navigation by judgement - their default navigation strategy. This allowed field agents to work alongside recipient country government policy-makers, discuss the project goals and strategy, revise them if necessary, and amidst all of these conversations, strengthen recipient government capacity. At around the same time as USAID built this health system, DFID undertook a project that was widely praised by interviewees. This project was credited by the Liberian president as the reason why the health department made more progress than any other department during that time period.
If DFID demonstrated a navigation strategy that was helpful most of the time, it was likely because of an authorising environment (Parliament) that valued and trusted the ability of DFID to navigate by judgement. The most direct pathway towards better aid projects is to change the authorising environments of IDOs. Ostensibly, donor country governments are accountable to their constituents, and it is conceivable that lobby groups already exist to encourage better foreign aid work. In assessing the difficulty of this task, we might benefit from comparing flexibility for different navigation strategies to that of another useful change; for example, taking politics of recipient countries into account when designing interventions. Similarly to the question of the extent to which we engage with the politics of recipient countries, donor countries have an influence on IDO navigation strategy. As the author points out, however, the advantage of changing IDO navigation strategy over, for example, challenging the status quo of recipient country politics is that the former might be an easier task in the long term - because changing donor country politics is easier than changing both donor and recipient country politics. One significant obstacle here is that donor country governments may feel as if we are throwing away efficiency and accountability in favour of a completely unmoored laissez-faire approach. In practice, this is not true. Navigation by judgement may entail changing how accountability is conceptualised to allow more flexibility for field agents and contractors; for example:
Descriptive instead of primarily quantitative reporting
A different frequency of reporting to HQ to preserve focus on quality of work
Working with contractors instead of handing over to contractors
Accountability to mid-level authorities in the field rather than top-level authorities abroad
Peer accountability relying on reputational impact of work quality
Flexible discursive accountability rather than static, reports-based accountability.
The nail in the coffin for USAID-style accountability (pre-set quantitative targets) in uncertain projects is that project staff are often being held accountable to achieve something counter-productive! In contrast, the methods above all allow iterative accountability to targets that can achieve the broader goals of the intervention - necessary amidst unforeseen events, political manoeuvring, contextual knowledge, and the thousands of tiny, inarticulable decisions which field staff must make.
As many have pointed out, we cannot keep designing projects requiring rigid top-down control. We cannot keep competing with or ignoring recipient country governments, designing irrelevant interventions, or solely assessing performance with a litany of distortionary quantitative measures. I am grateful to this book for clearly conceptualising and communicating these concerns. Honig expertly balances information overload with rigorous and careful argumentation based on both qualitative and quantitative evidence, and as such this book is worth reading for anyone interested in improving current development work.
