Failure Report 1.0
As I wrote earlier this month, PDT will be issuing an organization-wide failure report early next year along the lines of those provided by Engineers Without Borders. We are still refining the guidelines. Inspired by Cornell West, who asked “Yes it’s failure, but how good a failure?, we want a template that focuses on learning from these mistakes. So, in the spirit of failing well, I’ve chosen to draft our first report from the perspective of PDT’s management.
Please note this important disclaimer: The failures listed in this report don’t belong to the PDT staff. They belong to me and Jennifer and the rest of the leadership at HQ. Our people in the field do amazing work under difficult and often dangerous conditions. Each of the shortcomings listed is due to decisions or direction that came from the people at the top of the org chart.
- Our biggest accolades come for our Peace Dividend Marketplace projects that have redirected over $600m into the economies of Afghanistan, Haiti, and Timor. There are two problems with this. First, we really don’t know how much of that would have been spent in the local economies without our intervention. In some cases, we have not redirected, but accelerated international spending. For example, the donors would have spent several months trying to find a local vendor, but we helped them find it in a couple of weeks. Both outcomes are economically beneficial, but without being able to accurately distinguish the two it is very hard to measure the actual impact of these projects. This isn’t good enough. Going forward, we need to implement a new component to our results tracking systems that also looks at the nature of the contracts being tendered.
- The second problem with our Marketplace projects is that we track the gross economic impact of foreign spending, instead of the net impact. As you all know, $1m of donor money spent buying Chinese desks from a Kabul vendor is not the same as $1m spent in Helmand buying vegetables from a local farmer. Frankly, the latter will probably have a bigger economic impact than the former. It will create more jobs, and possibly even greater tax revenue and GDP growth. But we don’t have a grip on how to accurately analyze the direct effects of increased local spending, and the gross dollar figure approach is too crude. It distorts how we measure the true economic benefit of our activities, and it leads to prioritizing large dollar value spending that may have very low local benefit, and it entirely ignores job creation. Furthermore, we definitely can’t predict impact in advance that would allow us to efficiently focus the project activities. We need to be far better at this. With the expansion of our Economist team, Jennifer and I have asked that we develop a methodology for this analysis and that it be rolled out across our various country programs.
- Related to 1 & 2 above, we don’t do nearly enough impact measurement. It would be easy to blame the donors here, since for the most part they only care that we measure process: Budget performance, number of training sessions held, etc. But PDT believes passionately in enhancing impact through better data. We need to focus more of our resources on this. If we did, we could show that some of our projects are indeed producing remarkable results in terms of poverty reduction or job creation, but that others are simply not worth the effort (and then cut those projects to work elsewhere.) For new projects, we will write into the contribution agreements impact measurement activities and clauses to ensure that the project will be primarily judged on those metrics.
- Perhaps one of our biggest failures is our fundraising strategy. We have grown far too dependent on government donor agencies. This has slowed us down and made us more process driven. Our “unrestricted funding”, the money we can spend on finding and testing new ideas, is tiny. We should have been focusing on this far more, far earlier. With all the good press we’ve gotten, I should have been working far harder on turning that into financial support. In reality, we only started looking for private funders in the last 12 months! In the first quarter of 2011 we will be hiring a Development Director who will work directly with me to review and overhaul our funding strategy and activities.
- We’ve ignored the lessons learned by others. A lot of the “new ideas” that PDT develops aren’t actually new. Someone else in the aid industry thought of it before and tried it elsewhere. We need to put far more energy into ferreting this out. It will allow us to a) give credit where credit is due, b) focus our precious resources on genuinely new ideas, and c) build on the experience of others to fine tune or further innovate these concepts. How can we fix this? Partly we need to be more curious and more energetic in our research. Jennifer and I may also put up a big sign in the New York bullpen that reads “Great idea. Now find out who thought of it first.”
- We have wasted far too much time (and readily available donor money), measuring the “economic impact of aid”. That work was important and useful and groundbreaking 5 years ago when the Economic Impact of Peacekeeping initiative showed that only a small fraction of international spending was going into the local economy. But now, it’s not new. It’s not surprising. And it’s very safe to extrapolate the data collected in Liberia in 2007 to estimate the donor footprint in country x in 2010. But the donors keep offering to pay for another study, so we keep doing it. We need to stop. Aid is rarely made more effective by the production of yet another study. We’ve already cut some of these projects out of our pipeline. Going forward, we will ensure that new economic analysis projects clearly move the research forward in obvious ways.
- Our staffing systems are really lagging. To be fair, we’ve doubled in size almost every year over the last 6 years. This creates strains on our administrative processes. But still, if we want to keep our good people, and find more people like them, we need to be faster, smarter, and better on how we look after the PDT team. (This isn’t the fault of our HR team. They’ve been crying for more resources to do this, and I promise I’ll find them!) In the new year we will look at some new support structures for field teams and budget for additional resources in the HR unit.
- PDT’s communications efforts are terrible, I mean truly painfully awful. Have you seen our website? Or noticed that we have multiple websites? Our print material is even worse. We’ve spent so much time focused on the “doing” senior leadership (i.e. me) has totally forgotten about the “telling and advocating”. Our project staff are understandably frustrated by this, and even more frustrated because we keep promising to fix this. They need not only a plan, but a clear message, and assistance from HQ on how to deliver it. The fix for this can’t be summarized in a sentence. Early in the new year our new Communications Director will be circulating a detailed plan for not just fixing this problem but for implementing an aggressively innovative new approach for comms.
- We are too quick to criticize others. For example, just yesterday I took a cheap shot at Bob Geldoff in a newspaper interview (sorry Bob, we love you). I agree that some folks in the aid industry deserve all the criticism we can shovel on them. But in more cases there are complex and intractable reasons why projects, NGOs, and donors behave the way they do. We need recognize this, and be far less eager to blame the entities, but rather we should blame the industry wide conditions and norms that not only tolerate failure, but in some cases encourage it. To be frank, I’m not sure if we’re going to get better at this. But, perhaps, if we are quick to criticize ourselves it won’t seem quite so bad.
- Unbelievably, we have almost no Knowledge Management system. We have a shared intranet, where important documents are stored, but nothing that actually functions well. While we rail about the importance of the aid industry and the UN capturing what Donald Rumsfeld called the “known knowns”, we have failed to systematically capture our own knowledge. This makes it especially hard on our project staff who often are forced to “figure it out” without the benefit of knowing that another PDT staffer did exactly that, in the same project, 2 years earlier. Jennifer and I will be talking to you, our IT team, and others to first determine the scope of our needs, and then to propose some solutions.
- Related to the previous failure, PDT has been embarrassingly hypocritical in demanding reforms in the aid industry that we haven’t implemented ourselves. Here are just two examples. We’ve argued for years that the aid industry should bring in performance bonuses to reward excellence and encourage innovation. We also have tried to convince the UN to offer more staff recognition awards to highlight the best practices of their field staff. But in both cases, PDT has planned but failed to roll out similar programs internally. And refer back to items 1 through 3 above. We demand more data and analysis, but we don’t do enough of it ourselves. How do we fix this? By being more self-aware. By reminding ourselves that as we grow, we will draw the same scrutiny we apply to others. And by using that scrutiny as an opportunity to not only advocate for innovation, but also demonstrate innovation.
We welcome your comments and look forward to working with you to fix these failures in the year ahead.