What am I Doing Here Part 4: Monitoring and Evaluation

No matter what you do or where you work, if you are spending someone else’s money, eventually, they are going to ask you what the hell you are doing with it, how effective you have been, and what you have been able to accomplish. This need for accountability and documentation is, however, only one part of the much larger picture usually referred to by the conjunctive phrase: monitoring and evaluation (M&E). Having not worked in the social sciences or studied development in school, my eyes have been opened to this fascinating and convoluted world of M&E, [sometimes also called Planning, Monitoring, Evaluation and Learning (PMEL)], over the past year.

Though the idea of planning your work, making sure it is on track, evaluating the relative success or failure and trying to learn from the experience has been around forever, the M&E requirements of development projects have been turned into an art form. There are elaborate methodologies, old school tricks of the trade, hot new frameworks and cutting edge research papers being published on the topic monthly. Careers are being made and lost trying to prove what works and what does not, NGOs are pouring more of their budgets into checking all the latest and greatest boxes, and every few years there’s a renewed push by the major multilateral agencies and governments to “get M&E right this time”.

Despite all this, there still seems to be an air of general dissatisfaction with our collective ability to find out what works, make sure it happens, and to reproduce it somewhere else. There is also the frustrating realization that we may never be able to measure all the complex and nuanced subtleties that are inherent of change in human systems. Moreover, it may be that these unmeasurable changes are the most powerful and important, yet remain outside of our reach of understanding and reproducibility. As an always on point Albert Einstein noted, “Not everything that can be counted counts, and not everything that counts can be counted.”

einstein-not-everything-that-can-be-counted-counts

There is also a growing feeling, within the development world, that we need to stop looking at issues like diseases and poverty as linear effects stemming from directly linked causes. The so-called “systems approach”, which is the antithesis of this, has been gaining steam over the past decade. In this approach, the sum is greater than the individual parts, the relationships are just as important as the players, and everything is at once linked and constantly evolving. While this is a far more accurate representation of the complex, emergent, human systems found in development, it throws a bit of a wrench into the whole M&E thing. How do you know where you are going when you can’t see where you have been? How do you measure the change you intended to create, when everything is changing all the time? How do you learn from one instance of a system when those exact conditions are likely never to exist again? These issues are crucial across the board in development and social change projects. Within the impact investing world, where BDSA works and I find myself, M&E is where the rubber meets the road and is at the crux of proving the central concept that businesses can be drivers of positive social change.

Monitoring: The systematic & continuous assessment of the progress of a piece of work over time, which checks that things are ‘going to plan’ and enables adjustments to be made in a logical way.

Evaluation: The periodic assessment of the relevance, performance, efficiency, and impact of a piece of work with regards to its stated objectives.

The history of M&E begins several decades back, when the rigour of scientific studies and evidence based results began filtering over to the social sciences and government. Within a few more decades, these ideas had made their way into the development context and had been sufficiently evolved to the point where it became almost impossible to do a project without applying a logical framework (logfram) or Results Based Management (RBM) tool. In the former, the project planner (usually on-high, in an air conditioned office far, far away), lays out the overall intended outcome of the desired change. He or she then works backward to determine the measurable outputs or metrics that would indicate the outcome has been achieved, the activities needed to produce these outputs and finally the inputs needed for each activity. At the end of the project the impact of the work is determined by subtracting the counterfactual (i.e. what would have happened if no intervention had been made) from the outcome.

M&E Impact Chain

This approach works great in the sciences where experiments can be carefully controlled, economies of scale allow for statistically valid results, and where causes can be closely linked with effects. In the real world of development, these conditions are rarely met, and the logfram approach leaves you with a decent planning tool, but a horribly rigid and impractical measurement and change management tool.

Formula from the World Bank's guide on M&E
Sample formula from the World Bank’s guide on M&E

 

So, development project implementers find themselves in a bit of a bind: on one hand, they need rigour and accountability, and on the other they need flexibility and constant adaptation to a rapidly evolving reality. They need to be able to prove their interventions were effective, substantiate the attribution of the results to their donors (i.e. determine the counterfactual), and be relatively confident that these results are reproducible and sustainable (i.e. the project continues on, as designed). This drives an excessive focus on donor accountability; an obsession with control, causation, and attribution; and an overall rigidness and inflexibility that is more of a hindrance than a help. To say nothing of the fact that the beneficiaries/victims of these projects (i.e. the poor) are often cut out completely from the planning, evaluating and learning process and are left scratching their heads when the NGO declares the completion of yet another successful project.

In order to counter some of the obvious short comings of the traditional Results Based Management (RBM) approach, various tools, frameworks and approaches have been introduced over the years. Most of these attempt to capture the qualitative aspects of projects through questionnaires, interviews, stories and through participatory, beneficiary-driven planning and evaluation. They also attempt to design for, or at least acknowledge, the complex, emergent nature of the systems of which they are a part and to weigh the needs of the beneficiaries above those of the donors. And while there is no silver bullet or one-size-fits-all approach that is going to work everywhere, a mix of these different tools and approaches is helping to breakdown the orthodoxy of the purely quantitative result.

Here are a few key trends in M&E that have been getting attention lately and some sources on where to find more information. For a great overview of all these recent trends and others, see this paper.

Developmental Evaluation – This approach is basically an attempt to reduce the feedback cycle between learning, doing and correcting to almost zero. By collecting data in real time and making decisions based on a constant feedback cycle, the theory is that the project can adapt and evolve in conjunction with the system, thereby avoiding the need for major course corrections down the line.

Shared Measurement – In this case, common metrics are used across organizations on “scalable platforms” in order to facilitate the sharing and discussion of results and learnings on a much greater scale. It also helps organizations share responsibility for their data collection and learning.

Big Data – As the name suggests, this approach is based on the assumption that if some data is good, lots and lots of it must be better. Using short feedback cycles, real-time digital data from a variety of sources (such as website traffic, twitter, blogs, phone records, etc.), and data visualizations and infographics, it is hoped macro-trends and insights will emerge.

Problem Driven Iterative Adaptation – The PDIA approach is based on four key principles: First, focusing on solving locally nominated and defined problems in performance (as opposed to transplanting pre- conceived and packaged best practice solutions). Second, it seeks to create an authorizing environment for decision-making that encourages positive deviance and experimentation (as opposed to designing projects and programs and then requiring agents to implement them exactly as designed). Third, it embeds this experimentation in tight feedback loops that facilitate rapid experiential learning. Fourth, it actively engages broad sets of agents to ensure that reforms are viable, legitimate, relevant and supportable.

QUalitative Impact Protocol – Qualitative information is often hard to communicate between stakeholders, even though it provides rich and relevant learning. The QUIP approach is an attempt to get qualitative data taken seriously by collecting it in a systematic and structured way.

Most Significant Change – MSC is a story based approach to help identify the causes of a significant/critical change (positive or negative) relating to key objectives, rather than looking for trends related to a certain phenomenon. This makes it easier to track stories of changes related to less easily quantifiable issues such as “capacity building” or “gender equality”.

Here is a brief look at a few more:

M&E Summary

Even with all of these tools and techniques being simultaneously developed, piloted and perfected. There is still much to be done in changing the development system itself. Here are a few recommendations for the future:

  • First and foremost, there needs to be much greater trust between donors and implementers and a lot more freedom given to experiment, adapt and learn. By far the biggest hurdle standing in the way of creative solutions to poverty reduction is that donors don’t trust implementers with their money, and implementers don’t trust donors with their program designs.
  • Donors , implementers, beneficiaries and other stakeholders need to come together to create spaces for innovation, seed the soil for new ideas, and embrace the failure of some projects in the name of a better overall result.
  • Agreement on the big picture problem definition or mission is necessary between stakeholders. This shared understanding should then serve as the organising principle when adapting activities and plans to ensure that practitioners are beholden to the ultimate mission, rather than the activities themselves.
  • Direct attribution of an impact is neither possible nor desirable in a complex adaptive system. The need for implementers and NGOs to clearly attribute how their work created a specific change should never take precedence over achieving the highest quality and most impactful aggregate change in the overall system. If implementers were able to put as much effort into achieving absolute results, as they do into competing for and seeking credit, everyone would benefit.
  • Finally, we need to give up the obsession with finding a be-all and end-all, silver bullet solution to our M&E needs. There will never be one perfect technique, just lots of little imperfect ones and the goal should be to continuously inch them forward.

So what is this all about, and why should you care? Well, whether you are a rural farmer in Ghana, a student in the UK, or a hospital patient in Canada, your life is probably significantly affected by the type of M&E performed by the organization with whom you are interacting. It may be that your story is being left out, or that numbers are not accurately capturing the whole reality, or that the information and accounting needs of the donor/government are being put above the learning and adapting needs of the organization serving you. Or, it may be that people working in these organizations are not taking (or being given) enough time to adequately learn and reflect on their work due to outside pressures to reduce overhead and produce results. Whatever the case, if we can continue to push for a more holistic, systems-based, human centered monitoring and evaluation, we will at least have the chance to correct some of the major issues with the status quo and put ourselves on the path towards a better world.

References:

http://usaidlearninglab.org/lab-notes/taking-time-stop-and-think-shifting-aid-models-manage-systemic-change

http://www.hks.harvard.edu/centers/cid/programs/building_state_capability/what-is-pdia

Andrews, Prichett, and Woolcock. “Escaping Capability Traps through Problem Driven Iterative Adaptation (PDIA).” June 2012. Working Paper No 240. Center for International Development. Harvard University.

http://www.springfieldcentre.com/wp-content/uploads/2013/07/Evidence-Based-Policy-and-Systemic-Change1.pdf

http://www.seepnetwork.org/monitoring-and-measuring-change-in-market-systems—rethinking-the-current-paradigm-resources-937.php

http://www.undp.org/content/undp/en/home/librarypage/capacity-building/discussion-paper–innovations-in-monitoring—evaluating-results/

http://www.fsg.org/tabid/191/ArticleId/964/Default.aspx?srpush=true

http://tamarackcommunity.ca/downloads/vc/Developmental_Evaluation_Primer.pdf

http://blogs.worldbank.org/category/tags/big-data

http://www.intrac.org/data/files/resources/145/Using-Qualitative-Information-for-Impact-Assessment.pdf

http://www.mande.co.uk/docs/MSCGuide.pdf

 

 

What I am Doing Here Part 1: Historical Context and Impact Investing

[This is the first part of a much longer post about what I am involved with here in Ghana. Hopefully, it’s informative and not too long-winded! Again these are my opinions and ideas, not representative of any larger group. Next post I’ll have some pictures and more exciting stuff from my trip to Mole National Park this weekend.]

The world of aid and international development has been in a state of constant flux and commotion since the concept was invented near the end of the Second World War. The idea, namely, that a country or group of countries espoused to have “development” can pack it up and ship it out to places that are determined to need it, has proven extremely resilient and captivating despite an eye-popping history of failure, waste, and incompetence. The reasons why this area has been constantly picked apart, reorganized and reimagined are many, but some of the more important ones include the startling complexity of actually making this idea work, and the fact that everyone seems to have a different opinion about how to approach the issues or their own pet projects and ideas. This seems to lead, invariably, to a constant flow of development trends and fashions that rise and fall like the prices of their silver bullet point solutions.

I won’t attempt to cover the history of international development here, nor am I any sort of expert on the topic, but there have been countless books written, hands wrung, eyebrows furrowed and knickers knotted about what it is, why what happened in the past did not work and what new ideas would make it better.  Part of what fascinates me about this subject, is the multiplicity, dichotomy and contradiction of ideas involved and the fact that, despite the hard work of thousands of passionate people, the billions of dollars thrown at it over the decades, and the intellectual brain juices that have been squeezed into it, it is still hard to call international development a success with a straight face.

Trying to force development on a country or group of people seems to me akin to trying to make a tree healthy by air dropping soil, water and fertilizer on it, giving it periodic blasts of CO2 and UV radiation and showing it pictures of how other trees have grown. The tree already knows how to grow, in fact it has been around for quite a while already, what’s needed more than anything is a healthy environment, time to grow, and the removal of unfair barriers that are holding it back. Instead of poking and prodding it like a well-meaning but clumsy toddler with pudgy fingers, a better approach might be to begin with cleaning up the polluted air and water around it, removing the rocks impeding its roots, and maybe even trimming back some of the overgrown branches of the surrounding successful trees so that a little more sunlight can shine through.

Do not get me wrong, despite my delight in naysaying and cynicism, there have been many positive results, successful projects, and lives changed for the better. And, I wouldn’t be passionately interested and actively participating in this work currently if it was just a voyeuristic car-crash watching pastime that was beyond redemption. What I see is humanity struggling up against the boundaries of extreme complexity and coming to terms with its own deeply embedded systems that we shape and that shape us. Our collective ability to overcome massive hurdles from human rights and justice to poverty and development to climate change and global warming hinges on our being able to work with this complexity and each other.

Development is an answer no one was looking for to a question that still hasn’t been understood correctly. Still, the world keeps on spinning and the logical reductionism that says every problem must have a solution continues to hold fast.  The current iteration of this thinking finds us in the early 21st century having gone through the massive infrastructure projects of the 60s and 70s, the structural adjustments fiascos of the 80s, and the micro-finance explosion of the 90s, (among other trends). Arriving at the present day with a new set of buzzwords and theories with names like “impact investing”, “development impact bonds”, and “enterprise philanthropy”, we are once again just at the cusp of a new breakthrough solution.

Impact Investing is loosely the area of development in which I am working and is a relatively new trend, (or a rehashing of old ideas depending on your perspective). It falls under the larger umbrella of social finance which is an attempt to apply some of the successful approaches of business to the world’s social and environmental problems.

The core tenets of this approach are as follows:

1)      Business and capitalism have been massively successful and efficient at getting things done in many places in the world, and small businesses in particular are where a huge amount of the product and services we all rely on come from.

2)      “Historically, regulation – and to a lesser extent, philanthropy – was an attempt to minimize the negative social consequences of business activities. But there is a history of individual investors using socially responsible investing to express their values, usually by avoiding investments in specific companies or activities with negative effects.” (Wikipedia)

3)      “It’s going to take far more money than all the philanthropies and governments have at their disposal to make a significant impact on improving the lives of all the poor and vulnerable people in the world…Impact investing – which helps address social and/or environmental problems while also turning a profit – could unlock substantial for-profit investment capital to complement philanthropy in addressing pressing social challenges” (The Rockefeller Foundation)

4)      “Impact investments are investments made into companies, organizations, and funds with the intention to generate measurable social and environmental impact alongside a financial return. Impact investments can be made in both emerging and developed markets, and target a range of returns from below market to market rate, depending upon the circumstances.” (The GIIN)

5)      “At scale, an impact investing industry could allow for a renewable and sustainable form of financing for an endless array of initiatives, from poverty alleviation and affordable housing, to natural resource conservation, and even clean infrastructure projects.” (RBC)

Growth Potential for Impact Investing. (impactinvesting.marsdd.com/‎)
Growth Potential for Impact Investing. (impactinvesting.marsdd.com/‎)

This approach builds on the ideas of micro-finance, i.e. that money and credit make the world go ‘round and that by keeping these flowing freely people and businesses will be better able to bootstrap their way out of poverty. Impact Investing also adds in elements from the venture capital and investment worlds with tools and concepts like “portfolios”, “asset classes”, “investment due diligence”, and “payback periods”. It also attempts to improve upon them by adding requirements for “social and environmental impact” as part of the terms of the agreements.

Types of Impact Investments (impactinvesting.marsdd.com/‎)
Types of Impact Investments (impactinvesting.marsdd.com/‎)

The quantifying of social and environmental impacts, both positive and negative, has been notoriously difficult (or conveniently ignored?) in traditional capitalism and these effects are usually grouped under the term “externalities”. Part of what social finance attempts to do is build a framework on which to compare and measure the externalities created by businesses. By standardizing these measurements, the extent or existence of social and environmental impact can be determined and thereby “re-internalized” back into the core operations of the business. One such framework is the IRIS metrics created by the GIIN (Global Impact Investing Network) another is an assessment prepared by the B-Lab (Benefit Corp) and ANDE (Aspen Network for Development Entrepreneurs) which allows businesses to benchmark their activities and approaches to see how and where they might improve. This approach is still in its infancy, but I think it has enormous potential to create change as long as the right companies sign up for it.

Impact Capital Spectrum
Impact Capital Spectrum

Additionally, I see several other very positive things about this approach:

1)      Impact investing seems to be more equally applicable to both the developed and developing worlds. Unlike microfinance, which doesn’t find much of a niche in the developed world, Impact investing could conceivably be applied equally well to an agricultural initiative in Canada, or a housing project in Brazil. This might lead to us being a lot more fair and honest about what is working and what isn’t when it is in our backyard as well as someone else’s.

2)      Small and medium-sized businesses are not going away any time soon, and these entities meet many of the needs of society in a relatively efficient and fair way. (Large or multinational businesses beholden to shareholders are quite a different story). Giving these smaller players access to capital so that they can play their part in meeting these needs seems beneficial.

3)      Money makes the world go ‘round and it makes sense that if you want to have a profound and highly leveraged impact on the way the world works, you would go after businesses and the way the financial and economic world works.

4)      Impact investing is also helping challenge the core tenet of business, (that is, to make money at all costs), and is changing the conversation about what role business should play in society.

5)      As an investor myself, I’m very concerned about where and with whom my money is sleeping at night. Right now the choices for maintaining your values with your savings include storing them under your bed, investing in the typical mutual fund mixed bag of stocks and bonds, or maybe trying “socially responsible investing” which provides only a negative screen for the very worse companies. Having the option of putting my money somewhere that at least has the chance to do some active good, is very appealing.

Despite the potential, there are still many kinks to work out with the impact investing approach and questions left to answer. Here are a few to get you thinking:

1)      There appear to be few ways that fundamentally meet the needs of the poorest people that will generate any sort of return on investment. Healthcare, infrastructure, education are generally not regarded as money-making opportunities (though they can be) and as such are often highly subsidized by the government as examples of “common goods”. A corollary to this is do you really want to be making a profit off of a poor person’s healthcare or housing, and if so how much is acceptable?

2)      A lot of the major social issues that cause or are linked to poverty stem from existing government and/or market failures that require expensive and slow processes such as research and development, behavioural change, building a market, and maintaining ongoing services. The time frame for these sorts of processes is far slower than the 3-5 year horizon of most investments.

3)      What level of risk is acceptable for non-financial returns? If your stock investment loses 10%, the effects are fairly obvious. If your social impact investment in a housing project falls through, on the other hand, it could mean someone no longer has a place to live.

4)      Where did all this investment money come from in the first place? How did foundations like Bill and Melinda Gates or Rockefeller get their investment capital? And how might Impact Investing challenge or reinforce the already gross inequalities of wealth, power and privilege that exist in our world?

5)      Is it possible to maintain an objective, apolitical stance when dealing with social and environmental issues and values? Or is impact investing even purporting to be apolitical?

6)      How is all this investment and growth tied-in with the bounded ecological resources of the earth? Is it possible that all these new businesses will be able to operate in harmony with themselves, let alone with the planet?

Hopefully, your curiosity has been peaked about impact investing. Whether you see it as repackaged western imperialism, coached in the language of business and buttressed with pseudoscientific jargon, or as an extremely practical way to create change with a huge potential for worldwide impact, or somewhere in between, you’ll agree it is a very important topic. You will most definitely be hearing more about Impact Investing in the coming years so it’s important to be aware of it and start thinking critically about what is going on. What I am excited about is being involved with it from the early stages to see how rapidly and effectively we can test and iterate through all the bad ideas, before hopefully arriving at something positive. At this stage, I am equally concerned and hopeful, but there is still a lot to learn and explore.

In the next post in this series I’ll discuss how EWB Canada, ostensibly a group of practical engineers, ended up getting involved in this area and what we hope to achieve. Finally, in the last installment I’ll get into the details of the work I’m doing in Ghana and how it connects back to the big picture.

References and Resources:

http://en.wikipedia.org/wiki/Impact_investing

http://www.thegiin.org/cgi-bin/iowa/resources/about/index.html

http://www.aspeninstitute.org/

http://www.bcorporation.net/

http://www.ssireview.org/blog/entry/the_trouble_with_impact_investing_part_1

http://www.sig.uwaterloo.ca/sites/default/files/documents/Developmental%20Impact%20Investing%20-%20Geobey,%20Westley,%20Weber.pdf

http://evpa.eu.com/wp-content/uploads/2010/11/IESE_impact-investing-print-version.pdf

http://www.rockefellerfoundation.org/uploads/files/2b053b2b-8feb-46ea-adbd-f89068d59785-impact.pdf

http://www.mim.monitor.com/downloads/Financial%20Capability_Bridging_the_Gap_Monitor.pdf