What am I Doing Here Part 4: Monitoring and Evaluation

No matter what you do or where you work, if you are spending someone else’s money, eventually, they are going to ask you what the hell you are doing with it, how effective you have been, and what you have been able to accomplish. This need for accountability and documentation is, however, only one part of the much larger picture usually referred to by the conjunctive phrase: monitoring and evaluation (M&E). Having not worked in the social sciences or studied development in school, my eyes have been opened to this fascinating and convoluted world of M&E, [sometimes also called Planning, Monitoring, Evaluation and Learning (PMEL)], over the past year.

Though the idea of planning your work, making sure it is on track, evaluating the relative success or failure and trying to learn from the experience has been around forever, the M&E requirements of development projects have been turned into an art form. There are elaborate methodologies, old school tricks of the trade, hot new frameworks and cutting edge research papers being published on the topic monthly. Careers are being made and lost trying to prove what works and what does not, NGOs are pouring more of their budgets into checking all the latest and greatest boxes, and every few years there’s a renewed push by the major multilateral agencies and governments to “get M&E right this time”.

Despite all this, there still seems to be an air of general dissatisfaction with our collective ability to find out what works, make sure it happens, and to reproduce it somewhere else. There is also the frustrating realization that we may never be able to measure all the complex and nuanced subtleties that are inherent of change in human systems. Moreover, it may be that these unmeasurable changes are the most powerful and important, yet remain outside of our reach of understanding and reproducibility. As an always on point Albert Einstein noted, “Not everything that can be counted counts, and not everything that counts can be counted.”

einstein-not-everything-that-can-be-counted-counts

There is also a growing feeling, within the development world, that we need to stop looking at issues like diseases and poverty as linear effects stemming from directly linked causes. The so-called “systems approach”, which is the antithesis of this, has been gaining steam over the past decade. In this approach, the sum is greater than the individual parts, the relationships are just as important as the players, and everything is at once linked and constantly evolving. While this is a far more accurate representation of the complex, emergent, human systems found in development, it throws a bit of a wrench into the whole M&E thing. How do you know where you are going when you can’t see where you have been? How do you measure the change you intended to create, when everything is changing all the time? How do you learn from one instance of a system when those exact conditions are likely never to exist again? These issues are crucial across the board in development and social change projects. Within the impact investing world, where BDSA works and I find myself, M&E is where the rubber meets the road and is at the crux of proving the central concept that businesses can be drivers of positive social change.

Monitoring: The systematic & continuous assessment of the progress of a piece of work over time, which checks that things are ‘going to plan’ and enables adjustments to be made in a logical way.

Evaluation: The periodic assessment of the relevance, performance, efficiency, and impact of a piece of work with regards to its stated objectives.

The history of M&E begins several decades back, when the rigour of scientific studies and evidence based results began filtering over to the social sciences and government. Within a few more decades, these ideas had made their way into the development context and had been sufficiently evolved to the point where it became almost impossible to do a project without applying a logical framework (logfram) or Results Based Management (RBM) tool. In the former, the project planner (usually on-high, in an air conditioned office far, far away), lays out the overall intended outcome of the desired change. He or she then works backward to determine the measurable outputs or metrics that would indicate the outcome has been achieved, the activities needed to produce these outputs and finally the inputs needed for each activity. At the end of the project the impact of the work is determined by subtracting the counterfactual (i.e. what would have happened if no intervention had been made) from the outcome.

M&E Impact Chain

This approach works great in the sciences where experiments can be carefully controlled, economies of scale allow for statistically valid results, and where causes can be closely linked with effects. In the real world of development, these conditions are rarely met, and the logfram approach leaves you with a decent planning tool, but a horribly rigid and impractical measurement and change management tool.

Formula from the World Bank's guide on M&E
Sample formula from the World Bank’s guide on M&E

 

So, development project implementers find themselves in a bit of a bind: on one hand, they need rigour and accountability, and on the other they need flexibility and constant adaptation to a rapidly evolving reality. They need to be able to prove their interventions were effective, substantiate the attribution of the results to their donors (i.e. determine the counterfactual), and be relatively confident that these results are reproducible and sustainable (i.e. the project continues on, as designed). This drives an excessive focus on donor accountability; an obsession with control, causation, and attribution; and an overall rigidness and inflexibility that is more of a hindrance than a help. To say nothing of the fact that the beneficiaries/victims of these projects (i.e. the poor) are often cut out completely from the planning, evaluating and learning process and are left scratching their heads when the NGO declares the completion of yet another successful project.

In order to counter some of the obvious short comings of the traditional Results Based Management (RBM) approach, various tools, frameworks and approaches have been introduced over the years. Most of these attempt to capture the qualitative aspects of projects through questionnaires, interviews, stories and through participatory, beneficiary-driven planning and evaluation. They also attempt to design for, or at least acknowledge, the complex, emergent nature of the systems of which they are a part and to weigh the needs of the beneficiaries above those of the donors. And while there is no silver bullet or one-size-fits-all approach that is going to work everywhere, a mix of these different tools and approaches is helping to breakdown the orthodoxy of the purely quantitative result.

Here are a few key trends in M&E that have been getting attention lately and some sources on where to find more information. For a great overview of all these recent trends and others, see this paper.

Developmental Evaluation – This approach is basically an attempt to reduce the feedback cycle between learning, doing and correcting to almost zero. By collecting data in real time and making decisions based on a constant feedback cycle, the theory is that the project can adapt and evolve in conjunction with the system, thereby avoiding the need for major course corrections down the line.

Shared Measurement – In this case, common metrics are used across organizations on “scalable platforms” in order to facilitate the sharing and discussion of results and learnings on a much greater scale. It also helps organizations share responsibility for their data collection and learning.

Big Data – As the name suggests, this approach is based on the assumption that if some data is good, lots and lots of it must be better. Using short feedback cycles, real-time digital data from a variety of sources (such as website traffic, twitter, blogs, phone records, etc.), and data visualizations and infographics, it is hoped macro-trends and insights will emerge.

Problem Driven Iterative Adaptation – The PDIA approach is based on four key principles: First, focusing on solving locally nominated and defined problems in performance (as opposed to transplanting pre- conceived and packaged best practice solutions). Second, it seeks to create an authorizing environment for decision-making that encourages positive deviance and experimentation (as opposed to designing projects and programs and then requiring agents to implement them exactly as designed). Third, it embeds this experimentation in tight feedback loops that facilitate rapid experiential learning. Fourth, it actively engages broad sets of agents to ensure that reforms are viable, legitimate, relevant and supportable.

QUalitative Impact Protocol – Qualitative information is often hard to communicate between stakeholders, even though it provides rich and relevant learning. The QUIP approach is an attempt to get qualitative data taken seriously by collecting it in a systematic and structured way.

Most Significant Change – MSC is a story based approach to help identify the causes of a significant/critical change (positive or negative) relating to key objectives, rather than looking for trends related to a certain phenomenon. This makes it easier to track stories of changes related to less easily quantifiable issues such as “capacity building” or “gender equality”.

Here is a brief look at a few more:

M&E Summary

Even with all of these tools and techniques being simultaneously developed, piloted and perfected. There is still much to be done in changing the development system itself. Here are a few recommendations for the future:

  • First and foremost, there needs to be much greater trust between donors and implementers and a lot more freedom given to experiment, adapt and learn. By far the biggest hurdle standing in the way of creative solutions to poverty reduction is that donors don’t trust implementers with their money, and implementers don’t trust donors with their program designs.
  • Donors , implementers, beneficiaries and other stakeholders need to come together to create spaces for innovation, seed the soil for new ideas, and embrace the failure of some projects in the name of a better overall result.
  • Agreement on the big picture problem definition or mission is necessary between stakeholders. This shared understanding should then serve as the organising principle when adapting activities and plans to ensure that practitioners are beholden to the ultimate mission, rather than the activities themselves.
  • Direct attribution of an impact is neither possible nor desirable in a complex adaptive system. The need for implementers and NGOs to clearly attribute how their work created a specific change should never take precedence over achieving the highest quality and most impactful aggregate change in the overall system. If implementers were able to put as much effort into achieving absolute results, as they do into competing for and seeking credit, everyone would benefit.
  • Finally, we need to give up the obsession with finding a be-all and end-all, silver bullet solution to our M&E needs. There will never be one perfect technique, just lots of little imperfect ones and the goal should be to continuously inch them forward.

So what is this all about, and why should you care? Well, whether you are a rural farmer in Ghana, a student in the UK, or a hospital patient in Canada, your life is probably significantly affected by the type of M&E performed by the organization with whom you are interacting. It may be that your story is being left out, or that numbers are not accurately capturing the whole reality, or that the information and accounting needs of the donor/government are being put above the learning and adapting needs of the organization serving you. Or, it may be that people working in these organizations are not taking (or being given) enough time to adequately learn and reflect on their work due to outside pressures to reduce overhead and produce results. Whatever the case, if we can continue to push for a more holistic, systems-based, human centered monitoring and evaluation, we will at least have the chance to correct some of the major issues with the status quo and put ourselves on the path towards a better world.

References:

http://usaidlearninglab.org/lab-notes/taking-time-stop-and-think-shifting-aid-models-manage-systemic-change

http://www.hks.harvard.edu/centers/cid/programs/building_state_capability/what-is-pdia

Andrews, Prichett, and Woolcock. “Escaping Capability Traps through Problem Driven Iterative Adaptation (PDIA).” June 2012. Working Paper No 240. Center for International Development. Harvard University.

http://www.springfieldcentre.com/wp-content/uploads/2013/07/Evidence-Based-Policy-and-Systemic-Change1.pdf

http://www.seepnetwork.org/monitoring-and-measuring-change-in-market-systems—rethinking-the-current-paradigm-resources-937.php

http://www.undp.org/content/undp/en/home/librarypage/capacity-building/discussion-paper–innovations-in-monitoring—evaluating-results/

http://www.fsg.org/tabid/191/ArticleId/964/Default.aspx?srpush=true

http://tamarackcommunity.ca/downloads/vc/Developmental_Evaluation_Primer.pdf

http://blogs.worldbank.org/category/tags/big-data

http://www.intrac.org/data/files/resources/145/Using-Qualitative-Information-for-Impact-Assessment.pdf

http://www.mande.co.uk/docs/MSCGuide.pdf

 

 

What am I Doing Here? Part II: Social Enterprise, EWB and Business Development Services

About 3 or 4 years ago EWB found itself amidst a swirling movement of new thinking on development issues, disillusionment with traditional economic strategies, and the rise of the middle class in Africa, amongst other trends and developments. To understand a little better how a group of dyed in the wool, pocket-protecting, calculator mashing engineers became involved with business development work in Africa, it helps to know some of the recent history of EWB and the amplifying and colluding trends of business, finance and development of the past couple years.

In the last post I covered a few thoughts on the development world in general and the current state of affairs with regards to social finance and impact investing. This time I’ll drop down to the meza-level and explore a few of the main theories and concepts that have affected EWB’s current thinking and direction and provide an overview of our activities in these areas. This should help explain why Small and Growing Social Businesses (SGSBs) are a large part of EWB’s “systemic innovation” theory of change and how this connects upstream to the impact investing world and downstream to what we are doing on the ground in Ghana and Zambia.

If, after reading, you are interested in supporting this work, then head on over to our “We are” page which is part of EWB‘s year end fundraising campaign.

Concepts in EWB

EWB has always prided itself as being on the cutting edge of the latest thinking in the world of international development and, often, one of the few organizations small and versatile enough to immediately begin putting these theories to the test. Another strength of EWB has been its ability to gather and synthesize ideas and concepts from different fields (engineering, management consulting, finance, business, social science etc.) to create something unique. Here are a few of the main concepts that have had a significant influence on EWB over the past few years:

Systems Thinking – Something every engineer is intimately familiar with is the world of systems and how the interactions between individual parts work together to create a greater whole. Be it in control theory, system-on-a-chip design, assembly line optimization, or chemical processes, engineering more often than not is about combining increasingly smaller components into increasingly larger and more complex systems. This thinking began entering the world of business and management consulting beginning with the “Fifth Discipline” by Peter Senge in the early 90’s and has since been adopted by many other disciplines. The aim is take a highly complex ever-changing system of players and interactions and approach it from a holistic perspective instead of piece-by-piece. The ability to better understand the system as a whole allows one to better find and exploit leverage points, synergies, and loopholes and nudge the system towards a new paradigm instead of just patching up symptoms.

Social Enterprise – This fairly ambiguous term has been used to describe everything from a purely non-profit organization delivering only social good all the way to a purely for-profit, traditional business which just happens to be generating some social good (i.e. jobs, health, education) as a by-product of its primary profit incentive. While no one can really decide where to draw the line, the basic concept is that a social enterprise is an operation that generates highly scalable, wide-reaching, positive social or environment impact to its community, stakeholders, or customers. These are typically headed by social entrepreneurs who are very much like regular entrepreneurs, except instead of starting a business they aim to start a movement. You can read a lot more about the concept in the excellent book “How to Change the World” by David Bornstein.

Blended Value – This is another slippery concept which, like social enterprise, is open to broad interpretation. At its core, however, blended value is about businesses or non-profits delivering a blend of economic, social and environmental value as part of it mandate. This takes the concept of “triple-bottom line” a bit further by arguing that these aims should be part of a holistic approach instead of just separate categories. (Unfortunately, due to the nature of some businesses the blending is more like oil and water than milk and coffee). Read more about it here.

SGSB/SME – Small and Growing Social Businesses or Small and Medium Enterprises are two terms used to describe up and coming businesses that have the potential to scale up and deliver far-reaching impact across many communities.

Base of Pyramid – This concept holds that there are an awful lot of people who do not make or spend much money in the world and are hence largely ignored by most mainstream businesses. The theory goes that if a company can reach BoP consumers with their products or services, this will have a profound impact on the overall economy (for better or worse). By serving the needs and demands of the base of the pyramid effectively, companies can make large returns from relatively small margins due to the economies of scale. Read more here.

Participatory Approaches – Based on the work of Robert Chambers in “Whose Reality Counts?” participatory approaches and methods are about reversing the typical top-down, expert-driven power dynamic of traditional development. Instead of designing (usually poorly) the project from a far, implementing something using experts from overseas and leaving. P.A. is about leaving the conception and control squarely in to the hands of the people who are living with the project and giving them discretion about how and where to use outside “expert” help. EWB attempts to do this (though not perfectly) through being a value add partner that local businesses and governments can bring in as they choose. We are aiming to be “on tap”, not “on top”, but there are many systemic issues in the nature of development that prevent this from truly being the case.

EWB and Systemic Innovation

Through its earlier work (2005-2011) in water and sanitation in Malawi, food system in Ghana and Burkina Faso, and governance and rural infrastructure also in Ghana, EWB was exposed to many levels of interactions from high level aid decisions, to local government program implementation, to the effect of these programs on the citizens of these countries. Through this perspective a trend started to emerge that was starkly different from much previous technology-based development thinking. The realization was that instead of focusing on broken water pumps, low harvests, ineffective government programs and other inputs, outputs or technologies, a better way to achieve lasting, resilient, locally directed, and scalable change, was to take a much broader approach. Instead of focusing on implementing “solutions”, EWB decided it was better to first understand the “systems” from which these symptoms manifest and design ways to create change from within. This systemic approach has since become EWB’s specialty and has allowed it to use its limited resources and people on the ground much more effectively. This approach was codified after much internal debate and dialog into EWB’s current vision as a leader in “systemic innovation” in January 2012.

This grand sounding idea, while flashy, modern and, yes, a little presumptuous, has definitely left many a head scratching both inside and outside the organization. How exactly does one plan to create systemic change a priori? Can these efforts be measured, evaluated, attributed, correlated to outcomes? How big of a system are we talking about? The past two years we have been grappling with these questions and experimenting with a new model as an incubator of systemic innovations and social enterprises. Instead of operating like a traditional NGO with projects and programs, EWB works more like a venture capital firm for social entrepreneurs to pitch their ideas and get “investments” of people, connections and some funding.

Business Development Services Africa

While EWB has maintained its work in water and sanitation, governance, and agriculture; through its iterative progression, it has come to focus more on the business sector as the vehicle with the most potential to deliver large-scale impact. Business Development Services Africa was spun off of EWB’s successful work in agriculture value chains, (itself an iteration of working with farmers and noticing major barriers and inefficiencies in the way inputs and outputs flow along a value chain). The BDS venture was pitched with the idea that expanding the number and impact of SGSB’s (especially in the agriprocessing sector) could leverage massive higher order effects upstream for farmers and suppliers of raw foods and downstream by creating jobs and locally sourced food products for BoP consumers.

This leverage point at small businesses within the system of a local economy has been repeatedly cited by politicians, economists and NGO’s around the world as a primary source of jobs and economic growth (both major issues and topics other posts for sure).  The question is how to rapidly and effectively accelerate these companies while not creating even more problems down the line. The answer may be in combing ecosystems of socially and environmentally driven companies together along a value chain to produce the most efficient outputs while doing the least amount of collective harm. No one has quite figured out how to do this yet and the North American model surely isn’t the most inspiring, but BDS hopes to push forward the thinking around this and help find new ways of operating.

BDS is a relatively new venture within EWB and is still very much in a start-up, exploratory stage as we tighten our business model, experiment in different roles and develop our theory of change. Within the next several years, it is hoped that BDS will become its own separate entity; based directly in the countries it has operations and employing mostly, if not all local staff. As of right now, we are only operating in Ghana and Zambia with a handful of client organization, but aim to grow and diversify as we iterate. Our current numbers on the ground as of the end of this month will be four in Ghana and five in Zambia.

Our model of assistance is meant to harness the power of small business to develop a self-reliant and sustainable form of poverty reduction and social betterment. BDS only works with clients that request our services and are able to demonstrate strong entrepreneurial drive and business models with tangible social and environmental impact. In essence, we are helping entrepreneurs build better business systems, and we are connecting those entrepreneurs with the right investors to scale up. Our solutions are co-developed, co-owned and co-executed with the entrepreneur and, therefore, have their buy-in from the beginning.

There are a variety of systemic failures in markets, institutions, and rules and norms that contribute to the demise of many small enterprises. This makes it harder for entrepreneurs to access “growth capital” to expand their businesses. But, there is more to it than that: businesses also need to be able to utilize that capital efficiently and sustainably in order to effectively benefit from it. This is why BDS is primarily focused on capacity development of entrepreneurs. From our perspective, both investors and entrepreneurs are losing out because often the underlying issues that make businesses “unbankable” are surmountable with the right approach. As David Packard (the P in HP) said, “More organizations die of indigestion than starvation”.

BDS is piloting and testing its operation in three main categories:

–          Pre and post investment technical assistance through embedded consultants: Here we partner with pre-investment (start-up to growth stage) businesses and work with them to streamline their business and technical systems, and prepare business plans and record keeping for investor due diligence. We also partner with impact investors and work with them and the client organization to ensure a smooth transition post-investment. That is, direct, client-based consulting to understand, diagnose and remove the barriers that stand in the way of the client’s objectives.

–          Business incubators: In this role we work with existing NGO or government lead business incubation programs to develop the tools, training, selection processes and monitoring and evaluation frameworks for the incubated businesses.

–          Mobile business training clinics and coaching: This stream, currently based in Ghana, brings together cohorts of small business leaders in groups from each geographical area for a three-month training and coaching program delivered by local Ghanaian business leaders. Training modules include management and leadership development, finances and accounting practises, and project management techniques. The training sessions are accompanied by regular coaching sessions to help the small business owners to immediately begin implementing what they’ve learned.

We are still very much working on the overall direction to take to increase our reach and effectiveness and continuously learning and discussing with local partners about what needs are not being met. In particular a lot of work needs to be done around understanding how these intangible and unpredictable things like “business success” or “positive social and environmental impact” can be influenced, quantified, qualified, compared, measured and/or evaluated. As we increase our understanding this system, we will be narrowing in our specific leverage points and value adds.

In the final installment, I’ll address the day-to-day operations here in Tamale including my work with Taimako Enterprises, Mobile Business Clinics, and some of BDS’s other projects and clients in Ghana and Zambia.

Thanks for reading!