Program Evaluation for Foundations Applying Idealware's Pyramid of Metrics Elizabeth Pope - November 11, 2014 In our increasingly data-driven world, organizations — both nonprofits and foundations — need more than ever to be able to measure and monitor the effectiveness of their programs. It's difficult to improve program services or reach without first measuring current effectiveness. Although program evaluation really is a strategy, not a software program, there are tools that can help organizations accurately and confidently collect, measure, and monitor the outcomes and effectiveness of their programs. This article is courtesy of Idealware, which provides candid information to help nonprofits choose effective software. For more articles and reviews, go to www.idealware.org. Foundations are placing ever-greater emphasis on evaluating the performance of their grant program areas, which in turn places more responsibility on their nonprofit grantees to measure and report on their work. What software works best for program evaluation? For foundations and nonprofits alike, the answer is the same: Program evaluation is a strategy, not a tool. That's not to say there aren't lots of useful software options to support a foundation in program evaluation — there are. At Idealware, we've worked to classify them according to our established hierarchy of program evaluation metrics, which defines the difficulty of measuring many common outputs and outcomes. (Read about the hierarchy, which is aimed at both nonprofits and the organizations that fund them.) This article walks through the different levels of the hierarchy, as well as the types of tools that can help a foundation and its grantees collect and make sense of data at each level. We'll start with the least difficult kind of evaluation and move up to the most complicated — evaluation projects that we believe only the largest and most resource-rich foundations should attempt. Measuring Your Foundation's Own Activity Let's start at the base of the pyramid with the most accessible kind of metric: straightforward measurement of the work an organization has done. For a foundation, this would likely be information that's fairly easy to collect, such as how many grants you've awarded, how many program areas you currently fund, and how much money you've allocated in a certain year. But it might also include how many grant applications you received last year, what kinds of organizations submitted them, and what their geographic distribution was — information that ideally can be extracted from what we call your "central hub of program data," or your primary database. For foundations, this will probably be whatever you use as a grants management system, whether it's a dedicated grants tool, a Constituent Relationship Management system, or a custom database solution. Some data might live in your foundation's accounting system or calendaring system, too. And don't be surprised if a fair amount of information needs to be wrangled from Excel spreadsheets. For the most part, this data should be straightforward to locate and access. Participation The next level of evaluation begins to take your grantees into account — specifically, how they carry out your foundation's work. For instance, how many people did your grantees collectively serve last year? How many programs did they put on? As an example to help think through this type of evaluation, consider a workforce development program area that your organization funds across several different organizations. How many people participated? What were their demographics? Ideally, this information is also kept in your central hub of program data, especially if you've got a grants management system that lets your grantees submit this information through online forms. Some foundations will want to explore aggregating information across progress reports submitted by grantees — for this kind of sophisticated reporting, you may want to explore more powerful reporting software like Crystal Reports or Jasper Reports if your database isn't quite up to the task. Initial Perceived Satisfaction or Success At this level, your foundation is looking to collect information about how the programs you fund affect the lives of those who benefit from your grantmaking in the short-term — for instance, after they've completed a program. This data might exist either on paper or in online surveys administered by your grantees, or even in other formats — if, for example, your grantees are experimenting with collecting information from program participants using mobile apps. In an ideal world, your grantees could just provide this data in exactly the format you want — but in practice, it may be challenging for them to administer just the questions you want asked in the context of other data they need to collect from constituents. This means that you might have to "slice and dice" the data to compare results apple-to-apples, or even pay for some evaluation work yourself. If your grantees conduct interviews and focus groups to learn more about their programs' impact, consider Qualitative Narrative Analysis (QNA) software to help identify patterns and trends within textual information. If the information your grantees provide exists only in printed format, you might explore scanning those reports and using Optical Character Recognition (OCR) software to convert the text into machine-readable information that you can then analyze more easily. In our example of the workforce development program, the information might be survey data collected from program participants after they completed the training programs at different sites. Ideally, as the program's funder, your foundation would be able to get an idea of how well prepared they felt to enter the job market. Longer-Term Satisfaction or Activity Climbing the pyramid, metrics collection tends to become much more complex, as it requires grantees to follow participants or other beneficiaries of their programs over time and report results back to you. This could work for foundations that have both long-term relationships with their grantees and confidence in their ability to track this kind of information. In our workforce development example, grantees would need to keep track of the participants after they've completed the program and survey them regularly to find out whether they were able to find and keep jobs after going through the training. We chose this example specifically to demonstrate how quickly the tracking of longer-term satisfaction or activity can become problematic. Many government programs that fund workforce development consider people who have completed training programs to be consistently employed even if they have had 30 different jobs in 30 days — all that matters is that there is no gap in employment. Some even allow gaps between employment stints of two weeks or a month before classifying someone as "unemployed." So grantees might face the problem of collecting different metrics for you — a private funder — and a government agency to which they also report. And that means longer-term satisfaction can be a tricky metric for grantees to collect and for you to interpret. If it's important to you, consider funding your grantees specifically to track it, or hiring a third party firm to measure it for multiple grantees. Attributable Impact At the very top of our established hierarchy is "attributable impact," or understanding the means by which your foundation's grantmaking affects a community at large. For instance, you might attempt to demonstrate how your grants to workforce readiness charities in your city were connected to a drop in the unemployment rate. Imagine how daunting that evaluation would be — you would need to show that your programs did, in fact, have a measurable impact on the unemployment rate and show that it was not caused by the mayor's recent policy changes or a new bus line that enables people to more easily commute to work. This can be a very complex task to take on, as it requires academic-level rigor, control subjects, and lots of resources to execute properly. At this level, public data may come into play rather significantly as you try to benchmark your foundation's work against external forces. Maps and GIS data can also provide insight into your funding's impact through a geographic lens. You may need to use sophisticated data cleaning processes like ETL — extract, transform, and load — a process for migrating information from one database to another — or middleware, a generic term for a software program that acts as a bridge or patch between two unalike applications. You might also need to use a data warehouse, which can help assess performance across programs and systems and identify opportunities to improve grantmaking strategy. We generally recommend that only the largest, most sophisticated nonprofits focus on the upper tiers of the hierarchy of program evaluation, and our advice for foundations is the same. To understand whether your grantmaking programs are working, concentrate first on metrics that are easier to measure correctly, such as your foundation's own activity and the participation and initial perceived satisfaction or success reported by your grantees. Once you've tackled that, try to draw some conclusions about the longer-term satisfaction or activity generated by the programs that your foundation funds. Only a minority of foundations will be able to make sound conclusions about the attributable impact of their grantmaking — and even then, it's likely to be through a detailed study on the subject rather than by asking grantees to provide the data to them. A key part of this discussion is that, as a foundation, you have an opportunity not just to change but to improve this situation. Foundations unknowingly put a tremendous burden on their grantees through needlessly onerous or conflicting data collection and reporting requirements. Though program metrics can provide a wealth of useful information about how your funding is being used, the data collection can be a real hardship for smaller nonprofits without adequate technology infrastructure or a sophisticated understanding of data. Since foundations rarely collect uniform metrics from grantees, the nonprofits you support likely are not only gathering data that you request, but lots of additional data requested by other funders. For foundations interested in reducing the administrative burden on their grantees, we recommend focusing the information you request from the lower tiers of the pyramid: measuring activity, participation, and initial perceived satisfaction or success. The higher tiers are more accessible for larger, more established nonprofits. In fact, the upper tiers of the program evaluation pyramid might be best suited to evaluation efforts headed by large foundations themselves, as large foundations are better positioned and funded to be able to gather and analyze data from multiple sources and identify trends. The hope is that this pyramid of metrics will become more than just another means of thinking about program evaluation for foundations in an already crowded field; it will become a means of representing a less confusing path forward for grantmakers and grantees alike. When it comes to program evaluation in philanthropy, it makes sense to start small. Increase your efforts according to your capacity and don't try to take on all aspects of evaluating your grantmaking at once. Ideally, program evaluation shouldn't be overly confusing or frustrating, but a source of increased clarity about your funding strategy. Additional Resources Understanding Software for Program Evaluation: This report provides overviews of the types of software that, when brought together, can enable nonprofits and foundations to accurately and confidently collect, measure, and monitor the outcomes and effectiveness of their programs. Project Streamline: A highly valuable online resource for grantmakers interested in learning more about best practices in online reporting from grantees. The Urban Institute and the Center for What Works: A clearing house of information about nonprofit program evaluation, including a taxonomy of commonly tracked indicators. Many thanks to TechSoup for the financial support of this article. Image: Brian A Jackson / Shutterstock This work is published under a Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License.