Strategies for Program Evaluation in the 21st Century
Submitted on Mon, 8/19/2013 – 6:23am
By Kyle Henri Andrei
Research Analyst, Idealware
In our increasingly data-driven world, it is more important than ever for nonprofits to be able to measure and monitor the effectiveness of their programs. It’s difficult to improve program services or reach without first measuring their current effectiveness, and measurable numbers—how many meals served at a soup kitchen, how many students in a mentoring program graduate high school—are more important than ever to help organizations identify where they can improve their programs.
Strategies for program evaluation have been the subject of countless books and seminars, but a gap remains in the area of practical resources about the software for collecting, tracking, and reporting on program data. Like many big-data issues, the sector looks to technology for an answer to these questions. We have donor management databases, constituent management systems, and case management systems, but where are the program evaluation systems?
Unfortunately, there is no such thing. All-in-one program evaluation software doesn’t exist, because program evaluation really is a strategy, not a tool.
Surely there are tools – a number of types of software designed for various tasks that, when combined, can help nonprofits evaluate programs – but we found the knowledge on the subject lacking. Our recent free report, Understanding Software for Program Evaluation, uses our extensive research and interviews to identify a five-part, technology-based program evaluation strategy.
The Five Kinds of Software for Program Evaluation
Idealware has identified the five parts of a technology-based program evaluation strategy. When all the steps are combined, they enable your organization to accurately and confidently collect, measure, and monitor the outcomes and effectiveness of your programs, as outlined by the following chart:
Central Hub of Program Data. The foundation for your evaluation strategy is the system we’ve coined your Central Hub of Program Data – where the information from all the data you’ve collected and the findings you’ve analyzed from that data can be tracked and reported on together. From more generalized, customizable platforms like Constituent Relationship Management systems to Case Management Systems or even more specialized databases, the tool you choose will form the foundation of your organization’s program evaluation strategy. While Central Hubs are the core repository for program data, there are often more specific systems where nonprofits will need to dig for their data, like those listed below.
Auxiliary Data Systems. Nonprofits often don’t keep all their data in one place. Many organizations have two or more separate systems to track information on specific programs or constituents that, for one reason or another, need to be kept separate from other constituent data in the Central Hub of Program Data. While this is not an ideal way to manage program data, it’s sometimes necessary – for example, when dealing with confidential health records or other sensitive data, or when you need to track a lot of very detailed information about people or processes. When using an auxiliary data system, the key step is to integrate as much as possible with the program data stored in your Central Hub.
Proactive Data Gathering. Much of your program data will need to be actively collected, like survey results or text messages from constituents, or handwritten notes from your staff. It’s often necessary to go beyond your organization’s standard systems to collect the information you need to evaluate your programs. A common first step to better understanding the results of your programs is to simply ask your organization’s clients for their feedback. From online surveys to qualitative narrative analysis software to text messaging and mobile apps, there are a number of technologies to help you obtain and manage that data.
Pulling Existing Data. This includes the information that can be collected from public sources, such as what people are saying about your organization or services on social media, as well as public data from government agencies that can provide background information to add context to what you’ve already learned. Tools can help you monitor what people are saying about you online and on social media platforms, as well when clients, donors, or other constituents open your emails and access sections of your website. Demographic or geographic information collected and provided by government agencies can provide a baseline against which to compare your program data, or help you identify currently underserved communities. And your organization’s scheduling or calendaring tools can provide insight into how staff time is allocated.
Reporting and Visualizing. Now that you’ve collected all this data on your programs, you’ll need to make sense of what you’ve found before you can improve your service delivery. There are a wide range of tools that allow you to report on quantitative data—for example, you could easily see how many constituents have attended a specific program in the past month and compare that to previous months, or even against other programs. While your program director or a database manager will likely have no trouble making sense of the report quickly, in order to present your findings to other stakeholders – like board members, volunteers, or donors, for example – you may need to visualize the data through a chart or graph. For the most part, your Central Hub of Program Data should be able to handle most of your needs. But if not, there’s a wide range of additional analysis tools, from statistical packages to dashboards and other data visualization tools.
None of these tools are a requirement for a successful program evaluation strategy, but all might help. Many organizations have gone through these processes with little to no technology to assist them.
When deciding to add technology to your organization’s program evaluation strategy, keep in mind that different technologies will make sense for different organizations. For example, Electronic Medical Records may be essential for a health clinic, but not for an afterschool program. In lieu of that mythical all-in-one tool, nonprofits should pick-and-choose the software that makes sense for their own program or programs.
How do you find the software tools that would make sense for your strategy? Download Idealware’s free report, Understanding Software for Program Evaluation, to start. For organizations new to program evaluation, it will walk you through the strategy outlined above and highlight the tools that can help with each of the five areas. And if you’re already an evaluation veteran who wants to see how technology can make your job easier, consider it a reference guide to what’s out there for your specific needs.
Program evaluation is well within the reach of any nonprofit. Technology shouldn’t be a roadblock, but a means to make the process easier. While no software can do the work for you, the right technology can help make your evaluation more effective, more comprehensive, and ultimately easier on your staff.
As Research Analyst, Kyle is responsible for researching software through demos, interviews, and surveys, and using that information to create Idealware’s reports and articles.