Agency-Wide Databases in Multi-Program Nonprofits

Having worked (and often struggled) with developing and integrating client databases across multiple programs for over 15 years, I jumped at the chance to contribute an article to this forum. Many agencies successfully implement databases on a program by program basis, but get stymied when they need to report across multiple programs because of the high cost of programming, incompatible data sets, and the cost of hiring a database administrator to babysit such consolidation. I have come across wo examples that stand out as successful though, which is particularly interesting because they used different solutions.

Background: Why These Organizations Needed Databases

Two Boston-area multi-program agencies, each with roughly 150 employees, received a federal grant that requires annual consolidated reporting of demographics. To generate these reports and develop similar client counts for grant writing and internal management, both resolved to build agency-wide databases that would track all clients. Both agencies operated multiple sites and had several existing database applications to meet program-specific needs.

The key choice for both agencies was either to make use of the existing program-specific databases, consolidating them electronically, or to re-enter client data. With roughly 10,000 clients each, re-entry would have been a major undertaking. In the end, Agency One integrated the program-specific databases, while Agency Two built a simple demographic database and re-entered the data. Both are reasonably pleased, but the data collected by Agency Two is cleaner and more consistent.

Starting to Build Solutions: First Steps and Missteps

Both agencies started building databases that consolidated and rationalized data from their existing systems. This method was hard to develop because each of the existing systems coded data in different ways, so that consolidating fields as simple as ethnicity required complex logic. Any change in coded data in the existing systems required changes in the consolidated system! For example, after the system was programmed, the agency found that it needed to track earned income in two categories: employment wages and contracted hourly pay. They had to add these new income codes not only in the Head Start database where the need was identified, but also in the centralized database, and the programmer had to add logic to consolidate this new code into the federally mandated codes for income source. Every code change or addition carried the same high overhead.

Once these databases were complete, however, both agencies could generate their reports easily. Alas, both agencies then discovered that the reports were incomplete and inaccurate. They discovered, to their great surprise, that program staff were doing a poor job collecting and entering data: there were scores of incomplete records in most programs! On further investigation, it turned out that the program staff needed only a few key pieces of information to get their jobs done, but the databases they were using had long and complex data entry screens. They just skipped the fields for which they didn't have immediate uses. This is a very, very common problem that often has no consequences because the data never goes beyond the program. But in these agencies, the consolidated data was only as good as the entered data, and that data was incomplete.

At this point it became clear that different information was important to different people. Direct care staff cared about information needed in their immediate work. Program Directors cared about information needed to manage programs. Executives cared about federally mandated information. Both agencies tried to mandate better data entry by issuing directives, with predictably low success. Nonprofit staff members enter and use data because they need it, not because someone tells them to! They are too busy and overwhelmed meeting the needs of their clients to attend to entering data that appears useless. This is not an easy problem to solve, but it often helps at least to explain why a new database is being developed and to be extremely clear about which fields must be filled in to make that database work. At the very least, agency executives should be aware of how well the program staff is using their databases so that these problems are managed, rather than discovered after the fact.

Solutions

Agency One resolved to fix data entry problems in the existing systems and continue to consolidate them to produce the federal reports. They hired someone to work with the direct care and administrative staff to do three things: train users to operate their systems, supervise complete and accurate data entry, and develop reports that users actually needed. In this way, the users started to see significant benefits from better data entry, and the consolidated information became cleaner as a by-product. Importantly, they started to get more out of their systems at all levels: program reporting, managerial reporting, and agency-wide reporting. Despite these advances, however, they still failed to collect all of the federally mandated information successfully. Too many users still did not see the value of entering hundreds of pieces of data that they would not use, or they were too overwhelmed to do the work regularly and accurately. Agency One now sees this as a multi-year project requiring cultural change among users. After three years, they are well on their way to success and have had to invest in a 3/5 time employee to shepherd the process. Nonprofits often think of the cost of a database as the purchase price, but that's a fraction of the total cost, which also includes the cost of a database administrator, new hardware and network equipment, and periodic programming updates or enhancements.

Agency Two took a different track. They developed a universal paper intake form for every client and hired a person to enter all of the forms into a database. The program staff had only to fill out and send in their forms. This required less user trainer and resulted in very clean data, when the forms were submitted. It did require a lot of energy to keep pressure on users to submit the forms. In fact, Agency Two spent more time and energy than Agency One because the Agency One users get the data entered as part of their normal work, while Agency Two users had to re-write client data on forms. Nevertheless, the simplicity of Agency Two's process can't be beat. Everyone understands it and can follow it, so they continue to get cleaner, better data than Agency One.

In the end, neither approach is "right." One is more elegant and efficient, but less accurate because it's so hard to get good data entry at the program level and impossible to control the various third-party database systems used in programs. Two is more expensive and cumbersome, but for agencies that have long-term clients, the bulk of data entry is done in the first year, requiring only updates, additions, and deletions thereafter. On the other hand, Two is an easy project to manage; agencies without significant database project management experience should follow Agency Two.

Finally, both agencies ultimately had to confront the fact that successful implementation of a client database requires some cultural change: end users and direct care workers need to see information as an important tool in their work, and it takes time, effort, and thoughtful application to make that happen.

What's the Lesson?

There's no clear moral to this story. Different agencies with different capacities and needs find different solutions. However, there are a few critical points to note:

• A successful database implementation, even if it's painful, is so powerful that nonprofits gain great benefits from it
• The human side of database operation, especially data entry, is too often overlooked and has to get attention after the database has been designed; this is wasteful and sometimes harmful
• Agency managers should understand the databases they already have before trying to combine or enhance them
• You have to maintain a database, and this can be done manually or with some automated tools, but build this into your plans and funding.