Evaluating Infrastructure Development in Complex Home Visiting Systems.
Hargreaves, Margaret. Cole, Russell. Coffee-Borden, Brandon. Paulsell, Diane. Boller, Kimberly.
Mathematica Policy Research.
Published: June 2013
American Journal of Evaluation
Vol. 34, No. 2 , p. 147-169
2455 Teller Road
Thousand Oaks, CA 91320
Tel: 800-818-7243 805-499-0721 805-499-9774 (order pubs)
Fax: 800-583-2665 805-499-0871
In recent years, increased focus on the effectiveness and accountability of prevention and intervention programs has led to greater government funding for the implementation and spread of evidence-based health and human service delivery models. In particular, attention has been paid to programs that require significant infrastructure investment and systems change to support large scale replication. For conceptual and methodological reasons, such systems change initiatives can be a challenge to evaluate. To overcome these challenges, this article outlines a mixed methods approach to systems change evaluation and offers a case study of how this approach has been used to evaluate the development of system infrastructure supporting the implementation, spread, and sustainability of evidence-based home visiting projects. The approach combined systems concepts (boundaries, relationships, perspectives, ecological levels, and dynamics) and qualitative methods (project site visits, telephone interviews, reviews of project documents and logic models) with quantitative methods (a web-based partner survey) to directly measure the projects' system properties and contextual dynamics, and to assess how these factors were associated with the projects' infrastructure development. In the case study, the projects worked at four ecological levels (organization, community, state, and national) to build eight types of infrastructure (planning, collaboration, operations, workforce development, fiscal capacity, community and political support, communications, and evaluation). The evaluation found that the size of the projects' partner networks was not as important as the quality of their collaboration or their sharing of common goals in the projects' infrastructure development. (Author Abstract, modified)
program evaluation; home visiting programs; program implementation; systems analysis; case studies