Sustaining change: What does the evidence say about sustainability?

This brief summarizes the evidence on sustainability of newly implemented practices or programs (“interventions”). The evidence shows that:

·  Sustainability is possible, but successful implementation of a new intervention does not automatically guarantee that it will be sustained in the long term.

·  Sustainability planning should begin early in the implementation stage of any new intervention.

·  There are various frameworks and tools that can be used to understand, plan for, and evaluate sustainability.

·  Five main categories of influences on sustainability have been outlined in the research literature: intervention, external context, internal context, resources and capacity, and processes and interactions. However, more research is needed to understand the most important factors affecting long‐term sustainability. More research is also needed to determine which factors are most relevant for different interventions.

·  Sustainability efforts can be enhanced through ongoing monitoring, evaluation, and adaptation of the intervention and related processes, as needed.

Read this evidence brief below or download the PDF.

Background

You’ve successfully implemented a new intervention – now what? In healthcare settings, putting evidence into practice is one thing; sustaining it is another. While there are many definitions of sustainability, it can be broadly understood as the continuation of most or all of an intervention atier it has been implemented1.

Research shows that sustaining new interventions is possible, but successful implementation of an intervention does not automatically guarantee its long‐term sustainability2,3. It is also clear from the literature that it is more common for some aspects of an intervention to be sustained or for the intervention to be adapted than it is for all aspects of the intervention to be sustained over time1,3,4.

Although knowledge of how to best implement evidence‐based interventions is rapidly expanding1,5,6,7, less is known about how to sustain new interventions over time2,8. Failing to maintain a beneficial new intervention after it has been implemented contradicts the goal of bringing effective, evidence‐based care to people who can benefit from it1. It can also reduce stakeholders’ motivation to engage in future efforts to implement new programs and practices9.

Even though the importance of sustaining helpful new interventions is widely acknowledged in the literature, there are knowledge gaps around how to best sustain change. One challenge to building the knowledge base on this topic is that the existing research uses different terminology and different methods. As a result, there is a lack of clarity and consensus in the literature3. Another challenge arises when the desired benefits of interventions occur at the system or population level, because a long time frame (three to ten years) is often needed to see outcomes in these contexts1,10.

A number of frameworks, tools, and important factors are described in the literature. This information can help with sustainability planning and assessment. Experts suggest that existing knowledge be applied in the early stages of implementation and used to guide ongoing monitoring, evaluation, and adaptation of the new intervention and related implementation processes1.

The Evidence

Sustainability Frameworks and Tools

Conceptual frameworks (or models) provide ways to understand and study sustainability, while tools offer ways to measure and evaluate it. Below is a brief overview of several frameworks and tools that outlines the definition of sustainability used by the researchers and the main topics or factors addressed in the framework and/or tool. This does not necessarily reflect a complete list of all available sustainability frameworks and tools, given that the current brief was produced under tight time constraints and involved an environmental scan of the research and grey literature, not a systematic review of the evidence base.  

Overview of Sustainability Frameworks and Tools

Program Sustainability Model9

Definition of Sustainability: “Program continuation,” involving: (1) continued health benefits associated with the intervention; (2) persistence of the intervention within an organization; and (3) the capacity to maintain the intervention in the long term. 

Key Factors Addressed

·  Project Design and Implementation – project type, financing, duration, effectiveness; project negotiation processes; training;

·  Organization – institutional strength; integration with existing programs; champions;

·  Broader Community Environment – political and socioeconomic factors; community participation.

Tool: A series of questions related to the framework’s key factors are available to guide sustainability planning (see reference 9).

National Health Service Institute for Innovation and Improvement’s Sustainability Model11,12

Definition of Sustainability: Sustainability is achieved when new processes or practices associated with the intervention have become “the norm.”

Key Factors Addressed

·  Processes – credibility of intervention; extent to which staff view it as useful and sustainable;

·  Staff – leadership commitment; service provider awareness of/ involvement in change;

·  Organization – intervention fit with organization’s infrastructure, strategic aims, and culture.

Tool: A self‐assessment tool corresponding to the model can be used to examine whether an intervention is likely to be sustained and what actions are needed to increase sustainability11. The tool and additional information can be accessed here (http://www.institute.nhs.uk/sustainability_model/general/welcome_to_sust...).

Dynamic Sustainability Framework13

Definition of Sustainability: The extent to which an intervention delivers its intended benefits over an extended period of time after initial implementation support is withdrawn. To be sustainable and achieve maximum benefit, an intervention needs to be monitored and adapted over time to fit the changing context.

Key Factors Addressed:

·  Intervention – components; practitioners; outcomes; delivery platform;

·  Context (Practice Setting) – staffing; information systems; culture; business model; training; supervision;

·  Ecological System – other practice settings; policy; regulations; market forces; population characteristics.

Tool: No tool available

Capacity for Sustainability Framework14

Definition of Sustainability: The ability to maintain programming and its benefits over time.

Key Factors Addressed:

·  Internally‐focused domains – organizational capacity; program adaptation; evaluation; strategic planning; communications;

·  Externally‐focused domains – funding stability; political support; and partnershipsii.

Tool: Program Sustainability Assessment Tool (PSAT), developed from a literature review and expert concept mapping. This 40‐item scale covers the framework’s eight domains and was evaluated across programs from four different chronic disease areas. It is available for free here (https://sustaintool.org/). A step‐by-step sustainability planning process using this tool has been published (see reference 15).

Program Sustainability Index16

Definition of Sustainability: The capacity of a program to continuously respond to community issues, meeting its intended goals and serving its target audience.

Key Factors Addressed:

·  Leadership competence;

·  Effective collaboration;

·  Demonstrating program results;

·  Strategic funding;

·  Staff involvement and integration;

·  Program responsivityiii.

Tool: A 29‐item index, reduced from 53 items after preliminary evaluation. The authors recommend that future work continue to assess all 53 items outlined in the original index.

Level of Institutionalization Scales17

Definition of Sustainability: “Institutionalization” – the viability and integration of innovations in organizations in the long term.

Key Factors Adderssed: Factors relating to four organizational “subsystems” (Productive, Maintenance, Supportive, and Managerial) are used to assess the extent to which a program is embedded in an organization.

Tool: A 15‐item questionnaire, scored to define the level of institutionalization as low, medium, or high (see reference 17).

Routinization Interview10

Definition of Sustainability: The continuation of programs. Sustained programs are part of an organization’s “routines” and often rely on/are affected by institutional standards from the broader system.

Key Factors Addressed: 

·  Memory – shared interpretations of past experiences;

·  Adaptation – modification of activities to suit the context;

·  Values – an organization’s collective beliefs;

·  Rules – the rules guiding decision making and action around relevant activities.

Tool: A 15‐question interview is available to assess the sustainability of public health programs (see reference 10).

Intervention‐specific Sustainability Framework18

Definition of Sustainability: The continued use of an intervention after an initial implementation period. (Note: Reference 18 provides an extensive discussion of the various definitions of sustainability and the related implications.)

Key Factors Addressed: Provides hypotheses about the factors that could affect the sustainability of six types of interventions:

·  Implemented by individual service providers;

·  Requiring coordination among multiple staff within an organization;

·  Consisting of new policies, procedures, or technologies;

·  Focused on capacity or infrastructure building;

·  Consisting of collaborative partnerships or coalitions;

·  Involving broad‐scale system change.

Tool: No tool available

Influences on Sustainability

The frameworks and tools in Table 1 show that there is a broad range of possible influences on sustainability that need to be considered. Because the research literature has used different definitions of sustainability and has studied this topic in many different ways, it is not yet known which factors are most important for sustainability. It is also not known if different factors are more important for different interventions or different contexts1,18,19. Despite existing knowledge gaps, the research outlined below provides important information for stakeholders looking to plan for and assess the sustainability of recently implemented interventions.

Sustainability of Interventions within Organizations

A recent literature review19 looked at the factors that influence sustainability and identified five broad categories of influences:

·  Intervention characteristics (e.g., fit with organization, effectiveness); 

·  External context (e.g., policies, system support);

·  Internal context (e.g., leadership support, climate);

·  Resources/capacity (e.g., funding, workforce);

·  Processes and interactions (e.g., training, feedback, adaptation).

These different types of influences tend to overlap in the real world. The specific factors that are most important to consider depend on both the intervention and the context in which it is implemented3,18. The importance of different influences may also shift over time, depending on changes in the organization, the system, and the broader environment13.

Below is a detailed list of the specific influences in each of the five categories (Adapted from references 1 and 19.):

Intervention Characteristics

·  Fit with context

·  Flexibility

·  Complexity

·  Effectiveness or benefit

·  Ability to maintain fidelity

·  Added value/cost effectiveness

·  Ability to be modified

Context (internal and external)

·  Climate

·  Culture

·  Leadership

·  Role clarity

·  Readiness

·  Motivation

·  Lack of opposition

·  Seting characteristics (structure, policy)

·  System/policy change/support

·  Alignment/integration (i.e., program need and priority alignment)

Resources/Capacity 

·  Champions (internal or external)

·  Funding

·  Workforce (staffing, attributes)

·  Resources

·  Community/stakeholder support/involvement

·  Information technology systems and support

·  Allocated time

Processes and Interactions

·  Engagement/relationship‐building

·  Shared decision‐making among stakeholders

·  Adaptation/alignment

·  Integration of rules/policies

·  Evaluation and feedback

·  Training and education

·  Collaboration/partnership

·  Navigating competing demands

·  Ongoing support

·  Planning

·  Publicizing results

·  Developing local expertise

Sustainability of Collaborative Bodies and System‐level Initiatives

Most of the research described above focused on the sustainability of evidence‐based programs or practices within organizations. A much smaller body of research has looked at the sustainability of collaborative bodies and system‐level initiatives20,21,22. This research does not provide conclusive results, but suggests that the following factors are likely to play a role:

·  A history of collaboration;

·  Interagency partnerships/multi‐sectoral involvement;

·  Involving a mix of grassroots and professional organizations;

·  Supportive leadership/champions;

·  A clear vision with well‐defined guidelines and role clarity, along with a commitment to the goals and vision of the initiative;

·  Diversified and sufficient resources and funding;

·  Using evaluation data to inform activities.

According to a recent research article, assessing the sustainability of collaborative partnerships requires good measurement of relationships formed within the initiative, as well as an examination of how relationships within a collaborative body change over time23. The article also suggests that it is important to look at how changes in collaboration over time are related to the goals of the initiative23.

Recommendations for Future Work on Sustainability

To maximize the likelihood that interventions will be sustained, it is important to use the existing evidence to plan and carry out sustainability activities. It is also important to carefully monitor, document, and assess these activities on a consistent and ongoing basis. This will help fill existing knowledge gaps and enhance the likelihood of sustainability1.

Stirman et al.1 make the following specific recommendations for future work in this area:

(1) Use an operational definition/conceptual framework 

Sustainability planning and evaluation efforts should address and document:

·  whether, and the degree to which, the core elements of the intervention (those most closely linked to the desired benefit) are maintained;

·  the extent, nature, and impact of modifications to the intervention;

·  the extent to which desired benefits are maintained/improved after initial implementation supports are withdrawn; and

·  the capacity needed to maintain the desired benefits in the long term.

(2) Define the sustainability goals of the intervention

Both the desired impact and benefits of the intervention should be clearly identified in reference to long‐term sustainability goals.

(3) Choose an appropriate timeframe for assessing sustainability

There is no clear standard in the literature for defining when an intervention can be considered “sustained”1. Most studies use at least two years after implementation as a reasonable timeframe to examine sustainability. However, because it can be difficult to define when implementation has ended, and because some interventions are never fully implemented as originally intended, sustainability is best assessed over several years rather than at a single point in time1.

(4) Document fidelity and adaptation of the intervention

Not much is known about how evidence‐based interventions are changed during/after implementation and the impact of these changes on sustainability. Advancing knowledge in this area requires more information on the nature of changes made to interventions, the reasons for the changes, the processes associated with making changes, and the impact of these changes on health‐related outcomes1,13,24,iv. This information should be carefully documented throughout and after implementation.

(5) Conceptualize, measure, and assess influences on sustainability

Another knowledge gap is the importance of different influences on sustainability. Researchers say that it is important to continue to develop and test measures of sustainability. They also state that both quantitative and qualitative methods should be used to examine influences on sustainability. This will contribute to the existing knowledge base and inform the development of strategies to promote sustainability.

Conclusion

While the evidence provides some guidance on what is needed to sustain new interventions, it does not clearly define how sustainability can be achieved. Existing work in this area suggests that it is important to understand the possible factors that could influence sustainability, and to pay close attention to these factors when planning for and carrying out sustainability activities. The evidence also recommends using frameworks and tools on an ongoing basis to help with sustainability planning and evaluation. Careful documentation of changes to the various factors outlined in this brief, along with the impact of these changes on outcomes, will help increase the knowledge base and enhance stakeholders’ sustainability activities.

 

ii A ninth domain, Public Health Impacts, was eliminated from the model after evaluation and is excluded from the Program Sustainability Assessment Tool (PSAT). Because this specific domain was eliminated, the authors conclude that the PSAT is a reliable scale for assessing sustainability capacity outside of Public Health program settings.

iii A seventh element, Understanding the Community, was eliminated from the Program Sustainability Index after preliminary evaluation.

iv Stirman et al.24 provide a framework for documenting adaptations to evidence‐based interventions that can help guide documentation of changes.

References

1. Stirman, S.W., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012). The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implementation Science, 7: 17.

2. Martin, G.P., Weaver, S., Currie, G., Finn, R., & McDonald, R. (2012). Innovation sustainability in challenging health‐care contexts: embedding clinically led change in routine practice. Health Services Management Research, 25: 190‐199.

3. Scheirer, M.A. (2005). Is sustainability possible? A review and commentary on empirical studies of program sustainability. American Journal of Evaluation, 26(3): 320‐347.

4. McHugo, G.J., Drake, R.E., Teague, G.B., & Xie, H. (1999). Fidelity to assertive community treatment and client outcomes in the New Hampshire dual disorders study. Psychiatric Services, 50(6): 818‐824.

5. Chaudoir, S.R., Dugan, A.G., & Barr, C.H. (2013). Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implementation Science, 8: 22.

6. Damschroder, L., Aron, D., Keith, R., Kirsh, S., Alexander, J., & Lowery, J. (2009). Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation Science, 4: 50.

7. Durlak, J.A., & DuPre, E.P. (2008). Implementation matiers: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41: 327.

8. Luke, D.A., Calhoun, A., Robichaux, C.B., Ellioti, M.B., & Moreland‐Russell, S. (2014). The Program Sustainability Assessment Tool: a new instrument for public health programs. Preventing Chronic Disease, 11: 130184. DOI: http://dx.doi.org/10.5888/pcd11.130184

9. Shediac‐Rizkallah, M.C., & Bone, L.R. (1998). Planning for the sustainability of community‐based health programs: conceptual frameworks and future directions for research, practice and policy. Health Education Research Theory and Practice, 13(1): 87‐108.

10. Pluye, P., Potvin, L., & Denis, J.‐L. (2004). Making public health programs last: conceptualizing sustainability. Evaluation and Program Planning, 27: 121‐133.

11. Maher, L., Gustafson, D., & Evans, A. (2010). NHS Sustainability Model. NHS Institute for Innovation and Improvement. Retrieved 20 December 2014 from www.institute.nhs.uk/sustainability

12. Doyle, C., Howe, C., Woodcock, T., Myron, R., Phekoo, K., et al. (2013). Making change last: applying the NHS Institute for Innovation and Improvement sustainability model to healthcare improvement. Implementation Science, 8: 127.

13. Chambers, D.A., Glasgow, R.E., & Stange, K.C. (2013). The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implementation Science, 8: 117.

14. Schell, S.F., Luke, D.A., Schooley, M.W., Elliott, M.B., Herbers, S.H., et al. (2013). Public health program capacity for sustainability: a new framework. Implementation Science, 8: 15.

15. Calhoun, A., Mainor, A., Moreland‐Russell, S., Maier, R.C., Brossart, L., & Luke, D.A. (2014). Using the Program Sustainability Assessment Tool to assess and plan for sustainability. Preventing Chronic Disease, 11: 13015. DOI: http://dx.doi.org/10.5888/pcd11.130185

16. Mancini, J.A., & Marek, L.I. (2004). Sustaining community‐based programs for families: conceptualization and measurement. Family Relations, 53: 339‐347.

17. Steckler, A.B., Goodman, R.M., McLeroy, K.R., Davis, S., & Koch, G. (1992). Measuring the diffusion of innovative health promotion programs. American Journal of Health Promotion, 6(3): 19‐33.

18. Scheirer, M.A. (2013). Linking sustainability research to intervention types. American Journal of Public Health, 103(4): e73‐e80.

19. Stirman, S.W. (n.d.). Sustainability research: A review of concepts, methods, and findings. Retrieved 17 December 2014 from http://www.hsrd.research.va.gov/for_researchers/cyber_seminars/archives/qir‐061412.pdf

20. Harburger, D.S., Stephan, S.H., & Kaye, S. (2013). Children’s behavioral health system transformation: one state’s context and strategies for sustained change. Journal of Behavioral Health Services & Research, 40(4): 404‐415.

21. Rog, D., Boback, N., Barton‐Villagrana, H., Marone‐Bennett, P., Cardwell, J., & Hawdon, J., et al. (2004). Sustaining collaboratives: a cross‐site analysis of the National Funding Collaborative on Violence Prevention. Evaluation and Program Planning, 27: 249‐261.

22. Stroul, B.A., & Manteuffel, B.A. (2007). The sustainability of systems of care for children’s mental health: lessons learned. Journal of Behavioral Health Services & Research, 34(3): 237‐259.

23. Manning, M.A., Bollig‐Fischer, A., Bobovski, L.B., Lichtenberg, P., Chapman, R., & Albrecht, T.L. (2014). Modeling the sustainability of community health networks: novel approaches for analyzing collaborative organization partnerships across time. TBM, 4: 46‐59.

24. Stirman, S.W., Miller, C.J., Toder, K., & Calloway, A. (2013). Development of a framework and coding system for modifications and adaptations of evidence‐based interventions. Implementation Science, 8: 65.

About This Brief

This evidence brief was developed by Andrea Flynn, PhD, with Evidence Exchange Network (EENet). The information is a summary of available evidence and is designed to give readers a starting point in considering the current evidence relating to the sustainability of interventions. It is based on select peer reviewed and grey literature.

To identify relevant literature, we used PubMed, PsycINFO, Google Scholar, and Cochrane Review search platforms. Search terms included “sustainability,” “sustainment,” “continuation” and “maintenance” in reference to “interventions,” “innovations,” and “programs.” The search was limited to English language publications.

Other evidence briefs prepared by The Provincial System Support Program at the Centre for Addiction and Mental Health (CAMH) are available at http://eenet.ca/products‐tools/?search_term=&cat_dropdown=0&pt_sub_=other&pt_cb=other.

While care has been taken in the preparation of the materials included in this publication, the search for relevant literature was not exhaustive due to time constraints. As a result, information relevant to this topic may have been unintentionally missed.

If you have any comments or feedback about this Evidence Brief, please contact us at eenet [at] camh [dot] ca.

Sign up for our Newsletter