Systems Enhancement Evaluation Initiative

SEEi Research Studies

Dr. Paula Goering, Director of the Health Systems Research Consulting Unit (HSRCU) at CAMH, was the lead researcher on the Systems Enhancement Evaluation Initiative (SEEI). The HSRCU acted as the Coordinating Centre for the overall initiative and had two knowledge brokers coordinating the development of the network and SEEI communication and knowledge exchange activities.

Each of the SEEI Final Reports can be found by linking through to the SEEI Final Reports page.

The text below provides an overview of each of the nine SEEI studies.

The studies were rolled out in two phases:

Phase 1 Studies

The Health Systems Research and Consulting Unit at CAMH led the Phase 1 studies.

The Impact Study

Investigators: Dr. Janet Durbin and Dr. Elizabeth Lin

The Impact Study is a system-wide evaluation of the new funding entering the community mental health system in Ontario. The study is province-wide and will rely mainly on existing provincial health administrative data.

The project will address the following four questions:

  • What is the composition and capacity of the Ontario mental health system and how does it change as new resources are added to the system?
  • What system practices for organizing and delivering services are currently in place, and do they improve over time?
  • Is there a decline in use of hospital and criminal justice services by persons with mental illness?
  • Are better outcomes associated with enhanced systems of care?

The results will be reported at the LHIN level, and change in performance will be assessed at three points over time. The utility of understanding system performance from the perspective of local mental health service delivery systems within LHINs will be explored. An approach to measuring system integration will be developed and implemented.

Phase 1 of the study was completed by December 2006 with the publication of a Baseline Report.

The Report describes the status of the system before the implementation of the new funding and reports indicators of hospital and correctional facility utilization as well as mental health system capacity. Next steps, to take place through stakeholder meetings during Winter 2007, are to verify and refine the selected indicators, review local systems definitions within LHINs, and develop an approach for describing local system integration practices and identifying unique system challenges. In addition, 05/06 performance data will be calculated for the selected indicators, and change from baseline assessed.

The system evaluation project is being led by Drs Janet Durbin and Elizabeth Lin in the HSRCU, CAMH, and the Department of Psychiatry, University of Toronto. The team includes researchers from five other academic centres in the province. Each step of the project, from defining local systems to selecting performance indicators, to reviewing and interpreting results will be implemented in close collaboration with the project Knowledge Exchange Network and the field.

The Matryoshka Study

Investigator: Dr. Carolyn Dewa

The Matryoshka Project, will collect new information to obtain an in-depth picture of young people experiencing their first psychotic episode; and people with mental illness in contact with the criminal justice system. This study is being conducted in seven selected communities.

The purpose of the Matryoshka Project is to examine the effects of the investment of new funding into the mental health system on the continuity of care received by new and ongoing clients of the system. For the purpose of this evaluation, the focus is on five dimensions of continuity of care: (1) timeliness of service, (2) intensity of services, (3) comprehensiveness of services and (4) coordination of service provision and (5) accessibility. The project focus is on two types of specialized programs selected because the programs serve populations that are identifiable and, because Ministry funding was earmarked for both types of programs:

  • Those for young people experiencing their first psychotic episode
  • Court support programs for individuals with mental illness who have involvement with the justice system.

The Matryoshka Project is employing a mixed methods approach using both quantitative and qualitative methods and is examining three questions:

  • Does the continuity of care received and experienced by clients change in the period after new funding has been used?
  • What local system and program factors are related to the changes in continuity of care?
  • Is there a relationship between continuity of care and improved client outcomes?

The Matryoshka Project is led by Dr. Carolyn Dewa, in the HSRCU, and Associate Professor in the Department of Psychiatry at the University of Toronto. The research team is composed of a multi-disciplinary group of experienced researchers from the Centre for Addiction and Mental Health, the University of Toronto, the University of Western Ontario, the Centre for Research on Inner City Health at St. Michael’s Hospital and St. Joseph’s Hospital. Study partners also include the Regional Staff from the Centre for Addiction and Mental Health as well as the participating mental health programs.

Phase 2 Studies

The Phase 2 studies were conducted by other research teams in the province and supplemented the Phase 1 studies, allowing for more in-depth examination of other impacts to Ontario’s community mental health system. These studies were initiated in 2006 and ranged in duration from 18 months to 2 years.

An Evaluation of the Implementation and Outcomes of CMHA Ottawa’s Court Outreach

Investigators: Dr. Tim Aubry, University of Ottawa (PI), Dr. John Sylvestre, Ms. Donna Pettey and Ms. Marnie Smith.

Partner Organization: CMHA, Ottawa

This was an 18-month evaluation of a unique model of a court outreach or pre-trial diversion program, where intensive community support is integrated into the diversion process. The study evaluated the implementation and the outcomes of the program. The evaluation of program implementation was conducted using a “model-guided” approach assessing the extent the program was delivered as planned. The evaluation of outcomes was conducted on a group of 90-100 court outreach clients who had terminated with the program.

Waterloo Wellington Crisis System Evaluation: Understanding the Impact of Enhanced Programs and Coordination

Investigators: Ms. Eleanor Harder, Community Mental Health Clinic (Guelph) & Waterloo Regional Homes for Mental Health Inc. and Dr. Joan Nandal, CAMH (Co-PIs), Dr. Janos Botschner, and Mr. Paul Reeve.

Partner Organizations: Waterloo-Wellington LHIN-Wide Crisis Committee, which was comprised of: Waterloo-Wellington-Dufferin Self Help Alliance, Cambridge Memorial Hospital, CMHA (Waterloo), CMHA (Wellington-Dufferin), Community Mental Health Clinic (Wellington-Dufferin), Distress Centre (Wellington-Dufferin), Grand River Hospital (Kitchener), Guelph Police Service, Headwaters Health Care Organization (Orangeville), Homewood Health Centre (Guelph), Waterloo Region Assertive Community Treatment Team, Waterloo Region Homes for Mental Health, Waterloo Regional Police Service, and Wellington County Ontario Provincial Police.

This project entailed developing a system-wide program logic model and conducting an evaluation of the impact of the new and enhanced crisis support services on the quality of services and system capacity more generally. The work was undertaken in partnership with the Waterloo-Wellington LHIN-Wide Crisis Committee, which was comprised of 15 partner agencies involved in providing crisis support services. The investigative team included a mental health consumer, the crisis system manager/coordinator, and researchers.

An Evaluation of Community Based Discharge Planning

Investigators: Dr. Elsabeth Jensen, York University (PI), Dr. Cheryl Forchuk, Mr. Bill Seymour, Ms. Patty Chapman, Mr. Rick Boercamp, and Ms. Penny Whitcher

Partner Organizations: Bluewater Health and Sarnia-Lambton CMHA

This 18 month study evaluatee an in-reach community-based model of discharge planning designed to provide seamless care to people leaving hospital. In this model, community mental health staff meet with admitted clients to establish discharge plans. A total of sixty people were enrolled and evaluated at one and six months post-discharge.

Review Mobile Crisis Services

Investigators: Dr. Cheryl Forchuk, University of Western Ontario (PI), Dr. Elsabeth Jensen, Ms. Mary-Lou Martin and Dr. Rick Csiernik.

Partner Organizations: Lawson Health Research Institute, COAST Hamilton, St. Joseph’s Mountain Health Care Services

Three different models of crisis programs were examined using a case study approach and quantitative and qualitative methods. Each program and the communities being served were described. Frequencies of emergency room use and police calls were compared from a baseline point, if possible, and consumer and family satisfaction were assessed. The qualitative comparison involved research team members shadowing a team in each of the communities for a minimum of three shifts, taking note of key processes and the contexts of care provided. In addition, interviews with staff, managers, family members and consumers occured and, together, these data were used to provide a rich detailed description of each of the services provided.

An Evaluation of an Integrated Crisis-Case Management Service Model

Investigators: Dr. Terry Krupa, Queen’s University (PI), Mr. Alan Mathany and Dr. Heather Stuart

Partner Organizations: Frontenac Community Mental Health Services and Queen’s University

This 18 month study evaluated an integrated crisis-case management service model that was developed using the system enhancement funding from the Ontario Ministry of Health and Long-Term Care. The study used quantitative and qualitative methods to evaluate this new service with respect to accessibility to mobile services, the appropriateness of and accessibility to follow-up services (including reduction of hospitalization and hospital emergency use); and the acceptability of the new services to consumers and to the local network of community mental health and social services. Finally, the study developed a competency profile for the new transitional case manager positions.

Do Clients Receive More Appropriate Community Care and Fewer Hospital Resources Now That Community Mental Health System is Enhanced?

Investigators: Dr Heather Stuart, Queen’s University (PI) and Dr. Terry Krupa

Partners: Southeastern Ontario Mental Health Alliance

This study evaluated whether increased community mental health funding has improved the match between service needs and service use, and reduced hospital inpatient and emergency utilization using three approaches:

  1. We replicated a Comprehensive Community Assessment Project (C-CAP), originally conducted in 1999, which assessed the match between community mental health clients needs and the services they received.
  2. We followed both the 1999 and 2006 C-CAP samples through hospital administrative databases for a fixed period (1 year) to detail clients’ more immediate use of inpatient and emergency resources.
  3. We followed the clients enrolled in the 1999 C-CAP through the hospital databases for a full eight years until the end date of the study in 2007 to examine trends in hospital use and intensity to see whether improvements coincided with the known changes to the local funding base.

ACT Fidelity and Evaluation

Investigators: Dr. Lindsey George, McMaster University (PI), Dr. Sean Kidd, Ms. Gina Browne, Ms. Helen Kirkpatrick and Dr. John Sylvestre

Partner Organizations: Technical Advisory Panel for ACT (MOHLTC) (advisory group)

This was a two-year study of the ACT program in Ontario examining the degree to which a recovery focus and/or high fidelity to the ACT model effects client outcomes. The study answered five questions: (1) What aspects of recovery correlate with which expected outcomes? (2) Is fidelity to the ACT model significantly related to the provision of recovery-oriented services? (3) Does fidelity correlate with expected outcomes in the Ontario experience? (4) To what degree do ACT teams in Ontario achieve fidelity? (5) How useful is the Recovery Self Assessment Tool in discriminating amongst ACT teams with respect to the degree they are recovery focused?

Project Contact: 

eenet [at] camh [dot] ca