The Bridge Drop-In Center’s Program Evaluation

Introduction

Various social assistance programs, especially within the framework of the provision of mental health problems assistance, require evaluation to assess the effectiveness of interventions and their potential changes. Peer-run programs have unique characteristics and require professionals to pay attention to the main elements that distinguish these programs from traditional ones. This evaluation considers peer-run in-drop center The Bridge, offered by Hinds Behavioral Health Services. To evaluate the effectiveness of this program, the Fidelity Assessment Common Ingredients Tool (FACIT) is used, and an evaluation method based on a logical model is also proposed. Based on the data obtained, a number of recommendations are made to improve the effectiveness of the program and potential changes in the proposed interventions. Additionally, the timeline of the evaluation process and its budget justification are described.

Evaluation Studies Examination

When evaluating mental health programs, first of all, it is necessary to take into account their type, which largely affects the compilation of assessment items and the structuring of the interview. The researchers note that peer-run support programs have a number of distinctive characteristics that must be considered when choosing tools for their evaluation (Chinman et al., 2016; Gillard et al., 2021). In particular, the basis of peer-run services, including drop-in centers, is the integration of “an individual with personal experience of using mental health services is trained and employed to explicitly use that experience in supporting others currently using services (their peers)” (Gillard et al., 2021, p. 1903). In particular, such services are built on a peer-to-peer relationship based on shared experience, which is very different from the relationship between a professional and a client. The main task of peer specialists is to involve clients in treatment, illustrate the role model, as well as to assist in accessing the supportive community (Chinman et al., 2016). Thus, the basis for peer-run evaluation is how effectively peer specialists are selected, trained, and involved in the rehabilitation process of clients based on interaction with them and participation in group and individual therapy sessions.

However, evaluating the direct performance of peer specialists is not always easy since there is no unified idea of ​​what qualities and skills should be attributed to these professionals. Chinman et al. (2016) note that the Assertive Community Treatment (ACT), also offered by the US Department of Helath and Human Services (2008), can be used to evaluate peer-run services. This assessment tool is based on the examination of the fidelity community program within the framework of peer-run interaction. In particular, “fidelity scales have helped identify the critical components that have been associated with outcomes for models like Assertive Community Treatment” (Chinman et al., 2016, p. 3). Thus, the fidelity scores proposed for evaluating community mental health programs are the most relevant for peer-run programs.

Fidelity scores were developed on the basis of more general tools such as the Assertive Community Treatment (ACT), which allowed to take into account the main unique features of peer-run programs. In particular, such evaluation items as the effectiveness of the work of peers of specialists and customer satisfaction with interaction with them were included. US Department of Helath and Human Services (2008) offers the Fidelity Assessment Common Ingredients Tool (FACIT), which includes an evaluation of the institution’s services on multiple items, as well as interviews of members of the institution and its clients regarding the program being evaluated. In particular, the protocol contains 28 items that are required to rate as part of the program evaluation. The scoring protocol contains different categories that focus on human resources, organizational boundaries, as well as nature of services. Within each item, there are also scores from 1 to 5 that reflect the effectiveness of one or another item of the protocol evaluated as part of the program evaluation.

Most importantly, the protocol primarily evaluates the competence and number of staff members when interacting with program clients. In particular, the experience is evaluated, as well as the availability of a psychiatrist and a nurse assigned to work with clients. The protocol also includes an assessment of such indicators as responsibility for the services provided, the definition of the mission and target population of the program, as well as the absence of time limits for the provision of assistance. The protocol also explores the nature of the services, including a focus on community integration, as well as a focus on the treatment of certain disorders, including substance abuse. In general, this program evaluation system is aimed specifically at community services, which is relevant within the framework of this practice. Most importantly, the protocol presented also includes an assessment of the administrative and personnel aspects of the program to identify potential structural constraints to service delivery.

The researchers also note that the Fidelity Assessment Common Ingredients Tool (FACIT) was designed specifically to evaluate peer-run services. Basto (2017) emphasizes that peer-run evaluations were made for the development of the tool within the framework of the preliminary project drop-in centers, support groups, and educational programs. Poses based on the data received were for the development of assessment items, as well as “semi-structured interviews administered individually or in a group format with members, staff, and managers” (Basto, 2017, p. 45). The development of the tool was based on a search for commonalities that characterize different peer-run services. When determining the validity of the created tool, it was also found that “nearly 75% of the items were found in the programs, suggesting the tool has a degree of content validity” (Basto, 2017, p. 45). It is also noteworthy that when evaluating non-peer-run traditional mental health services using this tool, significant differences were found, which identifies the relevance of the Fidelity Tool specifically for peer-run programs.

The basis for program evaluation using FACIT is a comparison of the goals of the services provided and their effectiveness. Suhrheinrich et al. (2020) explain that the “measurement is necessary to demonstrate the relationship between the application of the treatment… and its effect on the targeted behavior” (p. 30). Thus, tools such as FACIT allow the identification of major program successes and problems during evaluation, which can later positively influence the development of appropriate interventions. The main advantage of this tool is the focus on organizational and personnel aspects, which allows you to adjust the administrative interaction for the delivery of a more efficient service. At the same time, this tool focuses on a limited range of services, including solely evaluating the effectiveness of providing mental health support through peer-run community programs. However, in general, it can be used to effectively assess initiatives such as drop-in centers, which include interaction with peer specialists and participation in the supportive community.

Thus, FACIT is the most relevant tool for evaluating peer-run mental health programs. First of all, it was developed on the basis of already well-developed tools, including ACT, which makes it valid and reliable. Additionally, research was conducted to identify significant differences between traditional mental health community programs and peer-run ones to account for all possible influencing factors. Additionally, there are currently similar protocols proposed by the Substance Abuse and Mental Health Services Administration (SAMHSA) that allow assessments to focus on delivering mental health rehabilitation services.

Program for Evaluation

The program selected for evaluation is The Bridge, offered by Hinds Behavioral Health Services. This program is a peer-run drop-in center that provides a variety of services for the rehabilitation of homeless individuals with severe mental illness. First of all, at the drop-in center, customers can receive basic needs services, including food, shelter, clothing, laundry, and mail services. The long-term goal of the program is to provide clients with access to housing, treatment, and recovery support. In particular, the center provides individuals with support to restore mental health, develop communication skills, find employment, find housing, and reintegrate into the community. The program includes the integration of peer professionals who have relevant experience and can participate in supportive activities as mentors to build client social skills such as self-confidence, self-advocacy, and support systems. The services of the center are available to all those in need and can also be chosen by them on their own. All activities of the center involve voluntary participation and have the main goal of helping the poor with serious mental health problems to reintegrate into society.

Evaluation and Data Collection Methods Application

As noted, FACIT is the most relevant tool for evaluating peer-run programs. The protocols for this tool are suggested by the US Department of Helath and Human Services (2008), and evaluation sheets are also included in the study by Basto (2017). As part of the evaluation, it is necessary to evaluate the program using the scores included in FACIT, as well as on the basis of semi-structured interviews with the administration, staff, and clients of the program. Based on the data collected, it will later be possible to determine the goals and resources of the program for developing a program outcome evaluation of social work practice with a logic model.

To determine the evaluation of the program within the FACIT scale, it is necessary to rank the program according to the following criteria in three domains:

• Human resources: Structure and composition

  • Small caseload (clients to provider ratio);
  • Team approach (focus on the group rather than individual assistance and the creation of a supportive community environment);
  • Program meeting (regular meetings for planning interventions);
  • Practicing ACT leader (provision of services directly by members of the program administration);
  • Continuity of staffing (maintaining the same staff members);
  • Staff capacity (full staffing operating);
  • Psychiatrist on the team (at least one full-time psychiatrist per 100 clients);
  • Nurse on the team (at least two full-time nurses per 100 clients);
  • Substance abuse specialist on the team (at least two full-time specialists per 100 clients with one-year clinical experience in substance abuse treatment);
  • A vocational specialist on the team (availability of at least two specialists with vocational rehabilitation and training);
  • Program size (availability of sufficient staff to provide full services coverage).
• Organizational boundaries

  • Explicit admission criteria (a well-defined mission to provide services to a specific population);
  • Intake rate (low intake rate to ensure service stability);
  • Full responsibility for treatment services (direct providing of psychiatric services, counseling/psychotherapy, housing support, substance abuse treatment, employment, and rehabilitative services);
  • Responsibility for crisis services (24-hour responsibility for covering psychiatric crises);
  • Responsibility for hospital admissions (involved in hospital admissions);
  • Responsibility for hospital discharge planning (involved in planning for hospital discharges);
  • Time-unlimited services (graduation rate) (providing a constant opportunity for clients to contact the program if necessary).
• Nature of services

  • Community-based services (work on the development of science in the community rather than in the office);
  • No dropout policy (high percentage of clients);
  • Assertive engagement mechanisms (using os street outreach and legal mechanisms whenever appropriate);
  • The intensity of service (high total amount of service time, as needed);
  • Frequency of contact (high number of service contacts, as needed);
  • Work with an informal support system (work with the client’s support network, including family, friends, community members);
  • Individualized substance abuse treatment (1 or more team members providing substance abuse disorder treatment);
  • Co-Occurring disorder treatment groups (using of group modalities for treatment);
  • Dual Disorders (DD) Model (considering the interaction of mental health and substance abuse);
  • Role of the client on the team (involving clients as team members for direct services provided).

Each of the presented aims also has a score from 1 to 5, where the highest value identifies a high degree of program compliance with the stated criteria. The basis for filling out this score sheet is an interview with the administration, staff members, and clients of the program, as well as organizational documents that can become a source of information. Based on the collection of data on program scores within FACIT, it can be determined that staff policy and integration within the program is the weakest point. This is due to the fact that the program is primarily peer-to-peer interaction, which involves attracting clients as direct service providers. However, to implement more efficient service delivery, the organization needs to focus on recruiting more professionals, including psychiatrists and nurses, and training peer specialists.

The main aspect for collecting relevant information for evaluation is interviews with members of the organization and clients on the presented items. They are also presented as part of the FACIT evaluation and are the most reliable source of data. Questions should be addressed to both the administration and staff of the program in order to obtain a complete picture and an objective assessment. The interview questions that form the basis of data collection are presented below:

1. In a typical 2-week period, what percentage of consumers see more than one team member?
2. How often does the administrative team meet as a full group to review services provided to each client?
3. How many clients are reviewed at each meeting?
4. Do you provide direct services to clients?
5. What percentage of your time is devoted to direct services?
6. What is the total number of staff positions on the program team?
7. Name the team members who have left in the past two years.
8. Have you had anyone who has been on leave for more than one month during the last 12 months?
9. What is the psychiatrist’s role on the team?
10. What is the nurse’s role on the team?
11. Does your program team have a clearly defined target population with whom you work?
12. What formal admission criteria do you use to screen potential clients?
13. Who makes referrals to the program team?
14. Are there circumstances where you have to take clients onto your team?
15. What recruitment procedures do you use to find clients for the program team?
16. Do you have some program clients who you feel do not need the intensity of ACT services?
17. How many new clients have you taken on, per month, during the last six months?
18. Do your consumers see other psychiatrists outside of the program team?
19. Do any consumers live in supervised group housing?
20. What is the nature of rehabilitation services?
21. What are 24-hour emergency services available for program consumers?
22. What is the program team’s role in providing 24- hour emergency services?
23. How often is the team involved in the decision to admit clients for psychiatric hospitalization?
24. Describe the process the team goes through when clients must be admitted to a hospital.
25. How many of the clients have you graduated because they no longer needed services?
26. What percentage of program consumers are expected to be discharged from their team within the next 12 months?
27. Does your team use a level or step-down system for clients who no longer require intensive services?
28. How many clients dropped out during the last 12 months?
29. For the clients who have moved, what efforts did the program team make to connect them to services in their new location?
30. How often do you close cases because clients refuse treatment or you lose track of them?
31. What factors does the team consider when closing a case?
32. What methods does the team use to keep consumers involved in the program?
33. What happens if consumers say they do not want your services?
34. Among clients with whom you have had at least one contact with their informal network in the last month, how frequently does the team have contact with the client’s informal network?
35. On average, how often do you work with the family, friends, or other informal support network members for each client?
36. How many clients have a substance-use disorder?
37. Of these clients, how many received structured individual counseling for substance use from the substance abuse counselor on the team or another program team member this last month?
38. How many treatment and rehabilitation groups are offered?
39. How many clients attend these groups?
40. What is the treatment model used to treat clients with substance abuse problems?
41. Do you see the goal as abstinence?
42. How are clients involved as members of your team?
43. If they are paid employees, are they full-time?

To conduct a comprehensive evaluation and assessment of the program within the framework of FACIT scores, it is also necessary to conduct interviews with program clients. They should be asked to answer the following questions:

1. Who have you seen from the professional team this week? How about last week?
2. Do you see the same person over and over or different people?
3. How often do you see the team psychiatrist?
4. Do you use the program team psychiatrist for medications?
5. How often do you see the team nurses?
6. Who helps you get your services for housing? For employment?
7. Who helps you besides the program team?
8. Where do you see people from the program team the most?
9. How often do you go to the program office?
10. What happens if clients say they do not want the program services anymore?
11. How many times have you seen program team members during the past week?
12. How often is there contact between the program team and your family? Your friends?
13. How are other clients involved as members of your team?

The questions presented are to obtain the necessary information to determine the FACIT scores of the program, as well as to draw up a plan for further interventions development. The most significant factor in this situation is also the use of available data to evaluate statistical information on some items. Thus, comprehensive work with documents and employees, as well as program clients, is necessary for appropriate evaluation.

Program Outcome Evaluation of Social Work Practice with a Logic Model

A logic model is a tool that is used to plan interventions based on creating logical connections between input and output. Input, in this case, describes the resources necessary for the implementation of the planned interventions. This includes personnel, time/effort, space, finances, supervisory, technology, materials, and other resources necessary to deliver the intervention (Openshaw et al., 2011). Output, in turn, describes the changes that the intervention is expected to bring in after implementation.

The logical model is used to achieve the following goals:

1. Defining the intervention during the evaluation process;
2. Explaining the reason for intervention activities;
3. Developing crucial points for intervention results evaluation;
4. Communication with stakeholders.

When using the logic model to evaluate a program, inputs, outputs, and measures for short-term and long-term outcomes should be included to determine the effectiveness of interventions. It is also necessary to clarify the purpose or mission of the program, as well as its intermediate tasks, which is the basis for the formation of a logical model. The main task of the logical model is to consider how all parts of the program function together, as well as to determine the necessary and unnecessary elements to optimize operations. Tab. 1 presents a possible logical model for evaluating The Bridge Drop-In Center based on the effectiveness of the services provided by the program.

Fig. 1. The logic model for The Bridge Drop-In Center evaluation

Thus, the proposed logical model is based on a comparison of the effectiveness of the utilization of available resources for the program and the outcomes associated with them. As part of peer-run programs, the greatest focus should be on the integration of clients into direct service delivery, which is the basis for building a rehabilitation community. Additionally, it is necessary to pay attention to the involvement of social workers and mental health specialists in the work within the framework of the program, which determines the level of professionalism. This indicator is also important when considering the effectiveness of training peer professionals to provide effective therapeutic support to clients. The proposed model contains the necessary measurements that are indicators of the effectiveness of the use of a particular resource and can help identify program gaps.

Interpretation of Evaluation Data Results

The evaluation procedure described, including the FACIT tool and the logic model, should give an idea of ​​how effective program interventions are at the moment and what potential changes are needed. Within the framework of the interview, rather limited information was collected about the administrative structure and activities of the program, which makes it difficult to conduct a comprehensive evaluation. However, based on the data obtained, several aspects can be identified that are important results of the procedure.

First of all, it is important to note that the program meets all the unique characteristics of the peer-run program, which makes the chosen evaluation methods relevant. It should also be noted that the program offers not only mental health treatment with a focus on substance abuse disorders but also basic needs services as well as rehabilitation and social reintegration services. Such a wide range of services needs a more versatile evaluation, which was not taken into account in this study. However, within the scope of the mental health services provided, it should be noted that the program puts emphasis on the active involvement of clients in direct service provision. This aspect allows for creating a community within the framework of the treatment program, which has a positive effect on the possibility of building social skills, as well as the prospects for the reintegration of clients into society.

It is also important that the program also involves individual work with clients to assess their needs. In particular, within the program, there are options for both group and individual therapy, which identifies a variety of therapeutic approaches. Additionally, within the framework of the evacuation, it was revealed that the administration and employees are involved more in the formation of the community than in the direct provision of services. This aspect identifies that the administration is developing interventions in such a way as to form a supportive environment based on peer-to-peer communication and assistance. Within the framework of peer-run programs, this factor is the most significant, as it determines the goals of the project and its potential results.

Evaluation makes it difficult to assess the long-term outcomes that the program offers to its clients. This aspect is due to the fact that clients often apply for repeated help or continuation of treatment but rarely report their further success in rehabilitation. However, a high level of satisfaction with the services identifies that the program is effectively achieving its goals. However, there is an aspect of concern that was identified as part of the evaluation. In particular, the program administration notes that participation in therapeutic interventions for clients is voluntary and does not impose certain obligations on them. In this regard, there is a risk that many clients are attracted solely by the possibility of meeting basic needs, while mental health and rehabilitation work is of lesser priority.

Thus, one of the recommendations for the program is to develop criteria for admission to the program, as well as the conditions for being in it. Undoubtedly, this step will reduce the number of clients and, in the long term, participants in the program. However, this recommendation will primarily achieve a higher percentage of rehabilitated clients, as well as create a more stable and determined community to help others. It is also important that this aspect can attract more significant investors to the program to expand the financial capabilities of the organization.

Evaluation Timeline

The evaluation timeline includes the need to prepare, collect data, interpret data, and generate evaluation results. Preparation for the evaluation process includes the need to notify all interview participants of the interview and prepare the necessary materials for the interview. This stage is expected to take three days, but the timeline varies depending on the availability and presence of employees and customers. During the data collection phase, interviews should be conducted with employees, members of the administration, and clients of the program. This stage takes two days, but the duration may also increase depending on the availability of the respondents. At the stage of interpreting the results, it is necessary to correlate the data obtained in the interview with the items presented in the FACIT score, which will take another day. Finally, a report on the results of the evaluation must be prepared, which will take another two days. Thus, the entire program evaluation process will take from 8 to 10 days, depending on the availability of interview respondents.

Budget Justification

The budget for the evaluation includes only the cost of remuneration of the professional conducting it. All interview materials can be presented to participants in electronic format, which does not require additional costs. Moreover, the entire evaluation process can take place remotely, which does not involve spending additional resources. Thus, the budget of the evaluation process is minimal and does not cost any funds for its admission.

Conclusion

Examination of available research in the field of evaluation of mental health community programs has identified that the application of the FACIT model is the most effective for assessing peer-run programs. The use of this framework, as well as the logic model for the evaluation of The Bridge program, made it possible to identify the need for changes in this program. In particular, the main recommendation is the need for changes in the process of accepting clients into the program and the conditions for their maintaining in it. Stronger client eligibility and program entry requirements will allow the organization to build a more stable community, as well as potentially attract investors to expand financial opportunities. In the long term, this move will allow better assistance to be provided through more professional peers and determined community members.

References

Basto, P. M. (2017). Validity of the Fidelity Assessment Common Ingredients Tool: Empowerment and satisfaction [Doctoral Dissertation]. The Faculty of Rutgers, The State University of New Jersey. Web.

Chinman, M., McCarthy, S., Mitchell-Miland, M., Daniels, K., Youk, A., & Edelen, M. (2016). Early stages of development of a peer specialist fidelity measure. Psychiatric Rehabilitation Journal, 39(3), 256-265.

Gillard, S., Banach, N., Barlow, E., Byrne, J., Foster, R., Goldsmith, L., Marks, J., McWilliam, C., Morshead, R., Stepanian, K., Turner, R., Verey, A., & White, S. (2021). Developing and testing a principle‑based fidelity index for peer support in mental health services. Social Psychiatry and Psychiatric Epidemiology, 56, 1903-1911.

Openshaw, L. L., Lewellen, A., & Harr, C. (2011). A logic model for program planning and evaluation applied to a rural social work department. Contemporary Rural Social Work Journal, 3(1), 40-49.

Suhrheinrich, J., Dickson, K. S., Chan, N., Chan, J. C., Wang, T., & Stahmer, A. C. (2020). Fidelity assessment in community programs: An approach to validating simplified methodology. Behavior Analysis in Practice, 13, 29-39.

US Department of Health and Human Services. (2008). Evaluation your program: Assertive Community Treatment. US Department of Health and Human Services. Web.