Introduction
Program planning and evaluation are critical steps in determining the success and anticipated effectiveness of the program, which ultimately influences the support of the program by its stakeholders and sponsors. To illustrate the general components of the program for Improving Literacy in the Seattle School District, the logic model application allows for outlining the primary inputs, activities, outputs, outcomes, and assumptions of the program (Carroll et al., n. d.). The logic model presented in Figure 1 depicts the essential components of Improving Literacy in the Seattle School District Program.
Program Evaluation Plan
Program Overview
The Improving Literacy in the Seattle School District Program is designed to meet the community needs for reading skills improvement in the elementary school student if the Seattle District. Initiated as part-private/part-public project, the program titled “Team Read” involves public elementary schools and attracts private entities and professionals to implement the program tasks. The program is funded by the Alliance for Education and is planned as a sequence of one-year-long reading courses. The sessions are held twice a week, last for one hour, and are conducted by a group of coaches (tutors) who are high-school students from the district’s schools. The objectives the program sets forward include the improvement of elementary school students’ reading skills, namely in such domains as fluency, comprehension, and vocabulary, as well as high-school students’ (tutors’) improved skills of tutoring, accomplishment, responsibility, and community work. The overall feedback from coaches, teachers, and students implies satisfactory results of the program. However, evidence-based, objective evaluation of the program is necessary to identify the strengths and weaknesses, as well as possible areas for improvement.
Key Questions for the Evaluation
Two essential issues are to be taken into consideration when defining the key questions for evaluation. They include stakeholder interests and the program’s goals. As stated above, the program’s objectives are elementary school students reading skills improvement and high-school students’ work experience (Centers for Disease Control and Prevention [CDC], 2017). The stakeholders within the program for Team Read are the founders Craig and Susan McCaw, Seattle School District Staff, Alliance for Education program representatives, community representatives, school principals and teachers, elementary school students, and coaches. Based on these two aspects, the following evaluation questions are formulated:
- What are the effects of the program on the educational outcomes of elementary school students?
- How does the work as tutors within the program affect high-school students’ outcomes?
- How does the program benefit the community?
- Are the program’s steps implemented as planned?
These questions aim to retrieve evidence about the program’s key components to identify its overall success, effectiveness, and necessity of improvement.
Proposed Evaluation Design
The identified evaluation questions set a particular direction for the evaluation process by implying the character of data that needs to be collected, interpreted, and analyzed. The mixed design is applied to combine the qualitative and quantitative data collection methods for better analysis of the program’s effectiveness and efficiency. Due to the overall qualitative nature of the inquiry, the qualitative approach to the program evaluation is chosen as the dominant one, although several quantitative data collection methods will be applied.
In addition, the process evaluation is applied to identify the in-progress results achieved by the program and to enable adjustments and changes to the program plan on the basis of the identified inconsistencies or drawbacks. According to Dahlberg and McCaig (2010), the application of qualitative methods for program evaluation allows for obtaining numerous advantages. Importantly, the collection and analysis of qualitative data provide an opportunity to explore under-investigated areas of the researched issue, explain the issue in detail, access the information in the most relevant manner, and apply sensitivity principles when tackling delicate problems (Dahlberg & McCaig, 2010). Thus, the choice of qualitative design is beneficial for the proposed program evaluation.
Moreover, when considering the costs for qualitative design in program evaluation, it is often assumed that this approach requires inadequate cost allocation. However, as Dahlberg and McCaig (2010) clarify, the costs are related to the time an evaluator spends on collecting and analyzing data and is not imposed by the equipment or software expenses. At the same time, the quantity of data that might be collected by means of qualitative design contributes to the credibility of evidence and the reliability of findings.
On the other hand, the quantitative approach in this mixed design will allow for collecting a more objective and numerical data that will measure specific outcomes of the program. The quantitative approach allows for obtaining a high level of objectivity, preserving participants’ anonymity, and efficiency of costs due to the minimized analysis and data collection efforts. Thus, the combination of the two approaches, qualitative and quantitative, with the dominant role of qualitative method will allow for finding credible and trustworthy answers to the evaluation questions.
Two Indicators of Success of the Program: Definition and Measurement
Based on the tasks pursued by the program and the stakeholder interests, the two main indicators of the program’s success are effectiveness and efficiency. The effectiveness indicator is of a multifaceted nature given the context of program activities implementation and the scope of participants. Indeed, since the program aims at increasing the level of outcomes for readers, coaches, and the community, the effectiveness measurement is applied accordingly. Firstly, the effectiveness of the reading sessions for elementary school students is an important contributor. Secondly, the improvement of high-school students’ work-related skills and qualities is an important element of effectiveness identification. Thirdly, the community outcomes connected to the program’s contribution are valuable. Thus, the indicator of effectiveness is crucial to the evaluation procedure.
Also, the indicator of efficiency is vital to address in the process of the program evaluation. The results obtain in one year are capable of demonstrating how efficiently the resources are used and whether the program goals are met in a timely manner. Therefore, it is essential to measure the compliance of the program activities with the plan. Also, the expenditures of funds need to be analyzed against the budget to assess the efficiency of the costs-related decisions. Finally, the time and efforts of the coaches and program coordinators need to be evaluated against the criterion of efficacy to identify whether these resources are allocated and distributed economically and productively.
Evaluation Implementation and Data Collection
For the process of the program evaluation to be feasible, the methods for data collection and analysis are chosen out of practicality, accessibility, and cost-effectiveness considerations. The implementation of the evaluation procedure will involve outside evaluators’ as well as program coordinators’ efforts in collecting and analyzing data. Both qualitative and quantitative data will be retrieved from the participants and from the documentation to determine how effective and efficient the program for improving literacy is.
To address the first evaluation question concerning the effect of the program on the educational outcomes of the elementary school students, the methods of questionnaires, participant observation, testing, and semi-structured interviews will be used (Dahlberg & McCaig, 2010). Firstly, the two questionnaires will be conducted with two separate groups of participants, including reading students and their school teachers. The questionnaires with children will be held under the guidance of the evaluator to ensure adequate answering the questions and the participants’ comprehension of the data that is expected to be provided. The questions will be targeted at obtaining students’ feedback on the quality of sessions and the reflection as per their improvement in reading skills. As for the collecting of data from teachers, a self-completion questionnaire will be used. According to Dahlberg and McCaig (2010), self-completion questionnaires are appealing for gathering large numbers of responses for relatively low costs and are especially suited to simple questionnaires” (p. 161). The results of questionnaires will show the perceived outcomes of the program and its effect on the direct recipients of the services.
Secondly, reading tests will be conducted for elementary school students to assess their reading skills against the common criteria used in public schools. The results of these tests will be compared with those of the pre-program period, and the differences will be identified. This method of data analysis will allow for objective evaluation of the Team Read’s outcomes. In addition, the participant observation method will be used to observe the reading students during the reading sessions and determine the level of quality of their work with the tutors (Dahlberg & McCaig, 2010). Finally, semi-structured interviews will be conducted with the Seattle School District’s teachers and principals to identify their perception of students’ outcomes after the year in the program.
When addressing the second evaluation question related to the effects of the program on the high-school students who work as reading coaches at Team Read, self-completion questionnaires will be conducted. The questions will concern the perception of working with children, the contribution to work experience, the skills and knowledge acquired in the process of program implementation, and the professional orientation outcomes. Also, the method of the focus group will be applied to discuss the most tentative issues related to the reading sessions, including class management, material accessibility, skills improvement, training quality, and other issues (Dahlberg & McCaig, 2010). Importantly, the coaches’ practice-informed opinions about the required improvements to the program activities will be brought to discussion to obtain relevant information as per the efficiency of process organization.
To investigate the third evaluation question, semi-structured interviews with community representatives will be conducted. The questions asked within the interviews will be aimed at identifying the effects of the Team Read on the community. The collected data will include the demographic characteristics of the student readers participating in reading sessions, the benefits of the community related to the engagement of the children in the project, as well as possible concerns on the community level related to the program. These data will allow for identifying how effective the program meets the interests and expectations of one of its main stakeholders, the community of the Seattle School District.
Finally, to evaluate the compliance of the program activities and results with the plan, the methods of documentation analysis and SWOT analysis will be used. These data will demonstrate the efficacy of resource utilization, as well as the advantages and possible drawbacks in the program implementation. The documentation analysis will involve the review of activities plan and budget, as well as reports on the first year completion. The comparison of these documents will help to detect possible inconsistencies and provide an opportunity for adjustment in the future. Also, the SWOT analysis method will be applied to identify strong and weak aspects of the program, threats to its success, and opportunities for increasing its effectiveness and efficiency.
Possible Challenges to Data Collection
Given the complexity of the planned evaluation procedure, the multiple stakeholders involved, and several evaluation questions addressed, several challenges are anticipated. Since the program serves elementary school children, the part of the data collection procedure involving students’ responses to questionnaires might be relatively problematic. Additional efforts will have to be made to organize the process and guide the participants through the process; there is a possibility of bias due to the young age of the students. The access to school-based documentation and materials might be difficult to obtain due to the administration’s possible mistrust toward the program-related activities.
Also, the process of retrieving information from coaches might be challenging due to the high rate of their drop-out of the program. Since a part of the data collection process (questionnaire and focus group) is reliant on the contribution and active participation of the high-school students working as tutors within the Team Read project, their presence or accessibility is essential. Moreover, it is important to include the tutors who have worked since the beginning of the program to diversify the scope of reflection on the program’s effectiveness. Also, the costs of this multifaceted evaluation procedure might be high due to the scope of targeted questions, the number of participants, and the diversity of applied tools.
Anticipated Ways of Overcoming Challenges
When anticipating the enlisted problems that might obstruct the completion of the evaluation or impose significant credibility concerns, the following measures for overcoming the challenges are proposed. The children-imposed bias and the difficulty in collecting data during the questionnaire with the elementary-school students might be eliminated by means of a guided questionnaire conducted by a program evaluator. The difficulty of obtaining access to school documentation on students’ educational outcomes might be resolved by establishing effective communication with administration as well as clarifying confidentiality issues. It will increase the level of trust and enable evaluators’ access to significantly important data. The issue of high-school students’ drop-out might be overcome by establishing contact via phone and the Internet. As for the cost concerns, several solutions might be introduced. Firstly, the questionnaires should be conducted online with the active utilization of the method of self-completing questionnaires. Thus the time and efforts of the evaluators will be saved. Secondly, volunteer work is encouraged to minimize personnel expenses. Overall, the planned evaluation is expected to provide an evidence-based foundation for credible data analysis and will demonstrate the effectiveness of the program from the perspective of key stakeholders.
References
Carroll, S., Cerreto, J., Erickson, E., & Maanibe, G. (n. d.). Understanding, developing and evaluating logic models. Web.
Centers for Disease Control and Prevention. (2017). A framework for program evaluation. Web.
Dahlberg, L., & McCaig, C. (Eds.). (2010). Practical research and evaluation: a start-to-finish guide for practitioners. Sage.