Introduction
Program evaluation entails a systematic method for collecting, analyzing, and application of information to determine the progress and outcome of a program. The collection of the data ensures that the program implementers are guided by scientific evidence of what works and what does not work (Zellner, 2011). This is important in reducing unintended outcomes. Zellner (2011) noted the use of data to generate scientific evidence is a shift from past interventions, which were implemented based on advocacy, legal, and intuition considerations. Evaluations help in demonstrating program effectiveness to the stakeholders and assist in the design and implementation of future programs. Evaluations can be carried out using an objective-based approach or action research method. The following paper presents a critique of an evaluation based on an action research method performed for an employment retraining program for Ruby Lake Community. The paper provides an alternative method that focuses on the use of the objective-based approach.
Background of Ruby Lake Community
In the Ruby Lake neighborhood of Riverbend City, unemployment and underemployment are high because traditional manufacturing companies have been leaving the city. As a result, the members of the state Workforce Investment Board came up with a plan of action to implement a retraining program in order to ensure that the people acquire skills for advanced manufacturing jobs. The evaluation of the need assessment was carried using the action research method.
Critique of the Action Research Method
The evaluation of the program was based on action research. This is participatory action research that focuses on the study of the community, and co-operative inquiry action learning (Waterman, Tillen, Dickson & Koning, 2001). The action research ensured a participatory approach in which the needs of the key stakeholders were identified. According to O’Sullivan (2012) the action research allows collaboration of teams in which various stakeholders are involved in the program activities. The action research is contextual and action-based. It is widely used in intervention programs that target oppressed or minority groups.
The main idea of the action research is to design a program based on the collaboration of the community members who have experienced a problem first hand. For example, in the case of the Ruby Community neighborhood of Riverbend City, the main challenge has been unemployment. Thus, engaging the locals to participate in the design of the possible solution is based on an ideological assumption that the people have the facts that can help in drawing solutions. Waterman et al. (2001) argued that action research has an ideological bias based on the perception that oppressed persons can generate facts that have truths. In addition, the action research applied in the study did not have clear objectives. The lack of clear objectives negated the ability to carry out both the formative and summative evaluations. Furthermore, action research could lead to personal bias, which limits the objective view of the evaluation process. This may result in overlooking the main objectives of the evaluation objectives. Therefore, there is a need for objective-based evaluation.
Objective-Based Evaluation
This involves an evaluation approach that centers on the measurement of outcomes and the specification of objectives. According to Zellner, (2011) the main emphasis of the approach is to produce information that can be applied for accountability. In addition, the information generated in objective-based evaluation provides a platform for decision making in which the program implementers and evaluators can develop and measure appropriate objectives. Zellner (2011) posited that objective-based evaluation involves the formulation of objectives, classification of the objectives, definition of the objectives, and identification of the situations in which the objectives can best be applied. Thus, based on these provisions by Zellner(2011), the objective-based approach provides a new dimension for carrying out the evaluation, which was not captured in the action research approach.
The objective-based approach can be applied to perform both formative and summative evaluations. Formative evaluation entails the continuous evaluation of the progress of a program. It used to provide evidence on whether the program is achieving its set objectives. On the other hand, summative evaluation entails the appraisal of the program outcome. The appropriate objectives for the Ruby lake Community neighborhood of Riverbend City include:
- To examine the perceptions of the workers with respect to the training program and their expectations.
- To examine the skills inherent among the unemployed and underemployed workers in the Ruby Lake Community neighborhood.
- To find out how the impact of the retraining program in equipping the employees with the desired skills.
- To determine the overall effect of the retraining program in relation to the anticipated outcome.
The objective-based evaluation is a procedure applied to determine the extent to which a program is achieving the set objectives. The procedures used in the objective-based evaluations involve identification of the operational objectives. This is followed by the collection and analyzing the crucial information to determine how the objectives of the program are being achieved. It is worth noting that this approach includes both formative and summative assessments. In relation to formative assessments, the stakeholders of the program are tasked with finding out whether the program is continuing as planned. This relates to the achievement of short-term goals. On the other hand, the summative assessment is used to determine whether the ultimate goal of an intervention has been achieved.
In respect to the Riverbend City Missions, the approach provides an evaluation guided by the objectives. The procedures for carrying out the evaluation using the evidence-based approach will be different from the ones outlined in the two segments from the Riverbend City Missions media. For example, the new approach will be based on a scientific collection of information to gauge the achievements of the program. The collection will be in an objective manner and hence reduce the personal bias that is related to an active approach.
According to Dick and Greenwood (2015), the action research method generates information based on inquiries performed in a participatory nature. In addition, the procedure used in the two segments from the Riverbend City Missions media entailed the collection of background information about cities that have lost traditional manufacturing jobs and transitioned to advanced manufacturing jobs. This called for the creation of a literature map that formed the basis to carry out the program and the subsequent summative ad formative evaluations. This is contrary to the objective-based approach in which independent professional bodies are incorporated in the study.
Advantages of Objective Based Evaluations over Action Research
Even though, the action research provides first-hand information based on the experiences of the various stakeholders, the objective based study transcends the abstract views. As a result, the objective based methods have become prevalent in program evaluations. Zellner (2011) noted they provide a common sense appeal. This is because they utilize technologies of behavioral objectives and uses criterion-referenced testing and performance assessments. However, Zellner (2011) noted that the approach may result in crediting of unworthy objectives.
The objective based evaluation measures the progress and outcome of a program based on the laid down objectives. The approach does not consider the subjective outcomes and the participants’ differences. Hence, the results obtained are clear. Zellner (2011), noted that the evaluators apply the set objectives and operationalize them in order to measure the progress and outcome of a project. This provides the capability to classify, quantify and qualify the results in respect to the expectations. This is unlike the action research methods in which results are prone to subjectivity. The objective based approaches help the program implementers to determine the degree of effectiveness of the program activities.
The objective based approach applies set expectations and standards instead of subjective opinions. Therefore, the progress of the program can be tracked consistently to determine the strong points of a program and possible weaknesses that need to be improved. This aspect is unlike the participatory approach used in the assessment of the Ruby Lake community neighborhood of Riverbend City. According to Dick and Greenwood (2015), action research generates practical knowledge and creates new forms of understanding. The process involves people and social contexts intended to ultimately change lives for better. Despite this noble aspect of collaboration, the method does not objectively determine the progress of a program and the outcome. Therefore, the objective-based approach generates evidence regarding the program in a scientific manner and minimizes the personal bias that is related with action research methods.
Possible Changes to the Groups
The groups involved people from different social, legal and economic sectors of the Riverbend City Community. This ensured that the opinions of the different people were used in the need assessment and the subsequent implementation and evaluation of the retraining program. Despite the current constituents of the group that seems inclusive, I will make changes in order to bring other experts who will be important in the implementation and the subsequent evaluation of the program. The action research used in the program was critical in ensuring that learning was through action that led to personal and professional development.
The main goal was to solve the issue of the underemployment and unemployment caused by the departure of the traditional manufacturing firms. Waterman stated that “action research involves a spiral of self reflective cycles of, planning a change, acting and observing the process and consequences of the change, reflecting on these processes and consequences and then re-planning, acting and observing, reflecting and others “(p. 5). Based on this understanding, all the key stakeholders in a program should be included. In order, to ensure the program runs as expected, the groups should have clear stipulated responsibilities. Thus, in order to promote inclusivity and feasibility, I will include more participants.
The Changes and the Reason for the Change
I will include the government agencies that deal with employment, legislators and tertiary learning institutions, representatives. The government agencies and legislators will be critical in the evaluation. Their responsibilities will be to advise on the laws and policies that relate to employment and the manufacturing companies. In addition, the two entities will play a critical law in the program implementation process such as the formulation of education policies to effect changes on the curriculum in order to align the training curriculum with the requirements of advanced manufacturing in the community institutions. On the other hand, the tertiary learning institutions will play a two-fold role. First they will help in the analysis and evaluation of the program by application of scientific approaches. Secondly, they will participate in the training programs to equip the underemployed and the unemployed with the required skills.
Conclusion
Program evaluation plays a critical role in the success of any program. It is a valuable tool for managers who intend to strengthen and enhance the outcome of their programs. The main aim of program evaluation is to provide answers to basic questions that relate to the program. This relates to the effectiveness of the program. There are different approaches that are used to carry out program evaluations. Examples of the approaches include the action research method and the objective based evaluations. The action research entails the participatory methods in which the community is involved in the program processes i.e. from needs assessment to summative evaluation. In relation to the Ruby Lake Community neighborhood of Riverbend City, action research was applied. In the critique of the approach, it was established that it can result in ideological and personal bias. The paper has analyzed the issues that relate to the action research and provided an alternative approach, the objective based approach. The objective based evaluation is critical in determining the extent to which a program is progressing and achieving the set objectives.
References
Dick, B., & Greenwood, D. (2015). Theory and method: Why action research does not separate them. Action Research, 13(2), 194-197.
O’Sullivan, R. (2012). Collaborative Evaluation within a framework of stakeholder-oriented evaluation approaches. Evaluation and Program Planning, 35(4), 518-522.
Waterman, H., Tillen, D., Dickson, R. & Koning, K. (2001). Action research: A systematic review and assessment for guidance. Health Technology Assessment, 5(23), 1-9.
Zellner, G. (2011). A structured evaluation of business process improvement approaches. Business Process Management Journal, 17(2), 203-237.