Summary of the evaluation plan
Evaluation is a very imperative process in the execution of any project/program. The ARTReach program is an 8-week community arts program that is meant to improve the way of living of marginalized young women by instilling life skills and capacity in them. Similar studies with varied activities encompassed in an arts’ program have shown positive results and the same is expected from this study. The evaluation of this program will focus on the effect that the program will bring among the marginalized young women. The objectives of this evaluation will be to determine any change related to self-awareness, artistic skills and economic gain. In order to capture the element of economic gain, the evaluation process will take place six months after the training. The study design to evaluate any change among the women after the training has been administered will be pre-post. Indicators to measure these outputs include completed assignments, designed artefacts, attrition rates, and monthly expenditure. Filling in questionnaires and conducting interviews will be the means through which information will be collected.
Description of Health Program
Goals and objectives
The main goal of the outreach health promotion program is to empower marginalized young women between the ages of 16 to 25 years with life skills and capacity through art therapy techniques. One of the program’s objectives is to give the women a chance to express significant scenarios in their lives through creativity. Other objectives of the program include guiding participants through self discovery, group interaction, taking part in community activities, education, and life skills.
Program strategies
The ARTReach health promotion program offers 4, eight-week programs in a year. Therefore, a single program will run for 8 weeks. The program targets ill women, women who are disabled, unemployed, abused and assaulted, drug and substance abusers, young mothers, or at risk of dropping out of school at an early age. This program intends to change the lives of these women by impacting knowledge and skills that can help them better their lives. The main stakeholders of this program are four community partners within the Geelong region (YWCA 2012).
Brief literature review of evaluations of similar
Various studies on the evaluation of community arts promotion programs show success and mainly involve a varied approach to determine effect on participants. However, debate on the social usefulness of arts is ongoing (Kelaher et al. 2012). It is through the lens of the affect of arts on health that the role of art on social capital, social inclusion, civic engagement, and the development of community networks has been deemed significant (Stern & Seifert 2002). In a study by South (2006) on “Community arts for health: an evaluation of a district”, the community arts health promotion program was aimed at promoting health and wellbeing of disadvantaged communities. It included varied types of activities. The study design used for evaluating this program that was carried out in an urban district in the UK was cross-sectional. Those interviewed were 16 implementers from various organizations and the program was found to be a success. In an evaluation of community partnerships for human rights program targeting Muslims as a marginalized community in Australia, pre-post evaluation method was used. This program was a 3 year program that aimed at increasing the awareness and understanding of human rights issues, decreasing discrimination against this Muslim community, and promoting social inclusion and participation. The program encapsulated various projects and among them was a community arts and culture initiative with Muslim Australians. This initiative was meant to give the Muslims a chance to integrate into the community by understanding the art and culture defining the surrounding communities (Wutzke 2012).
Kelaher et al. (2012) study on evaluating community outcomes of participation in community arts: A case for civic dialogues uses a mixed approach. This study aimed at assessing the effects of community arts on civic dialogue through use of interviews and audience evaluations. It was assumed that through the arts program, there would be increased awareness of self and social issues hence, provoking civic dialogue. The finding showed success of the program.
A study by Wright et al. (2008), the effect of the community arts program on the youth was also determined using a mixed approach. Evaluation was on the effect of age-specific art on youths’ conduct and emotions, task completion, art skills development, and pro-social skills development. This study showed positive results in as far as youth development was concerned. Kay and Watt’s (2000) study on role of arts in regeneration did not use a particular evaluation approach. However, the valuation process entailed conducting interviews, focus group discussions, and filling in questionnaires. Positive effects of a community art program in relation to regeneration could be attained if it were part of an integrated programme. Yoon (2008) has reviewed various evaluation models and among them is evaluating community arts and community well being using an Arts Victoria model. This model provides information to community arts practitioners and build on to knowledge for the community arts sector. In addition, arts organizations are able to select and evaluate sections of community arts projects that are relevant to them. The evaluation model entails examining 22 criteria. This evaluation model is limiting with regard to data collection methods and equipment.
The need for evaluation
It is very clear that all these studies have aimed at evaluating both knowledge and behaviour for change to be identified. Most of the community arts programs focus on effect of this program on the health of the participants. There is a dearth of literature that shows an evaluation of such a program on artistic skills of participants, as well as on their socio-economic status. This evaluation project therefore will seek to further establish the artistic skills obtained by the marginalized young women as well as how these artistic studies have impacted their lives. This is something that has not been addressed in other prior studies
Evaluation Strategy
An impact evaluation to determine the effectiveness of the program will be carried out.
Evaluation questions or objectives
The objectives of the evaluation plan will seek to establish the impact of the program on self-discovery and relationship with others, artistic skills, as well as on the participants’ economic status.
Description of the evaluation design including: what is the design?
A mixed-method study design will be used to determine this (Fraser et al. 1995). Both qualitative and quantitative data will be collected.
Rationale for the evaluation design
In comparison to other evaluation approaches, the pre-post, which is part of the mixed method design used to evaluate the ARTReach program, is less expensive and gives valid results. The mixed method approach is associated with increased validity because of the different dimensions used for assessment (Fraser et al. 1995, p. 223). Randomized controlled study design is considered a gold-standard but it is associated with limitations like ethical concerns and very expensive (Nutbeam & Bauman 2006). In addition, in a community set up like this one, it is difficult to execute the RCTs because the community might not understand why one group should be given the intervention and not the other, yet they all believe that they all need the intervention. In a clinical setting, a placebo is usually given and therefore participants do not feel discriminated against. In a community, the participants and especially those receiving no intervention might drop out leaving the study with no control group. A cross-sectional study on the hand is limiting compared with the pre-post in giving causality. A mixed method study design has a stronger effect in showing causality compared with other study designs used on their own. Therefore, as of now, it remains as the ideal approach to use in evaluating this particular ARTReach program.
Data collection methods
Sample
The target population of the program is marginalised young women between the ages of 16 to 25. Apparently, four community partners are the means through which the study population will be obtained. Out of the women who will be referred by the community partners, a sample will be chosen to represent the other women who will take part in the program. The sample size will be determined using the percentage role of thumb (Zamani-Gallaher 2011). Data will be collected using questionnaires and interview schedules as discussed above.
Data collection instruments
Questionnaires will be used to evaluate the level of self-awareness and also possession of artistic skills. The questions to evaluate self-awareness will be indirect for example in assessing for assertiveness, instead of asking are you assertive, the questions will be framed something like, “How do you cope with peer pressure?” In addition, there will be a scale of 1 to 5 to supplement this. For example, “How often do you seek the approval of other people for you to do something?” 1=never, 2=rarely, 3=sometimes, 4=often, 5=always. To assess possession of artistic skills, questions like, “Have you ever developed an artefact or engaged in any work of art?” Socio-economic status will be assessed using questions like, “What is your highest level of education?”
Observation checklist will be used to list down the observations made. Completed artefacts and indicators for living standards of the women will be listed. The artefacts will be listed and will be indicators to supplement acquisition of artistic skills. Observation of the living conditions of the young marginalized will augment the socio-economic questionnaire used.
Procedure
Questionnaires will be administered twice with regard to self evaluation and assessment of acquisition of artistic skills and socio-economic status. As for artistic skills, in addition to filling in the questionnaires, completion of assigned tasks will be assessed through observation. In addition, a follow-up after six months will be conducted to assess whether the artistic skills gained had a life-changing effect on the participants. This will include assessing whether the participants continued to make good use of the artistic skills they gained and if they did, whether it economically uplifted these participants as a form of income generating activity. It is because of this follow-up that a socio-economic status assessment will be done prior to the training.
To reduce the occurrence of bias and increase validity, the baseline characteristics of all participants in terms of level of education and socio-economic status will be the same. Before executing the evaluation method, consultations with the stakeholders of the ARTReach community art health promotion program will be carried out. This will promote a participatory approach among the stakeholders.
At the beginning of the training program, the women will be given questionnaires with a set of questions aimed at assessing the level of self-awareness within the group. After the training the same questionnaires will be administered to the same women assuming that no woman will have dropped out of the program. It will be assumed that after the training and increased awareness on self-discovery, the perceptions of participants will change. Therefore, the second set of answers will be statistically different from the first.
Assessment of artistic skills will take a similar approach where at the beginning of the training the participants will be given a questionnaire to evaluate their artistic skills. At the end of the training, the same questionnaires will be given to the participants and a statistical difference between the two sets of questionnaires will be derived. Observation will be used to supplement the questionnaire for assessing design of artefacts by the participants within and outside the workshop training ground within the eight weeks. Interviews with the participants will be held to determine how the training would have an impact on their lives.
Since this evaluation entails an economic element, the participants will be followed up six months after to determine the effect of the training program on their lives because that was the whole intention of the program. During this time, interviews will be carried out with the participants to determine how the skills obtained from the community arts program transformed their lives.
References
Fraser, E, Bryce, C, Crosswaite, C, McCann, K & Platt, S 1995, ‘Evaluating health promotion: doing it by numbers’, Health Education Journal, vol. 54, pp.214-25.
Kay, A & Watt, G 2000, The role of the arts in regeneration, Scottish Executive Central Research, Edinburgh.
Kelaher, M, Berman, N, Dunt, D, Johnson, V, Curry, S & Joubert, L 2012, “Evaluating community outcomes of participation in community arts: A case for civic dialogue”, Journal of Sociology, pp. 1-18.
Nutbeam, D & Bauman, A 2006, Evaluation in a nutshell: a practical guide to the evaluation of health promotion programs, McGraw Hill, N.S.W.
South, J 2006,”Community arts for health: an evaluation of a district programme”, Health Education, vol. 106, no. 2, pp. 155 – 168.
Stern, MJ & Seifert, SC 2002, Culture Builds Community Evaluation: Summary Report, Social Impact of the Arts Project, University of Pennsylvania, Philadelphia, PA.
Wright, R, Lindsay, J, Alaggia, R, Duku, E & Tanya, M 2008, “Do Community Arts Programs Promote Positive Youth Development?” Critical Social Work, vol. 9, no. 1, Web.
Wutzke, S 2012, The Evaluation of Community Partnerships for Human Rights Program: Working with and for Muslim Communities, Web.
Yoon, N 2008, “Evaluation of Community Arts Projects: Comparison among three program evaluation models”, Master’s Dissertation, University of Oregon.
YWCA 2012, ARTReach-100% goddess, YWCA, Geelong.
Zamani-Gallaher, EM 2011, “Evaluating with Small Sample Sizes”, ATE PI Conference, Web.