Determining the validity and reliability of the research instruments when reading a research study
When reading and assessing a quantitative research study, it is paramount to determine the validity and reliability of the instruments utilized in that study to understand whether the findings of that study can be trusted.
The validity of an instrument (construct validity) assesses the accuracy of that instrument (Cozby & Bates, 2015, p. 73); it determines whether the instrument measures what it is supposed to measure. If an instrument is not valid, it means that its results cannot be trusted; for instance, a ruler is not a valid instrument for measuring weight, so a hypothetical study measuring weight with a ruler cannot be trusted to produce adequate results (Cozby & Bates, 2015).
The reliability of an instrument estimates whether it produces similar results if used to measure the same thing in non-changing conditions (Cozby & Bates, 2015). If a tool is reliable, then it measures the magnitude in the question precisely; but if it is highly unreliable, the measurement will be very inaccurate, which also makes the results not adequate.
How does the validity and reliability of an instrument influence the credibility of the study and the value of the evidence?
The validity of an instrument has a crucial impact on the credibility of a study and the value of the evidence. High validity means that the instrument measured what it was supposed to measure, which is a necessary condition for the credibility of a study, whereas evidence gained via non-valid instrument may simply be misleading (Cozby & Bates, 2015).
The reliability of an instrument is also paramount for the credibility of a study and the value of the evidence. If an instrument is unreliable, it means that it will give non-precise results of measurements. This might mean, for example, that an otherwise significant difference between two values may not be detected by a statistical test due to the increased variance of data resulting from unreliable instruments only (O’Dwyer & Bernauer, 2014). This makes a study using non-reliable instruments non-credible and diminished the value of the evidence.
What characteristics should be appraised when evaluating qualitative and quantitative research design?
When evaluating a quantitative research study, it is needed to appraise such characteristics as (Cozby & Bates, 2015; O’Dwyer & Bernauer, 2014):
- Measurement reliability;
- Construct validity;
- External validity (generalizability);
- Replicability (the possibility to repeat the study in question and gain similar results);
- Rigor (the degree to which a study is careful and exact);
- Objectivity (the degree to which the conclusions and findings based on facts rather than the emotions of the authors).
However, when assessing a qualitative study, different criteria should be evaluated. These criteria are less well-defined because qualitative studies by definition are less rigorous and standardized. These criteria include (“Qualitative Validity,” n.d.):
- Credibility (the degree to which the results are believable from respondents);
- Transferability (generalizability);
- Dependability (taking into account the constantly changing context);
- Confirmability (the degree to which a study can be confirmed in other studies).
Which characteristics are the most important and why?
It might be possible to state that among the named desired characteristics of quantitative research, construct validity, rigor, objectivity, and measurement reliability are the most important ones: without construct validity, the study will measure a different phenomenon; without rigor or objectivity, the conclusions might be false; without measurement reliability, the gained results may be very imprecise, and the conclusions – false. On the other hand, external validity seems less important because the results of a study with low external validity still may be true, although limited to a very small group; replicability also seems less important because if all the other criteria are satisfied, the results of non-replicable research still might be true (at least theoretically).
As for qualitative research, credibility appears important because qualitative research is often, by its nature, supposed to reflect the views of respondents; dependability is also pivotal because qualitative studies often matter in a particular context, and changing context might completely change the situation (Maxwell, 2013). Transferability seems less important because qualitative research often attempts to study a situation in a very particular context, so, unsurprisingly, some results might be non-generalizable. Confirmability also appears less important because unique contexts are hard to replicate in other studies.
References
Cozby, P. C., & Bates, S. C. (2015). Methods in behavioral research (12th ed.). New York, NY: McGraw-Hill Education.
Maxwell, J. A. (2013). Qualitative research design: An interactive approach (3rd ed.). Thousand Oaks, CA: SAGE Publications.
O’Dwyer, L. M., & Bernauer, J. A. (2014). Quantitative research for the qualitative researcher. Thousand Oaks, CA: SAGE Publications.
Qualitative validity. (n.d.)