The Sample Employed
This paper essays a review of data-collection methods and their associated reliability. In one, the team of researchers employs a compact meta-analysis of published literature to survey the evidence for best nursing practices concerning mechanically-ventilated patients in intensive care units (Couchmana, Wetzig, Coyerc, and Wheeler, 2007). In the second study, Meltzer, Steinmiller, Simms, Li, and Grossman (2009) set out to investigate whether and how pediatric patients and their parents who proved difficult versus placid in a complex medical care situation influenced the degree of “engagement” and attention provided by attending physicians and nurses.
Couchman et al. justify their reliance on literature review because the available literature is disparate, “fragmentary” and frequently fails to give due consideration to both patient safety and comfort. In general, medical researchers may employ literature review when doing exploratory research or embarking on mixed-methods programs of study.
Given the nature of this research approach and the assertion of Houser (2007) that reviews of the literature bolster claims about the importance of the topic, the “sample” in this case is the universe of studies published over a decade – between 1996 and 2006 – which seems both comprehensive and recent enough to have didactic value for day-to-day evidence-based nursing practice.
In the case of the Meltzer et al. (2009) study on complex pediatric medicine, the sample consisted of 68 physicians and 85 registered nurses at The Children’s Hospital of Philadelphia. The participants engaged in a case study method, reviewing and rating their involvement in eight made-up patient record vignettes.
Bias in Sample Selection
As Couchmana et al. (2007) describe their search strategy for professional journal articles, the first part of the research that focused on patient safety (Part II, published separately, dealing with patient comfort) embarked on a comprehensive search for “current literature” in the electronic databases MEDLINE, CINAHL, EMBASE, and Psych-Review. The search was filtered by these terms: mechanical ventilation, patient assessment, airway management, sedation, and comfort. Since little work had been done in the area of safety for mechanically-ventilated patients, no exclusions were set for the 12 years 1995 to 2006 and the team, therefore, gathered what was effectively a census of all available literature.
The Meltzer et al. (2009) sample amounted to a sampling rate of 7.5% and 4%, respectively, from what was in effect the universe of 899 attending physicians and 2,231 registered nurses at this large tertiary care institution. As empirical studies go, such a sampling rate is comparatively high, yielding as it does an ±8.0% standard error of estimate at the 95% confidence level and given common assumptions about the proportion to be tested and the margin of error.
On the manner of sampling selection, however, the study seems less a systematic random survey and a more convenient, self-selected sample. While Meltzer et al. took care to diversify recruits across the typical medical division and inpatient units such as Cardiology, Critical Care Medicine, Endocrinology, General Pediatrics, Hematology, Oncology, and Pulmonary Medicine, the manner of respondent selection within each “stratum” depended on volunteering in the course of a regularly scheduled department or research meeting. This affects external validity to the extent that not every attending physician or nurse at Children’s Hospital of Philadelphia had a chance to be selected systematically.
Limitations in Collecting and Managing Data
Calling even the seven dozen-odd sources finally included in the study “scant”, Couchmana et al. (2007) supplemented findings from the literature review with peer and expert review. This is to be expected when the state of knowledge in a primary care area is not much more than exploratory.
The limitations that apply to secondary research (which is what a review of literature entails) chiefly consist of accessibility. That is, the authors Couchmana et al. could have missed unpublished clinical trials and pilot-scale assessments. Given the incentives for publication and the comprehensive nature of contemporary online databases, the likelihood of having missed something has to be very low.
The rationale for Measurement Strategies and Devices
Strictly speaking, the rationale for embarking on a review of literature is intrinsic to exploratory or mixed-methods studies. In the case of Couchman et al., the team had aimed to consolidate the varying aspects of care for mechanically ventilated patients. Secondly, the authors did take care to synthesize their findings according to the patient safety/comfort dichotomy and with due consideration to three clinical survey stages. The “primary survey” involved an examination of airway, breathing, circulation, disability, and exposure. A “secondary survey” tasks nurses with assessment by system: neurological, respiratory, cardiovascular, gastrointestinal, metabolic, renal, and skin integrity. Thirdly, there are emergency equipment and safety checks. The consolidated literature review reiterates for nurses, often on 1:1 assignment ratios with critically ill patients, just how intensive clinical practice has to be.
In the complex pediatric medicine study, the researchers justify the use of short clinical vignettes principally on the basis that ‘‘complex care patients’’ are subject to multi-regimen care (and hence, inter-specialty teamwork), is hampered with behavioral or developmental issues, and present with psychiatric diagnoses as well. The clear need for psychosocial support must therefore be filled by those who come in contact with them daily, i.e. physicians and nurses. The core research question then becomes how both professional types respond to the most difficult of patients and their families.
The stimulus device ran to eight vignettes, each featuring a combination of patient and family being cooperative or not, concerning either treatment or the health care staff, and length of stay. Participants rated these in random sequence, to avoid order-of-presentation bias.
Face/internal validity was enhanced by pre-testing the stimulus vignettes in focus groups of 10 attending physicians and 10 experienced RN’s.
Validity and Reliability of the Measurement Strategy
Since the work by Couchmana et al. was meant to be a compendium for clinical practitioners and their superiors, validity, and reliability are not called to question beyond the competence of clinical practitioners to correctly synthesize what seems a thorough literature review.
The Mertzel et al. research employed case studies in the form of vignette/scenarios that described short- versus long-stay patients (30 days being the cut-off) and their parents who were either cooperative or difficult. On reviewing each one, participants rated themselves on a scale from highly engaged/responsive to distancing/disconnected behaviors vis-à-vis the patients and their families.
Given that the study participants were both physicians and nurses and that one analysis cross-break focused on the differences in mean ratings between them, the study design was a 2 X 2 X 2 matrix. This means that the independent variable of engaged/distancing rating was hypothesized as an outcome of both professional level and patient type.
Threats to Internal and External Validity
For didactic purposes, one must nevertheless address considerations of internal and external validity of the research reported. In the case of the literature review approach, internal validity can be taken for granted owing to the universal access to electronic journal databases and to the presumption of professional competence of clinical practitioners such as Couchmana et al. A researcher wanting to subsequently replicate the findings on mechanically-ventilated patients need only employ the very same keywords. Anybody else with comparable diligence, professional and clinical training ought to arrive at the same conclusions (Houser, 2007; Paradis, 2008).
On the other hand, the generalizability or external validity of the findings in the literature review rests on the diversity by locale and the clinical status of patients reported in the literature.
Owing to the haphazard distribution of the case study vignettes during departmental meetings at Children’s Hospital of Philadelphia, external validity for the Meltzer et al. study is flawed. That is, one lacks confidence that the study can even be generalized to the population of physicians and nurses with privileges at, or employed by the hospital. On the other hand, convenience sampling involving patients one has ready access to is a fact of life in medical research.
More important perhaps is the fact the measurement strategy is flawed for relying exclusively on self-ratings. Surely, internal validity would have been immeasurably enhanced by including engagement feedback from patients and their families.
References
Couchmana, B. A., Wetzig, S. M., Coyerc, F. M., Wheeler, M. K. (2007). Nursing care of the mechanically ventilated patient: What does the evidence say? Part I. Intensive and Critical Care Nursing, 2007 (23), 4-14.
Houser, J. (2007). Nursing research: Reading, using, and creating evidence. Boston: Jones and Bartlett Publishers.
Meltzer, L. J., Steinmiller, E., Simms, S., Li, Y. & Grossman, M. (2009). Staff engagement during complex pediatric medical care: The role of patient, family, and treatment variables. Patient Education and Counseling, 74, 77–83.
Paradis, C., MD (2008). Bias in surgical research: Internal and external validity. Annals of Surgery, 248 (2):180-188.