In a multi-site empirical study on cross-organizational Enterprise Resource Planning (ERP) size and effort estimation, Daneva (2008) identifies some limitations and potential threats to validity and reliability of the research findings. The first major threat to external validity identified is that ERP projects that the study focused on might not be representative of the entire population of cross-organizational ERP projects. To address this threat, the author mentions that a replication study will be undertaken in the Netherlands to make the findings more generalized for all ERP projects. Besides, the author plans to gain an understanding of the size and effort estimation norms that are specific to new ERP projects.
Daneva also asserts that the key threat to internal validity of the findings arises from ‘alternative explanations’ for the observations on the size and effort estimation practices. However, to ensure that the research had evidence that what ERP adopters and consultants observed as common were also identified by external analysts, the research team included some representatives of ERP market research companies. This strategy for improving validity will be implemented in all future studies.
In a second ERP study, Niu et al (2011) undertake a case study of exploiting ERP requirements. The authors identify internal validity to the study findings as relating to establishing a causal relationship, whereby certain conditions are shown to lead to other conditions. The main source of data for the study was interviews using surveys and questionnaires. Due to geographic limitations, the researchers were unable to observe the actual developments on site. Therefore, participants in the study may have omitted vital facts when answering questions, or the researchers may have misrepresented them. To eliminate this source of uncertainty, the researchers intend to undertake observational field studies. Another threat to internal validity of the data was bias of the respondents during the survey processes since some questions were stated in a relative way. A solution to this threat to validity was made by reformulating some questions in an absolute way with inverse Likert-scale polarity to guide future investigations.
External validity of the research findings in the study related to establishing the domain to which a study’s findings can be generalized. Consequently, the researchers point out that the findings might not be applied beyond the research model used (AIM RE). This model was chosen by the partner company and the research team had little control over it. They propose that further studies are needed to examine the effects of other models such as Accelerated SAP (ASAP) or Baan’s Dynamic Enterprise Modelling. Niu et al also note that the pilot projects that incrementally adopted ERP package that was used for the study. The authors propose further analysis on this issue.
References
Daneva, M. (2008).Preliminary Results in a Multi-site Empirical Study on Cross-organizational ERP Size and Effort Estimation. Software Process and Product Measurement, Volume 4895/2008, 60-71.
Guillemin, M. and Gillam, L. (2004). Ethics, Reflexivity, and “Ethically Important Moments” in Research. Qualitative Inquiry, vol. 10 no. 2: 261-280.
Niu, N., Jin, M., and Cheng, J. C. (2011). A case study of exploiting enterprise resource planning requirements. Enterprise Information Systems, Vol. 5, No. 2: 183–206.