It is important to note that overfitting is a major problem in data science. It takes place when “a statistical model fits exactly against its training data,” which means that algorithms are unable to perform effectively when faced with unseen data (IBM Cloud Education, 2021, para. 1). The issue is tied to the key features of Big Data, which are volume, velocity, and variety of data (McAfee and Brynjolfsson, 2012). The common method of detection of overfitting is k-folds cross-validation, where a series of evaluations are made on k-folds with scorings to assess performance (IBM Cloud Education, 2021). The approach is “more accurate than the training error, while not being overly computationally expensive as the deleted estimate, which is considered an unbiased estimate of the actual risk” (Abou–Moustafa and Szepesvari, 2018, p. 1). In other words, it is a potent and powerful method to test and analyze the general success rate of a model utilized for classification purposes (Marcot and Hanea, 2021, p. 2009). Therefore, consistent and consecutive assessments are made in order to measure the performance of an algorithm.
However, overfitting can be avoided by implementing a number of techniques. A critical analysis shows that these include early stopping, training with more data, data augmentation, and regularization (IBM Cloud Education, 2021). Some of the strategies are forms of supervised learning frameworks (Delua, 2021). The former refers to halting training before the noise is learned by the model, but this poses a risk of underfitting (Gupta and Sharma, 2022). Expanding the dataset is also effective, but the “strategy is proposed for complicated models to fine-tune the hyper-parameters sets with a great amount of data” (Ying, 2019, p. 1). Data augmentation is a careful method of adding noise alongside relevant data. It “can improve the performance of their models and expand limited datasets to take advantage of the capabilities of big data” (Shorten and Khoshgoftaar, 2019, p. 1). Lastly, regularization refers to reducing the number of features in a dataset (Provost and Fawcett, 2013). For example, Lasso regularization or L1 regularization can reduce data noise by penalizing input parameters with big coefficients (Xu et al., 2019). Thus, there are effective measures one can take to avoid overfitting.
An ideal organizational example would be IBM, which uses all forms of techniques to avoid overfitting. For instance, IBM uses ensemble methods to decrease variance but utilizes regularization for their complex models (IBM Cloud Education, 2021). However, in most cases, IBM simply trains its models with more data because the company can do so due to its infrastructure and access to information (IBM Cloud Education, 2021). Therefore, the techniques deployed by the company further support and illustrates the previously shown methods.
Reference List
Abou–Moustafa, K., and Szepesvari, C. (2018) ‘An exponential tail bound for Lq stable learning rules application to k–folds cross–validation’, Association for the Advancement of Artificial Intelligence, 1, pp. 1-12. Web.
Delua, J. (2021) Supervised vs. unsupervised learning: what’s the difference? Web.
Gupta, G. K., and Sharma, D. K. (2022) ‘A review of overfitting solutions in smart depression detection models’, 2022 9th International Conference on Computing for Sustainable Global Development, 2022, pp. 145-151.
IBM Cloud Education. (2021) Overfitting. Web.
Marcot, B. G., and Hanea, A. M. (2021) ‘What is an optimal value of k in k-fold cross-validation in discrete Bayesian network analysis’, Computational Statistics, 36, pp. 2009-2031. Web.
McAfee, A., and Brynjolfsson, E. (2012) ‘Big data: the management revolution’, Harvard Business Review, Web.
Provost, F., and Fawcett, T. (2013) Data science for business: what you need to know about data mining and data-analytic thinking. 1st edn. Sebastopol: O’Reilly Media.
Shorten, C., and Khoshgoftaar, T. M. (2019) ‘A survey on image data augmentation for deep learning’, Journal of Big Data, 6(60), pp. 1-48.
Xu, Q. et al. (2019) ‘Overfitting remedy by sparsifying regularization on fully-connected layers of CNNs’, Neurocomputing, 328, pp. 69-74.
Ying, X. (2019) ‘An overview of overfitting and its solutions’, Journal of Physics: Conference Series, 1168(2), pp. 1-6.