Introduction
Data warehousing and Big Data have both been designed to facilitate business analytics. Data Warehousing is an established information management concept that is championed by vast and well-established methodologies. However, Big Data is still a paradigm under development, which aims to address individual aspects of the considerable data volume challenge but does not have an integrated solution.
Some research portrays Big Data as a Data Warehouse evolution, others as its replacement; some recommend extending Data Warehouse to support some Big Data features and the probability of merging the two. However, it is essential to note that Data Warehouse overrides Big Data on data quality; hence, some organizations that do not want to compromise data quality and governance are reluctant to shift. This paper aims to differentiate between Data Warehouse and Big Data in relation to their structure, principles, and mode of action.
Data Warehousing
Components of a Data Warehouse (DW) architecture
A DW is an information environment that contains data obtained from multiple sources and stored in a uniform schema. The information present is used to perform queries and analyses, hence enabling improved decision-making. DW has different architectures that comprise centralized, federated, hub and spoke, and independent data marts (Ariyachandra & Watson, 2008). Architecture refers to the proper arrangement of components in a system.
A DW is usually built with software and hardware components that are arranged in a specific manner to suit the organizational requirements and for maximum benefits. As a result, a typical DW usually consists of the following basic components: data source, data staging, data storage, information delivery, metadata, and management and control. DWs in every organization utilize the same building blocks; however, the difference lies in how they are arranged, and thus some components might be made to be stronger than others (Ariyachandra & Watson, 2008).
In the source data component, the electronic information entering a warehouse is obtained from several systems. It can also include internal data, archived data, and external data. This is followed by the data staging component that entails the preparation of extracted data to be stored for querying and analysis. The methods of preparation include extraction, transformation, and loading (ETL). During extraction, it is essential to ensure that the most suitable techniques are employed for each data source. This is because the data might be from several source machines that are in different formats. Data transformation is a critical step in the full web data integration process.
Moreover, depending on the integration process, data may be required to be cleaned, merged, deduplicated, converted, and summarized for storage and use. This is followed by data loading that entails two distinct processes. When going live for the first time, large volumes of data is loaded into the repository for a significant amount of time. As the data repository continues functioning, the variations in the data source are continuously extracted, transformed, and the incremental data revisions are fed to the repository on an ongoing basis.
In the data storage component, data for the warehouse is separately kept from that of operational systems as it has to be stored in a form suitable for analysis, that is, read-only. Therefore, when analysts perform an evaluation, they are assured that it is stable and represents snapshots at particular periods. The fourth component – information delivery – comprises different methods such as complex queries, Executive information systems (EIS) feed, multidimensional analysis, and data mining. This information is essential for novice users as it enables them to perform complex analyses.
The fifth is the metadata component which is referred to as the data catalog in a database management system. It contains information regarding the logical data structures, files and addresses, and indexes among others. Last is the management and control component which sits on top of all other building blocks. This is because it coordinates services and activities in the DW.
Current key trends in data warehousing
DW has ceased to be a purely new area for study and implementation. It has become mainstream and every business, regardless of their sizes. DW has transformed business analytics and how it is used in decision-making. In 2017, the global data warehousing market was the highest ever in the BFSI segment and was valued at $18.61 billion. Moreover, it is expected to increase at a CAGR of 8.2% from 2018-2025 (Gaul, 2019). The reasons for its growth are attributed to the heightened need for an efficient repository system for the increasing data volume and the demand for low-latency, real-time view, and analytics for BD. However, high implementation costs adversely impact the expansion of the data warehousing market, especially for small and medium-sized enterprises.
Currently, North American companies have dominated the data warehousing market share, and approximately 90% of multinational corporations have committed to data warehousing (Gaul, 2019). Furthermore, in relation to the market segment, the on-premise deployment dominated the cloud and hybrid because of the presence of conventional IT infrastructure in most of the companies (Gaul, 2019).
Big Data
Understanding of Big Data (BD) and how it is applied
BD refers to large volumes of structured and unstructured data that cannot be stored or processed by conventional computing techniques within a given duration (Grable & Lyons, 2018). It purposes to reveal hidden patterns.
BD has three main characteristics: volume, velocity, and variety. Therefore, based on these three dimensions, BD can be said to contain the high volume, velocity, and variety of data assets that require innovative and cost-effective processing techniques to enhance decision-making. Volume is defined as the sheer scale of processed information. Today, the volume of data collected from people by organizations continues to grow and is existing in petabytes.
It possesses the greatest opportunity, as BD could enable corporations to understand people better and allocate resources more efficiently. However, the conventional techniques by relational databases are not scalable to deal with data of this magnitude. On the other hand, velocity is defined as the rate by which information flows into organizations. Users are increasingly demanding real-time data, which can prove to be a challenge for conventional analytics since data is in constant motion. Lastly, variety comprises the data types to be processed. BD is characterized to contain both structured and unstructured raw data formats.
The significant data variety cannot be categorized and processed by traditional analytical methods. Other characteristics of BD include veracity, which focuses on ambiguity, in other words, noise and abnormalities within data. The presence of a vast volume of data makes it challenging to differentiate between essential data and distractions.
To deal with the challenges of traditional analytical techniques in regards to the three main dimensions of BD, researchers have designed “predictive analytics.” The power of predictive analytics is based on the development of learning algorithms that identify patterns having predictive power (Grable & Lyons, 2018). As a result, BD overcomes the challenges of traditional methods and enables organizations to gain insight and make well-informed decisions from a large volume of data. Currently, the most publicized areas of BD use are in the retail industry to understand consumers’ behaviors and preferences.
Organizations are dedicated to expanding their traditional data with other information from social networks and browser logs to have a clear and comprehensive picture of their clients. For example, Walmart can predict which products will sell. Walmart is a global retailer, and thus to efficiently conduct its operations, it created an analytics hub known as the Data Café. The Data Café allows petabytes of information to be quickly modeled, manipulated, and visualized.
Demands BD is placing on organizations and data management technology
Although BD has several benefits to various sectors, it is a promising technological innovation with the capacity to alter data management technology and organizations in many ways. For instance, currently, organizations that have employed BD have already restructured. Proctor & Gamble has built control towers to maintain the continuously updated control of its supply chain (Slinger & Morrinson, 2014).
Companies are restructuring themselves as BD allows flexible resource allocation; hence, enterprises are finding it easier to move employees, capital, and other resources across roles, sites, and positions. If this continues at the enterprise level, BD might change organizational structure because large corporations will restructure to add a structural dimension. Some new functions will integrate BD operations that will differentiate themselves from the rest of the organization.
Moreover, BD might change the source of influence in an organization. For instance, HiPPOs (Highest-Paid People in the Organization) states that their judgment-based decision-making has sometimes been overruled by data-driven decision-making (Slinger & Morrinson, 2014). Intriguingly, Slinger and Morrinson (2014) also claim that the restructuring of organizational structures will result in the generation of new tensions, and the most successful enterprises will be those that effectively and efficiently handle the conflicts of interest.
In regards to data management technology, the availability of BD in analyzing large data volumes has shifted the business towards data-driven decision-making. Organizations and vendors are continually researching new tools and models to improve the current BD utilization. New models and concepts are continuously appearing in the market, while the old ones are fading away. There are presently efforts to employ the Internet of Things to merge streaming analytics and machine learning. Usually, machine learning uses stored data for training in a controlled learning environment.
However, in the new model, the availability of streaming data will enable the provision of real-time data to facilitate learning in a less controlled environment. Furthermore, there are efforts to use Artificial Intelligence (AI) in processing BD. This is considered a significant improvement as it will lead to the faster and efficient gathering of business intelligence.
Conclusion
Information is a valuable primary resource of enterprises as it supports decision-making. With the advancement of technology, the volume of data is substantially increasing hence posing challenges of increasing complexity to the storage, updating, and efficient exploitation. DW and BD are new concepts that have been designed to mitigate these challenges. However, the DW is more of a conventional approach, and its efficiency has been hampered by the ever-increasing volume of data. This led to the development of BD. Nevertheless, the scope of BD is not clearly understood, and it still lacks a standardized proposal; therefore, its future success or failure cannot be predicted.
References
Ariyachandra, T., & Watson, H. J. (2008). Which data warehouse architecture is the best? Communications of the ACM, 51(10), 146-147. Web.
Gaul, V. (2019). Data warehousing market by type of offering (Extraction, Transportation & Loading (ETL) solutions, statistical analysis, data mining, and others), type of data (Unstructured and semi-structured & structured), deployment (On-premise, cloud, and hybrid), organization size (Small & medium sized enterprises and large enterprises), and industry vertical (BFSI, telecom & IT, government, manufacturing, retail, healthcare, media & entertainment, and others): Global opportunity analysis and industry forecast, 2018 – 2025. Portland, OR: Allied Market Research. Web.
Grable, J. E., & Lyons, A. C. (2018). An introduction to Big Data. Journal of Financial Service Professionals, 72(5), 17-20.
Slinger, G., & Morrinson, R. (2014). Will organization design be affected by BD? Journal of Organization Design, 3(3), 17-26. Web.