The ETL process which stands for extract transform load is significantly important in the process of analyzing data: Research in Computing Thesis, NCI, Ireland

The ETL process which stands for extract transform load is significantly important in the process of analyzing data in a real-time data warehouse environment. High availability and low latency are the key demand features of such an environment to ensure better functionality.

So, it can be concluded that ETL processes are becoming a hurdle for real-time data warehouse environments because of the high number of data transmission stages, the increased complexity number of machines in data processing, and the influence of individuals in the creation of new ETL processes.

Therefore, to minimize this impact and offer resonance to the ETL processes on cloud platforms a distinctive metadata framework employing big data tools is required that can supervise the development of new data processes and pipelines. This report aims to describe the ETL metadata and its significance in the execution of processes.

In this work, proprietary approaches are also provided for designing the process control that is based on metadata. This approach can minimize the complexity and improve the ETL processes’ resilience by allowing adaptive node reorganization. Implementation of the metadata Framework presented in this report is based on the use of open-source big data tools and technologies.

This also presents the architecture and connection with the outer systems templates quality metrics functions and data model. For evaluating the proposed framework, a test of the experimental Apache Airflow DAG is used which stands for the directed acyclic graph.