<abstract xmlns="http://www.w3.org/1999/xhtml"><p>Today’s ETL tools provide capabilities to develop custom code as user-defined functions (UDFs) to extend the expressiveness of the standard ETL operators. However, while this allows us to easily add new functionalities, it also comes with the risk that the custom code is not intended to be optimized, e.g., by parallelism, and for this reason, it performs poorly for data-intensive ETL workflows. In this paper we present a novel framework, which allows the ETL developer to choose a design pattern in order to write parallelizable code and generates a configuration for the UDFs to be executed in a distributed environment. This enables ETL developers with minimum expertise in distributed and parallel computing to develop UDFs without taking care of parallelization configurations and complexities. We perform experiments on large-scale datasets based on TPC-DS and BigBench. The results show that our approach significantly reduces the effort of ETL developers and at the same time generates efficient parallel configurations to support complex and data-intensive ETL tasks.</p></abstract>

Today’s ETL tools provide capabilities to develop custom code as user-defined functions (UDFs) to extend the expressiveness of the standard ETL operators. However, while this allows us to easily add new functionalities, it also comes with the risk that the custom code is not intended to be optimized, e.g., by parallelism, and for this reason, it performs poorly for data-intensive ETL workflows. In this paper we present a novel framework, which allows the ETL developer to choose a design pattern in order to write parallelizable code and generates a configuration for the UDFs to be executed in a distributed environment. This enables ETL developers with minimum expertise in distributed and parallel computing to develop UDFs without taking care of parallelization configurations and complexities. We perform experiments on large-scale datasets based on TPC-DS and BigBench. The results show that our approach significantly reduces the effort of ETL developers and at the same time generates efficient parallel configurations to support complex and data-intensive ETL tasks.

Parallelizing user–defined functions in the ETL workflow using orchestration style sheets

International Journal of Applied Mathematics and Computer Science

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

Today’s ETL tools provide capabilities to develop custom code as user-defined functions (UDFs) to extend the expressiveness of the standard ETL operators....

Parallelizing user–defined functions in the ETL workflow using orchestration style sheets

Published Online: Mar 29, 2019

Page range: 69 - 79

Received: Mar 04, 2018

Accepted: Dec 10, 2018

DOI: https://doi.org/10.2478/amcs-2019-0005

Keywords
ETL workflow, parallel ETL operators, parallel algorithmic skeletons, user-defined functions

© 2019 Syed Muhammad Fawad Ali et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

Parallelizing user–defined functions in the ETL workflow using orchestration style sheets

Published Online: Mar 29, 2019

Page range: 69 - 79

Received: Mar 04, 2018

Accepted: Dec 10, 2018

DOI: https://doi.org/10.2478/amcs-2019-0005

KeywordsETL workflow, parallel ETL operators, parallel algorithmic skeletons, user-defined functions

© 2019 Syed Muhammad Fawad Ali et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.

Keywords
ETL workflow, parallel ETL operators, parallel algorithmic skeletons, user-defined functions