Towards Scalability and Data Skew Handling in GroupBy-Joins using MapReduce Model

• Main Authors: Al Hajj Hassan, Mohamad, Bamha, Mostafa
• Other Authors: Lebanese International University (LIU), Laboratoire d'Informatique Fondamentale d'Orléans (LIFO), Université d'Orléans (UO)-Ecole Nationale Supérieure d'Ingénieurs de Bourges
• Format: Conference Object
• Language: English
• Published: HAL CCSD 2015
• Subjects: Join and GrouBy-join operations; Data skew; MapReduce programming model; Distributed file systems; Hadoop framework; Apache Pig Latin; [SCCO.COMP]Cognitive science/Computer science; Iceland
• Online Access: https://hal.science/hal-01160931

International audience For over a decade, MapReduce has become the leading programming model for parallel and massive processing of large volumes of data. This has been driven by the development of many frameworks such as Spark, Pig and Hive, facilitating data analysis on large-scale systems. However, these frameworks still remain vulnerable to communication costs, data skew and tasks imbalance problems. This can have a devastating effect on the performance and on the scalability of these systems, more particularly when treating GroupBy-Join queries of large datasets.In this paper, we present a new GroupBy-Join algorithm allowing to reduce communication costs considerably while avoiding data skew effects.A cost analysis of this algorithm shows that our approach is insensitive to data skew and ensures perfect balancing properties during all stages of GroupBy-Join computation even for highly skewed data. These performances have been confirmed by a series of experimentations.