Diarchy: An Optimized Management Approach for MapReduce Masters
International audience The MapReduce community is progressively replacing the classic Hadoop with Yarn, the second-generation Hadoop (MapReduce 2.0). This transition is being made due to many reasons, but primarily because of some scalability drawbacks of the classic Hadoop. The new framework has ap...
Published in: | Procedia Computer Science |
---|---|
Main Authors: | , , |
Other Authors: | , , , , , , , , , , |
Format: | Conference Object |
Language: | English |
Published: |
HAL CCSD
2015
|
Subjects: | |
Online Access: | https://hal.archives-ouvertes.fr/hal-01249151 https://doi.org/10.1016/j.procs.2015.05.179 |
id |
ftccsdartic:oai:HAL:hal-01249151v1 |
---|---|
record_format |
openpolar |
spelling |
ftccsdartic:oai:HAL:hal-01249151v1 2023-05-15T16:50:20+02:00 Diarchy: An Optimized Management Approach for MapReduce Masters Memishi, Bunjamin Pérez-Hernández, María S. Antoniu, Gabriel Universidad Politécnica de Madrid (UPM) Scalable Storage for Clouds and Beyond (KerData) Inria Rennes – Bretagne Atlantique Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria)-SYSTÈMES LARGE ÉCHELLE (IRISA-D1) Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA) CentraleSupélec-Télécom Bretagne-Université de Rennes 1 (UR1) Université de Rennes (UNIV-RENNES)-Université de Rennes (UNIV-RENNES)-Institut National de Recherche en Informatique et en Automatique (Inria)-École normale supérieure - Rennes (ENS Rennes)-Université de Bretagne Sud (UBS)-Centre National de la Recherche Scientifique (CNRS)-Institut National des Sciences Appliquées - Rennes (INSA Rennes) Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées (INSA)-CentraleSupélec-Télécom Bretagne-Université de Rennes 1 (UR1) Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées (INSA)-Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA) Université de Rennes (UNIV-RENNES)-Université de Rennes (UNIV-RENNES)-École normale supérieure - Rennes (ENS Rennes)-Université de Bretagne Sud (UBS)-Centre National de la Recherche Scientifique (CNRS)-Institut National des Sciences Appliquées - Rennes (INSA Rennes) Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées (INSA) Reykjavík, Iceland 2015-06 https://hal.archives-ouvertes.fr/hal-01249151 https://doi.org/10.1016/j.procs.2015.05.179 en eng HAL CCSD info:eu-repo/semantics/altIdentifier/doi/10.1016/j.procs.2015.05.179 hal-01249151 https://hal.archives-ouvertes.fr/hal-01249151 doi:10.1016/j.procs.2015.05.179 ICCS 2015: Proceedings of the International Conference on Computational Science, Computational Science at the Gates of Nature https://hal.archives-ouvertes.fr/hal-01249151 ICCS 2015: Proceedings of the International Conference on Computational Science, Computational Science at the Gates of Nature, Jun 2015, Reykjavík, Iceland. pp.9--18, ⟨10.1016/j.procs.2015.05.179⟩ [INFO.INFO-DC]Computer Science [cs]/Distributed Parallel and Cluster Computing [cs.DC] info:eu-repo/semantics/conferenceObject Conference papers 2015 ftccsdartic https://doi.org/10.1016/j.procs.2015.05.179 2021-10-24T10:58:52Z International audience The MapReduce community is progressively replacing the classic Hadoop with Yarn, the second-generation Hadoop (MapReduce 2.0). This transition is being made due to many reasons, but primarily because of some scalability drawbacks of the classic Hadoop. The new framework has appropriately addressed this issue and is being praised for its multi-functionality. In this paper we carry out a probabilistic analysis that emphasizes some reliability concerns of Yarn at the job master level. This is a critical point, since the failures of a job master involves the failure of all the workers managed by such a master. In this paper, we propose Diarchy, a novel system for the management of job masters. Its aim is to increase the reliability of Yarn, based on the sharing and backup of responsibilities between two masters working as peers. The evaluation results show that Diarchy outperforms the reliability performance of Yarn in different setups, regardless of cluster size, type of job, or average failure rate and suggest a positive impact of this approach compared to the traditional, single-master Hadoop architecture. Conference Object Iceland Reykjavík Reykjavík Archive ouverte HAL (Hyper Article en Ligne, CCSD - Centre pour la Communication Scientifique Directe) Reykjavík Procedia Computer Science 51 9 18 |
institution |
Open Polar |
collection |
Archive ouverte HAL (Hyper Article en Ligne, CCSD - Centre pour la Communication Scientifique Directe) |
op_collection_id |
ftccsdartic |
language |
English |
topic |
[INFO.INFO-DC]Computer Science [cs]/Distributed Parallel and Cluster Computing [cs.DC] |
spellingShingle |
[INFO.INFO-DC]Computer Science [cs]/Distributed Parallel and Cluster Computing [cs.DC] Memishi, Bunjamin Pérez-Hernández, María S. Antoniu, Gabriel Diarchy: An Optimized Management Approach for MapReduce Masters |
topic_facet |
[INFO.INFO-DC]Computer Science [cs]/Distributed Parallel and Cluster Computing [cs.DC] |
description |
International audience The MapReduce community is progressively replacing the classic Hadoop with Yarn, the second-generation Hadoop (MapReduce 2.0). This transition is being made due to many reasons, but primarily because of some scalability drawbacks of the classic Hadoop. The new framework has appropriately addressed this issue and is being praised for its multi-functionality. In this paper we carry out a probabilistic analysis that emphasizes some reliability concerns of Yarn at the job master level. This is a critical point, since the failures of a job master involves the failure of all the workers managed by such a master. In this paper, we propose Diarchy, a novel system for the management of job masters. Its aim is to increase the reliability of Yarn, based on the sharing and backup of responsibilities between two masters working as peers. The evaluation results show that Diarchy outperforms the reliability performance of Yarn in different setups, regardless of cluster size, type of job, or average failure rate and suggest a positive impact of this approach compared to the traditional, single-master Hadoop architecture. |
author2 |
Universidad Politécnica de Madrid (UPM) Scalable Storage for Clouds and Beyond (KerData) Inria Rennes – Bretagne Atlantique Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria)-SYSTÈMES LARGE ÉCHELLE (IRISA-D1) Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA) CentraleSupélec-Télécom Bretagne-Université de Rennes 1 (UR1) Université de Rennes (UNIV-RENNES)-Université de Rennes (UNIV-RENNES)-Institut National de Recherche en Informatique et en Automatique (Inria)-École normale supérieure - Rennes (ENS Rennes)-Université de Bretagne Sud (UBS)-Centre National de la Recherche Scientifique (CNRS)-Institut National des Sciences Appliquées - Rennes (INSA Rennes) Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées (INSA)-CentraleSupélec-Télécom Bretagne-Université de Rennes 1 (UR1) Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées (INSA)-Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA) Université de Rennes (UNIV-RENNES)-Université de Rennes (UNIV-RENNES)-École normale supérieure - Rennes (ENS Rennes)-Université de Bretagne Sud (UBS)-Centre National de la Recherche Scientifique (CNRS)-Institut National des Sciences Appliquées - Rennes (INSA Rennes) Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées (INSA) |
format |
Conference Object |
author |
Memishi, Bunjamin Pérez-Hernández, María S. Antoniu, Gabriel |
author_facet |
Memishi, Bunjamin Pérez-Hernández, María S. Antoniu, Gabriel |
author_sort |
Memishi, Bunjamin |
title |
Diarchy: An Optimized Management Approach for MapReduce Masters |
title_short |
Diarchy: An Optimized Management Approach for MapReduce Masters |
title_full |
Diarchy: An Optimized Management Approach for MapReduce Masters |
title_fullStr |
Diarchy: An Optimized Management Approach for MapReduce Masters |
title_full_unstemmed |
Diarchy: An Optimized Management Approach for MapReduce Masters |
title_sort |
diarchy: an optimized management approach for mapreduce masters |
publisher |
HAL CCSD |
publishDate |
2015 |
url |
https://hal.archives-ouvertes.fr/hal-01249151 https://doi.org/10.1016/j.procs.2015.05.179 |
op_coverage |
Reykjavík, Iceland |
geographic |
Reykjavík |
geographic_facet |
Reykjavík |
genre |
Iceland Reykjavík Reykjavík |
genre_facet |
Iceland Reykjavík Reykjavík |
op_source |
ICCS 2015: Proceedings of the International Conference on Computational Science, Computational Science at the Gates of Nature https://hal.archives-ouvertes.fr/hal-01249151 ICCS 2015: Proceedings of the International Conference on Computational Science, Computational Science at the Gates of Nature, Jun 2015, Reykjavík, Iceland. pp.9--18, ⟨10.1016/j.procs.2015.05.179⟩ |
op_relation |
info:eu-repo/semantics/altIdentifier/doi/10.1016/j.procs.2015.05.179 hal-01249151 https://hal.archives-ouvertes.fr/hal-01249151 doi:10.1016/j.procs.2015.05.179 |
op_doi |
https://doi.org/10.1016/j.procs.2015.05.179 |
container_title |
Procedia Computer Science |
container_volume |
51 |
container_start_page |
9 |
op_container_end_page |
18 |
_version_ |
1766040504571527168 |