A Framework for Management of Distributed Data Processing and Event Selection for the Icecube Neutrino Observatory
IceCube is a one-gigaton neutrino detector designed to detect high-energy cosmic neutrinos. It is located at the geographic South Pole and was completed at the end of 2010. Simulation and data processing for IceCube requires a significant amount of computational power. We describe the design and fun...
Main Author: | |
---|---|
Format: | Text |
Language: | unknown |
Published: |
ScholarWorks
2013
|
Subjects: | |
Online Access: | https://scholarworks.boisestate.edu/td/616 https://scholarworks.boisestate.edu/context/td/article/1623/viewcontent/Diaz_Velez_Juan_Carlos_thesis_Aug_2013.pdf |
id |
ftboisestateu:oai:scholarworks.boisestate.edu:td-1623 |
---|---|
record_format |
openpolar |
spelling |
ftboisestateu:oai:scholarworks.boisestate.edu:td-1623 2023-10-29T02:40:13+01:00 A Framework for Management of Distributed Data Processing and Event Selection for the Icecube Neutrino Observatory Díaz Vélez, Juan Carlos 2013-08-01T07:00:00Z application/pdf https://scholarworks.boisestate.edu/td/616 https://scholarworks.boisestate.edu/context/td/article/1623/viewcontent/Diaz_Velez_Juan_Carlos_thesis_Aug_2013.pdf unknown ScholarWorks https://scholarworks.boisestate.edu/td/616 https://scholarworks.boisestate.edu/context/td/article/1623/viewcontent/Diaz_Velez_Juan_Carlos_thesis_Aug_2013.pdf Boise State University Theses and Dissertations Computer Science Distributed Computing Grid Computing Directed Acyclic Graphs text 2013 ftboisestateu 2023-09-29T15:10:52Z IceCube is a one-gigaton neutrino detector designed to detect high-energy cosmic neutrinos. It is located at the geographic South Pole and was completed at the end of 2010. Simulation and data processing for IceCube requires a significant amount of computational power. We describe the design and functionality of IceProd, a management system based on Python, XMLRPC, and GridFTP. It is driven by a central database in order to coordinate and administer production of simulations and processing of data produced by the IceCube detector upon arrival in the northern hemisphere. IceProd runs as a separate layer on top of existing middleware and can take advantage of a variety of computing resources including grids and batch systems such as GLite, Condor, NorduGrid, PBS, and SGE. This is accomplished by a set of dedicated daemons that process job submission in a coordinated fashion through the use of middleware plug-ins that serve to abstract the details of job submission and job management. IceProd fills a gap between the user and existing middleware by making job scripting easier and collaboratively sharing productions more efficiently. We describe the implementation and performance of an extension to the IceProd framework that provides support for mapping workflow diagrams or DAGs consisting of interdependent tasks to an IceProd job that can span across multiple grid or cluster sites. We look at some use-cases where this new extension allows for optimal allocation of computing resources and addresses general aspects of this design, including security, data integrity, scalability, and throughput. Text South pole Boise State University: Scholar Works |
institution |
Open Polar |
collection |
Boise State University: Scholar Works |
op_collection_id |
ftboisestateu |
language |
unknown |
topic |
Computer Science Distributed Computing Grid Computing Directed Acyclic Graphs |
spellingShingle |
Computer Science Distributed Computing Grid Computing Directed Acyclic Graphs Díaz Vélez, Juan Carlos A Framework for Management of Distributed Data Processing and Event Selection for the Icecube Neutrino Observatory |
topic_facet |
Computer Science Distributed Computing Grid Computing Directed Acyclic Graphs |
description |
IceCube is a one-gigaton neutrino detector designed to detect high-energy cosmic neutrinos. It is located at the geographic South Pole and was completed at the end of 2010. Simulation and data processing for IceCube requires a significant amount of computational power. We describe the design and functionality of IceProd, a management system based on Python, XMLRPC, and GridFTP. It is driven by a central database in order to coordinate and administer production of simulations and processing of data produced by the IceCube detector upon arrival in the northern hemisphere. IceProd runs as a separate layer on top of existing middleware and can take advantage of a variety of computing resources including grids and batch systems such as GLite, Condor, NorduGrid, PBS, and SGE. This is accomplished by a set of dedicated daemons that process job submission in a coordinated fashion through the use of middleware plug-ins that serve to abstract the details of job submission and job management. IceProd fills a gap between the user and existing middleware by making job scripting easier and collaboratively sharing productions more efficiently. We describe the implementation and performance of an extension to the IceProd framework that provides support for mapping workflow diagrams or DAGs consisting of interdependent tasks to an IceProd job that can span across multiple grid or cluster sites. We look at some use-cases where this new extension allows for optimal allocation of computing resources and addresses general aspects of this design, including security, data integrity, scalability, and throughput. |
format |
Text |
author |
Díaz Vélez, Juan Carlos |
author_facet |
Díaz Vélez, Juan Carlos |
author_sort |
Díaz Vélez, Juan Carlos |
title |
A Framework for Management of Distributed Data Processing and Event Selection for the Icecube Neutrino Observatory |
title_short |
A Framework for Management of Distributed Data Processing and Event Selection for the Icecube Neutrino Observatory |
title_full |
A Framework for Management of Distributed Data Processing and Event Selection for the Icecube Neutrino Observatory |
title_fullStr |
A Framework for Management of Distributed Data Processing and Event Selection for the Icecube Neutrino Observatory |
title_full_unstemmed |
A Framework for Management of Distributed Data Processing and Event Selection for the Icecube Neutrino Observatory |
title_sort |
framework for management of distributed data processing and event selection for the icecube neutrino observatory |
publisher |
ScholarWorks |
publishDate |
2013 |
url |
https://scholarworks.boisestate.edu/td/616 https://scholarworks.boisestate.edu/context/td/article/1623/viewcontent/Diaz_Velez_Juan_Carlos_thesis_Aug_2013.pdf |
genre |
South pole |
genre_facet |
South pole |
op_source |
Boise State University Theses and Dissertations |
op_relation |
https://scholarworks.boisestate.edu/td/616 https://scholarworks.boisestate.edu/context/td/article/1623/viewcontent/Diaz_Velez_Juan_Carlos_thesis_Aug_2013.pdf |
_version_ |
1781068182445883392 |