The IceProd framework: Distributed data processing for the IceCube neutrino observatory

• Main Authors: Aartsen, M. G., Hanson, Kael, Heereman von Zuydtwyck, David, Meures, Thomas, O'Murchadha, Aongus, Pinat, Elisa, Aguilar Sanchez, Juan Antonio, Toscano, Simona
• Format: Article in Journal/Newspaper
• Language: English
• Published: 2014
• Subjects: Physique des particules élémentaires; Astrophysique; Data management; Distributed computing; Grid computing; Monitoring; South Pole; South pole
• Online Access: http://hdl.handle.net/2013/ULB-DIPOT:oai:dipot.ulb.ac.be:2013/207489; https://dipot.ulb.ac.be/dspace/bitstream/2013/207489/1/Elsevier_191116.pdf

IceCube is a one-gigaton instrument located at the geographic South Pole, designed to detect cosmic neutrinos, identify the particle nature of dark matter, and study high-energy neutrinos themselves. Simulation of the IceCube detector and processing of data require a significant amount of computational resources. This paper presents the first detailed description of IceProd, a lightweight distributed management system designed to meet these requirements. It is driven by a central database in order to manage mass production of simulations and analysis of data produced by the IceCube detector. IceProd runs as a separate layer on top of other middleware and can take advantage of a variety of computing resources, including grids and batch systems such as CREAM, HTCondor, and PBS. This is accomplished by a set of dedicated daemons that process job submission in a coordinated fashion through the use of middleware plugins that serve to abstract the details of job submission and job management from the framework. 0 SCOPUS: ar.j info:eu-repo/semantics/published