The IceProd framework: Distributed data processing for the IceCube neutrino observatory

• Main Authors: Aartsen, MG, Abbasi, R, Ackermann, M, Adams, J, Aguilar, JA, Ahlers, M, Altmann, D, Arguelles, C, Auffenberg, J, Bai, X, Baker, M, Barwick, SW, Baum, V, Bay, R, Beatty, JJ, Tjus, J Becker, Becker, K-H, BenZvi, S, Berghaus, P, Berley, D, Bernardini, E, Bernhard, A, Besson, DZ, Binder, G, Bindig, D, Bissok, M, Blaufuss, E, Blumenthal, J, Boersma, DJ, Bohm, C, Bose, D, Böser, S, Botner, O, Brayeur, L, Bretz, H-P, Brown, AM, Bruijn, R, Casey, J, Casier, M, Chirkin, D, Christov, A, Christy, B, Clark, K, Classen, L, Clevermann, F, Coenders, S, Cohen, S, Cowen, DF, Silva, AH Cruz, Danninger, M, Daughhetee, J, Davis, JC, Day, M, De Clercq, C, De Ridder, S, Desiati, P, de Vries, KD, de With, M, DeYoung, T, Díaz-Vélez, JC, Dunkman, M, Eagan, R, Eberhardt, B, Eichmann, B, Eisch, J, Euler, S, Evenson, PA, Fadiran, O, Fazely, AR, Fedynitch, A, Feintzeig, J, Feusels, T, Filimonov, K, Finley, C, Fischer-Wasels, T, Flis, S, Franckowiak, A, Frantzen, K, Fuchs, T, Gaisser, TK, Gallagher, J, Gerhardt, L, Gladstone, L, Glüsenkamp, T, Goldschmidt, A, Golup, G, Gonzalez, JG, Goodman, JA, Góra, D, Grandmont, DT, Grant, D, Gretskov, P, Groh, JC, Groß, A, Ha, C, Ismail, A Haj, Hallen, P, Hallgren, A, Halzen, F, Hanson, K
• Format: Article in Journal/Newspaper
• Language: unknown
• Published: eScholarship, University of California 2015
• Subjects: Distributed Computing and Systems Software; Information and Computing Sciences; Data management; Grid computing; Monitoring; Distributed computing; cs.DC; Computer Software; South pole
• Online Access: https://escholarship.org/uc/item/15s4q7wf

IceCube is a one-gigaton instrument located at the geographic South Pole, designed to detect cosmic neutrinos, identify the particle nature of dark matter, and study high-energy neutrinos themselves. Simulation of the IceCube detector and processing of data require a significant amount of computational resources. This paper presents the first detailed description of IceProd, a lightweight distributed management system designed to meet these requirements. It is driven by a central database in order to manage mass production of simulations and analysis of data produced by the IceCube detector. IceProd runs as a separate layer on top of other middleware and can take advantage of a variety of computing resources, including grids and batch systems such as CREAM, HTCondor, and PBS. This is accomplished by a set of dedicated daemons that process job submission in a coordinated fashion through the use of middleware plugins that serve to abstract the details of job submission and job management from the framework.