The IceProd framework: Distributed data processing for the IceCube neutrino observatory

• Published in: Journal of Parallel and Distributed Computing
• Main Authors: Aartsen, M. G., Abbasi, R., Ackermann, M., Adams, J., Aguilar, J. A., Ahlers, M., Altmann, D., Arguelles, C., Auffenberg, J., Bai, X., Baker, M., Barwick, S. W., Baum, V., Bay, R., Beatty, J. J., Tjus, J. Becker, Becker, K. -H., Benzvi, S., Berghaus, P., Berley, D., Bernardini, E., Bernhard, A., Besson, D. Z., Binder, G., Bindig, D., Bissok, M., Blaufuss, E., Blumenthal, J., Boersma, D. J., Bohm, C., Bose, D., Boeser, S., Botner, O., Brayeur, L., Bretz, H. -P., Brown, A. M., Bruijn, R., Casey, J., Casier, M., Chirkin, D., Christov, A., Christy, B., Clark, K., Classen, L., Clevermann, F., Coenders, S., Cohen, S., Cowen, D. F., Silva, A. H. Cruz, Danninger, M.
• Format: Text
• Language: unknown
• Published: San Diego, Academic Press Inc Elsevier Science 2015
• Subjects: South Pole; South pole
• Online Access: https://doi.org/10.1016/j.jpdc.2014.08.001; http://infoscience.epfl.ch/record/205415

IceCube is a one-gigaton instrument located at the geographic South Pole, designed to detect cosmic neutrinos, identify the particle nature of dark matter, and study high-energy neutrinos themselves. Simulation of the IceCube detector and processing of data require a significant amount of computational resources. This paper presents the first detailed description of IceProd, a lightweight distributed management system designed to meet these requirements. It is driven by a central database in order to manage mass production of simulations and analysis of data produced by the IceCube detector. IceProd runs as a separate layer on top of other middleware and can take advantage of a variety of computing resources, including grids and batch systems such as CREAM, HTCondor, and PBS. This is accomplished by a set of dedicated daemons that process job submission in a coordinated fashion through the use of middleware plugins that serve to abstract the details of job submission and job management from the framework. (C) 2014 Elsevier Inc. All rights reserved.