The IceProd framework: Distributed data processing for the IceCube neutrino observatory

• Published in: Journal of Parallel and Distributed Computing
• Main Authors: Aartsen, M. G., Abbasi, R., Baker, M., Hebecker, D., Heereman, D., Heinen, D., Helbing, K., Hellauer, R., Hickford, S., Hill, G. C., Hoffman, K. D., Hoffmann, R., Homeier, A., Barwick, S. W., Hoshina, K., Huang, F., Huelsnitz, W., Hulth, P. O., Hultqvist, K., Hussain, S., Ishihara, A., Jacobi, E., Jacobsen, J., Jagielski, K., Baum, V., Japaridze, G. S., Jero, K., Jlelati, O., Kaminsky, B., Kappes, A., Karg, T., Karle, A., Kauer, M., Kelley, J. L., Kiryluk, J., Bay, R., Kläs, J., Klein, S. R., Köhne, J.-H., Kohnen, G., Kolanoski, H., Köpke, L., Kopper, C., Kopper, S., Koskinen, D. J., Kowalski, M., Beatty, J. J., Krasberg, M., Kriesten, A., Krings, K.
• Format: Article in Journal/Newspaper
• Language: English
• Published: Elsevier 2015
• Subjects: info:eu-repo/classification/ddc/004; South Pole; South pole
• Online Access: https://bib-pubdb1.desy.de/record/293114; https://bib-pubdb1.desy.de/search?p=id:%22PUBDB-2016-00279%22

IceCube is a one-gigaton instrument located at the geographic South Pole, designed to detect cosmic neutrinos, identify the particle nature of dark matter, and study high-energy neutrinos themselves. Simulation of the IceCube detector and processing of data require a significant amount of computational resources. This paper presents the first detailed description of IceProd, a lightweight distributed management system designed to meet these requirements. It is driven by a central database in order to manage mass production of simulations and analysis of data produced by the IceCube detector. IceProd runs as a separate layer on top of other middleware and can take advantage of a variety of computing resources, including grids and batch systems such as CREAM, HTCondor, and PBS. This is accomplished by a set of dedicated daemons that process job submission in a coordinated fashion through the use of middleware plugins that serve to abstract the details of job submission and job management from the framework.