The IceProd Framework: Distributed Data Processing for the IceCube Neutrino Observatory

• Main Authors: Aartsen, M. G., Abbasi, R., Baker, M., Hebecker, D., Heereman, D., Heinen, D., Helbing, K., Hellauer, R., Hickford, S., Hill, G. C., Hoffman, K. D., Hoffmann, R., Homeier, A., Barwick, S. W., Hoshina, K., Huang, F., Huelsnitz, W., Hulth, P. O., Hultqvist, K., Hussain, S., Ishihara, A., Jacobi, E., Jacobsen, J., Jagielski, K., Baum, V., Japaridze, G. S., Jero, K., Jlelati, O., Kaminsky, B., Kappes, A., Karg, T., Karle, A., Kauer, M., Kelley, J. L., Kiryluk, J., Bay, R., Kläs, J., Klein, S. R., Köhne, J. -H., Kohnen, G., Kolanoski, H., Köpke, L., Kopper, C., Kopper, S., Koskinen, D. J., Kowalski, M., Beatty, J. J., Krasberg, M., Kriesten, A., Krings, K.
• Format: Report
• Language: English
• Published: 2013
• Subjects: South Pole; South pole
• Online Access: https://bib-pubdb1.desy.de/record/166426; https://bib-pubdb1.desy.de/search?p=id:%22DESY-2014-01330%22

IceCube is a one-gigaton instrument located at the geographic South Pole, designed to detect cosmic neutrinos, iden- tify the particle nature of dark matter, and study high-energy neutrinos themselves. Simulation of the IceCube detector and processing of data require a significant amount of computational resources. IceProd is a distributed management system based on Python, XML-RPC and GridFTP. It is driven by a central database in order to coordinate and admin- ister production of simulations and processing of data produced by the IceCube detector. IceProd runs as a separate layer on top of other middleware and can take advantage of a variety of computing resources, including grids and batch systems such as CREAM, Condor, and PBS. This is accomplished by a set of dedicated daemons that process job submission in a coordinated fashion through the use of middleware plugins that serve to abstract the details of job submission and job management from the framework.