The IceProd framework: Distributed data processing for the IceCube neutrino observatory
IceCube is a one-gigaton instrument located at the geographic South Pole, designed to detect cosmic neutrinos, identify the particle nature of dark matter, and study high-energy neutrinos themselves. Simulation of the IceCube detector and processing of data require a significant amount of computatio...
Published in: | Journal of Parallel and Distributed Computing |
---|---|
Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
Format: | Article in Journal/Newspaper |
Language: | English |
Published: |
Elsevier
2015
|
Subjects: | |
Online Access: | https://bib-pubdb1.desy.de/record/293114 https://bib-pubdb1.desy.de/search?p=id:%22PUBDB-2016-00279%22 |
id |
ftdesyvdb:oai:bib-pubdb1.desy.de:293114 |
---|---|
record_format |
openpolar |
spelling |
ftdesyvdb:oai:bib-pubdb1.desy.de:293114 2023-05-15T18:22:28+02:00 The IceProd framework: Distributed data processing for the IceCube neutrino observatory Aartsen, M. G. Abbasi, R. Baker, M. Hebecker, D. Heereman, D. Heinen, D. Helbing, K. Hellauer, R. Hickford, S. Hill, G. C. Hoffman, K. D. Hoffmann, R. Homeier, A. Barwick, S. W. Hoshina, K. Huang, F. Huelsnitz, W. Hulth, P. O. Hultqvist, K. Hussain, S. Ishihara, A. Jacobi, E. Jacobsen, J. Jagielski, K. Baum, V. Japaridze, G. S. Jero, K. Jlelati, O. Kaminsky, B. Kappes, A. Karg, T. Karle, A. Kauer, M. Kelley, J. L. Kiryluk, J. Bay, R. Kläs, J. Klein, S. R. Köhne, J.-H. Kohnen, G. Kolanoski, H. Köpke, L. Kopper, C. Kopper, S. Koskinen, D. J. Kowalski, M. Beatty, J. J. Krasberg, M. Kriesten, A. Krings, K. DE 2015 https://bib-pubdb1.desy.de/record/293114 https://bib-pubdb1.desy.de/search?p=id:%22PUBDB-2016-00279%22 eng eng Elsevier info:eu-repo/semantics/altIdentifier/wos/WOS:000346552500017 info:eu-repo/semantics/altIdentifier/issn/0743-7315 info:eu-repo/semantics/altIdentifier/arxiv/arXiv:1311.5904 info:eu-repo/semantics/altIdentifier/doi/10.1016/j.jpdc.2014.08.001 info:eu-repo/semantics/altIdentifier/issn/1096-0848 https://bib-pubdb1.desy.de/record/293114 https://bib-pubdb1.desy.de/search?p=id:%22PUBDB-2016-00279%22 info:eu-repo/semantics/openAccess Journal of parallel and distributed computing 75, 198 - 211 (2015). doi:10.1016/j.jpdc.2014.08.001 info:eu-repo/classification/ddc/004 info:eu-repo/semantics/article info:eu-repo/semantics/publishedVersion 2015 ftdesyvdb https://doi.org/10.1016/j.jpdc.2014.08.001 2022-06-30T20:13:24Z IceCube is a one-gigaton instrument located at the geographic South Pole, designed to detect cosmic neutrinos, identify the particle nature of dark matter, and study high-energy neutrinos themselves. Simulation of the IceCube detector and processing of data require a significant amount of computational resources. This paper presents the first detailed description of IceProd, a lightweight distributed management system designed to meet these requirements. It is driven by a central database in order to manage mass production of simulations and analysis of data produced by the IceCube detector. IceProd runs as a separate layer on top of other middleware and can take advantage of a variety of computing resources, including grids and batch systems such as CREAM, HTCondor, and PBS. This is accomplished by a set of dedicated daemons that process job submission in a coordinated fashion through the use of middleware plugins that serve to abstract the details of job submission and job management from the framework. Article in Journal/Newspaper South pole DESY Publication Database (PUBDB) South Pole Journal of Parallel and Distributed Computing 75 198 211 |
institution |
Open Polar |
collection |
DESY Publication Database (PUBDB) |
op_collection_id |
ftdesyvdb |
language |
English |
topic |
info:eu-repo/classification/ddc/004 |
spellingShingle |
info:eu-repo/classification/ddc/004 Aartsen, M. G. Abbasi, R. Baker, M. Hebecker, D. Heereman, D. Heinen, D. Helbing, K. Hellauer, R. Hickford, S. Hill, G. C. Hoffman, K. D. Hoffmann, R. Homeier, A. Barwick, S. W. Hoshina, K. Huang, F. Huelsnitz, W. Hulth, P. O. Hultqvist, K. Hussain, S. Ishihara, A. Jacobi, E. Jacobsen, J. Jagielski, K. Baum, V. Japaridze, G. S. Jero, K. Jlelati, O. Kaminsky, B. Kappes, A. Karg, T. Karle, A. Kauer, M. Kelley, J. L. Kiryluk, J. Bay, R. Kläs, J. Klein, S. R. Köhne, J.-H. Kohnen, G. Kolanoski, H. Köpke, L. Kopper, C. Kopper, S. Koskinen, D. J. Kowalski, M. Beatty, J. J. Krasberg, M. Kriesten, A. Krings, K. The IceProd framework: Distributed data processing for the IceCube neutrino observatory |
topic_facet |
info:eu-repo/classification/ddc/004 |
description |
IceCube is a one-gigaton instrument located at the geographic South Pole, designed to detect cosmic neutrinos, identify the particle nature of dark matter, and study high-energy neutrinos themselves. Simulation of the IceCube detector and processing of data require a significant amount of computational resources. This paper presents the first detailed description of IceProd, a lightweight distributed management system designed to meet these requirements. It is driven by a central database in order to manage mass production of simulations and analysis of data produced by the IceCube detector. IceProd runs as a separate layer on top of other middleware and can take advantage of a variety of computing resources, including grids and batch systems such as CREAM, HTCondor, and PBS. This is accomplished by a set of dedicated daemons that process job submission in a coordinated fashion through the use of middleware plugins that serve to abstract the details of job submission and job management from the framework. |
format |
Article in Journal/Newspaper |
author |
Aartsen, M. G. Abbasi, R. Baker, M. Hebecker, D. Heereman, D. Heinen, D. Helbing, K. Hellauer, R. Hickford, S. Hill, G. C. Hoffman, K. D. Hoffmann, R. Homeier, A. Barwick, S. W. Hoshina, K. Huang, F. Huelsnitz, W. Hulth, P. O. Hultqvist, K. Hussain, S. Ishihara, A. Jacobi, E. Jacobsen, J. Jagielski, K. Baum, V. Japaridze, G. S. Jero, K. Jlelati, O. Kaminsky, B. Kappes, A. Karg, T. Karle, A. Kauer, M. Kelley, J. L. Kiryluk, J. Bay, R. Kläs, J. Klein, S. R. Köhne, J.-H. Kohnen, G. Kolanoski, H. Köpke, L. Kopper, C. Kopper, S. Koskinen, D. J. Kowalski, M. Beatty, J. J. Krasberg, M. Kriesten, A. Krings, K. |
author_facet |
Aartsen, M. G. Abbasi, R. Baker, M. Hebecker, D. Heereman, D. Heinen, D. Helbing, K. Hellauer, R. Hickford, S. Hill, G. C. Hoffman, K. D. Hoffmann, R. Homeier, A. Barwick, S. W. Hoshina, K. Huang, F. Huelsnitz, W. Hulth, P. O. Hultqvist, K. Hussain, S. Ishihara, A. Jacobi, E. Jacobsen, J. Jagielski, K. Baum, V. Japaridze, G. S. Jero, K. Jlelati, O. Kaminsky, B. Kappes, A. Karg, T. Karle, A. Kauer, M. Kelley, J. L. Kiryluk, J. Bay, R. Kläs, J. Klein, S. R. Köhne, J.-H. Kohnen, G. Kolanoski, H. Köpke, L. Kopper, C. Kopper, S. Koskinen, D. J. Kowalski, M. Beatty, J. J. Krasberg, M. Kriesten, A. Krings, K. |
author_sort |
Aartsen, M. G. |
title |
The IceProd framework: Distributed data processing for the IceCube neutrino observatory |
title_short |
The IceProd framework: Distributed data processing for the IceCube neutrino observatory |
title_full |
The IceProd framework: Distributed data processing for the IceCube neutrino observatory |
title_fullStr |
The IceProd framework: Distributed data processing for the IceCube neutrino observatory |
title_full_unstemmed |
The IceProd framework: Distributed data processing for the IceCube neutrino observatory |
title_sort |
iceprod framework: distributed data processing for the icecube neutrino observatory |
publisher |
Elsevier |
publishDate |
2015 |
url |
https://bib-pubdb1.desy.de/record/293114 https://bib-pubdb1.desy.de/search?p=id:%22PUBDB-2016-00279%22 |
op_coverage |
DE |
geographic |
South Pole |
geographic_facet |
South Pole |
genre |
South pole |
genre_facet |
South pole |
op_source |
Journal of parallel and distributed computing 75, 198 - 211 (2015). doi:10.1016/j.jpdc.2014.08.001 |
op_relation |
info:eu-repo/semantics/altIdentifier/wos/WOS:000346552500017 info:eu-repo/semantics/altIdentifier/issn/0743-7315 info:eu-repo/semantics/altIdentifier/arxiv/arXiv:1311.5904 info:eu-repo/semantics/altIdentifier/doi/10.1016/j.jpdc.2014.08.001 info:eu-repo/semantics/altIdentifier/issn/1096-0848 https://bib-pubdb1.desy.de/record/293114 https://bib-pubdb1.desy.de/search?p=id:%22PUBDB-2016-00279%22 |
op_rights |
info:eu-repo/semantics/openAccess |
op_doi |
https://doi.org/10.1016/j.jpdc.2014.08.001 |
container_title |
Journal of Parallel and Distributed Computing |
container_volume |
75 |
container_start_page |
198 |
op_container_end_page |
211 |
_version_ |
1766201882613645312 |