Reusing distributed computing software and patterns for midscale collaborative science

• Main Authors: Paschos, Paschalis, Rynge, Mats, Riedel, Benedikt, Wuerthwein, Frank, Gardner, Robert William
• Format: Conference Object
• Language: unknown
• Published: Zenodo 2019
• Subjects: South Pole; South pole
• Online Access: https://dx.doi.org/10.5281/zenodo.3599635; https://zenodo.org/record/3599635

Many of the challenges faced by the LHC experiments (aggregation of distributed computing resources, management of data across multiple storage facilities, integration of experiment-specific workflow management tools across multiple grid services) are similarly experienced by "midscale" high energy physics and astrophysics experiments, particularly as their data set volumes are increasing at comparable rates. Often these (international, multi-institution) collaborations have outgrown the computing resources offered by their home laboratories, or the capacities of any single member institution. Unlike the LHC experiments, however, these collaborations often lack the manpower required to build, integrate and operate the systems required to meet their scale. In the Open Science Grid, we have organized a team designed to support collaborative science organizations re-use proven software and patterns in distributed processing and data management, often but not restricted to software developed for the LHC. Examples are re-use of the Rucio and FTS3 software for reliable data transfer and management, XRootD for data access and caching, Ceph for large scale pre-processing storage, and Pegasus for workflow management across heterogeneous resources. We summarize experience with the VERITAS gamma ray observatory, the South Pole Telescope (CMB detector), and the XENON dark matter search experiment.