Distributed data storage for modern astroparticle physics experiments
The German-Russian Astroparticle Data Life Cycle Initiative is an international project launched in 2018. The Initiative aims to develop technologies that provide a unified approach to data management, as well as to demonstrate their applicability on the example of two large astrophysical experiment...
Main Authors: | , , , , , |
---|---|
Format: | Article in Journal/Newspaper |
Language: | unknown |
Published: |
arXiv
2019
|
Subjects: | |
Online Access: | https://dx.doi.org/10.48550/arxiv.1907.06863 https://arxiv.org/abs/1907.06863 |
id |
ftdatacite:10.48550/arxiv.1907.06863 |
---|---|
record_format |
openpolar |
spelling |
ftdatacite:10.48550/arxiv.1907.06863 2023-05-15T18:30:55+02:00 Distributed data storage for modern astroparticle physics experiments Kryukov, Alexander Nguyen, Minh-Duc Bychkov, Igor Mikhailov, Andrey Shigarov, Alexey Dubenskaya, Julia 2019 https://dx.doi.org/10.48550/arxiv.1907.06863 https://arxiv.org/abs/1907.06863 unknown arXiv arXiv.org perpetual, non-exclusive license http://arxiv.org/licenses/nonexclusive-distrib/1.0/ Distributed, Parallel, and Cluster Computing cs.DC High Energy Astrophysical Phenomena astro-ph.HE Instrumentation and Methods for Astrophysics astro-ph.IM FOS Computer and information sciences FOS Physical sciences article-journal Article ScholarlyArticle Text 2019 ftdatacite https://doi.org/10.48550/arxiv.1907.06863 2022-03-10T16:47:14Z The German-Russian Astroparticle Data Life Cycle Initiative is an international project launched in 2018. The Initiative aims to develop technologies that provide a unified approach to data management, as well as to demonstrate their applicability on the example of two large astrophysical experiments - KASCADE and TAIGA. One of the key points of the project is the development of a distributed storage, which, on the one hand, will allow data of several experiments to be combined into a single repository with unified interface, and on the other hand, will provide data to all participants of experimental groups for multi-messenger analysis. Our approach to storage design is based on the single write-multiple read (SWMR) model for accessing raw or centrally processed data for further analysis. The main feature of the distributed storage is the ability to extract data either as a collection of files or as aggregated events from different sources. In the last case the storage provides users with a special service that aggregates data from different storages into a single sample. Thanks to this feature, multi-messenger methods used for more sophisticated data exploration can be applied. Users can use both Web-interface and Application Programming Interface (API) for accessing the storage. In this paper we describe the architecture of a distributed data storage for astroparticle physics and discuss the current status of our work. Article in Journal/Newspaper taiga DataCite Metadata Store (German National Library of Science and Technology) |
institution |
Open Polar |
collection |
DataCite Metadata Store (German National Library of Science and Technology) |
op_collection_id |
ftdatacite |
language |
unknown |
topic |
Distributed, Parallel, and Cluster Computing cs.DC High Energy Astrophysical Phenomena astro-ph.HE Instrumentation and Methods for Astrophysics astro-ph.IM FOS Computer and information sciences FOS Physical sciences |
spellingShingle |
Distributed, Parallel, and Cluster Computing cs.DC High Energy Astrophysical Phenomena astro-ph.HE Instrumentation and Methods for Astrophysics astro-ph.IM FOS Computer and information sciences FOS Physical sciences Kryukov, Alexander Nguyen, Minh-Duc Bychkov, Igor Mikhailov, Andrey Shigarov, Alexey Dubenskaya, Julia Distributed data storage for modern astroparticle physics experiments |
topic_facet |
Distributed, Parallel, and Cluster Computing cs.DC High Energy Astrophysical Phenomena astro-ph.HE Instrumentation and Methods for Astrophysics astro-ph.IM FOS Computer and information sciences FOS Physical sciences |
description |
The German-Russian Astroparticle Data Life Cycle Initiative is an international project launched in 2018. The Initiative aims to develop technologies that provide a unified approach to data management, as well as to demonstrate their applicability on the example of two large astrophysical experiments - KASCADE and TAIGA. One of the key points of the project is the development of a distributed storage, which, on the one hand, will allow data of several experiments to be combined into a single repository with unified interface, and on the other hand, will provide data to all participants of experimental groups for multi-messenger analysis. Our approach to storage design is based on the single write-multiple read (SWMR) model for accessing raw or centrally processed data for further analysis. The main feature of the distributed storage is the ability to extract data either as a collection of files or as aggregated events from different sources. In the last case the storage provides users with a special service that aggregates data from different storages into a single sample. Thanks to this feature, multi-messenger methods used for more sophisticated data exploration can be applied. Users can use both Web-interface and Application Programming Interface (API) for accessing the storage. In this paper we describe the architecture of a distributed data storage for astroparticle physics and discuss the current status of our work. |
format |
Article in Journal/Newspaper |
author |
Kryukov, Alexander Nguyen, Minh-Duc Bychkov, Igor Mikhailov, Andrey Shigarov, Alexey Dubenskaya, Julia |
author_facet |
Kryukov, Alexander Nguyen, Minh-Duc Bychkov, Igor Mikhailov, Andrey Shigarov, Alexey Dubenskaya, Julia |
author_sort |
Kryukov, Alexander |
title |
Distributed data storage for modern astroparticle physics experiments |
title_short |
Distributed data storage for modern astroparticle physics experiments |
title_full |
Distributed data storage for modern astroparticle physics experiments |
title_fullStr |
Distributed data storage for modern astroparticle physics experiments |
title_full_unstemmed |
Distributed data storage for modern astroparticle physics experiments |
title_sort |
distributed data storage for modern astroparticle physics experiments |
publisher |
arXiv |
publishDate |
2019 |
url |
https://dx.doi.org/10.48550/arxiv.1907.06863 https://arxiv.org/abs/1907.06863 |
genre |
taiga |
genre_facet |
taiga |
op_rights |
arXiv.org perpetual, non-exclusive license http://arxiv.org/licenses/nonexclusive-distrib/1.0/ |
op_doi |
https://doi.org/10.48550/arxiv.1907.06863 |
_version_ |
1766214542699790336 |