DASHBOARD.awi.de: Streaming and monitoring solutions for near real-time data

• Main Authors: Koppe, Roland, Gerchow, Peter, Macario, Ana, Haas, Antonie, Schäfer-Neth, Christian, Rehmcke, Steven, Walter, Andreas, Düde, Tobias, Weidinger, Philipp, Schäfer, Angela, Pfeiffenberger, Hans
• Format: Conference Object
• Language: unknown
• Published: 2018
• Subjects: Drake Passage; Antarc*; Antarctica
• Online Access: https://epic.awi.de/id/eprint/46854/; https://epic.awi.de/id/eprint/46854/1/O2A_dashboard_rda_2018_03_21.pdf; https://hdl.handle.net/10013/epic.1fb41d54-d912-4371-91eb-4a2284d74fab; https://hdl.handle.net/

DASHBOARD.awi.de is a component of our data flow framework designed to enable a semi-automated flow of sensor observations to archives (acronym O2A). The dramatic increase in the number and type of platforms and respective sensors operated by Alfred Wegener Institute along with complex project-driven requirements in terms of satellite communication, sensor monitoring, quality control and validation, processing pipelines, visualization, and archival under FAIR principles, led us to build a generic and cost-effective data flow framework. Most important, all components and services which make up this framework are extensible and exchangeble, were built using open access technologies (e.g. elastic search) and vocabularies (SeaVox NERC 2.0 vocabulary) and are compliant with various interoperability standards recommended by the international community. In this poster we illustrate the DASHBOARD.awi.de component which is a web-based monitoring environment for supporting scientists in the graphing, mapping and simple analysis of time series. With a set of fit-for-purpose widgets including data download, scientists are able to identify gaps and outliers in the streamed data. Morover, we are in the process of building alerting solutions for individual parameters using the parameter properties available from SENSOR.awi.de (e.g. min/max parameter range). The streaming services attached to this component are using the near real-time data transfered from remote field sites to local databases and storage systems. For this sake, we are participating in the RDA "Array Database Assessment" Working Group and "Big Data" Interest Group. The graphing solutions built within DASHBOARD.awi.de can be easily re-used in other context (e.g. project web pages) . Our solutions support the OGC standard Observations and Measurements (O&M), JSON and CSV as exchange data format. To date, ~ 150 individual parameters are being transfered in near real-time from remote sites to our onshore storage systems and monitored by scientists. In the week of the 11th RDA Plenary Berlin, we will be able to interact with live data from Polarstern crossing the Drake Passage, on its way to Antarctica. Scientific Discipline/Research Area: Data flow, standardized vocabulary, Visualisation, Streaming. Relevance/Link to RDA: The streaming services attached to this component are using the near real-time data transfered from remote field sites to local databases and storage systems. For this sake, we are participating in the RDA "Array DB Assesment" Working Group and "Big Data" Interest Group.