Improving reproducibility of geospatial conference papers: lessons learned from a first implementation of reproducibility reviews

• Published in: Septentrio Conference Series
• Main Authors: Nüst, Daniel, Ostermann, Frank, Granell, Carlos, Kmoch, Alexander
• Format: Conference Object
• Language: English
• Published: Septentrio conference series 2021
• Subjects: Reproducible research; Peer review; Open science; Reproducibility; Noruega; Tromsø
• Online Access: http://hdl.handle.net/10234/191130; https://doi.org/10.7557/5.5601

Ponència presentada a The 15th Munin Conference on Scholarly Publishing celebrat a Tromsø, Noruega, el 18 de novembre de 2020 In an attempt to increase the reproducibility of contributions to a long-running and established geospatial conference series, the 23rd AGILE Conference on Geographic Information Science 2020 (https://agile-online.org/conference-2020) for the first time provided guidelines on preparing reproducible papers (Nüst et al., 2020) and appointed a reproducibility committee to evaluate computational workflows of accepted papers ( https://www.agile-giscience-series.net/review_process.html). Here, the committee’s members report on the lessons learned from reviewing 23 accepted full papers and outline future plans for the conference series. In summary, six submissions were partially reproduced by reproducibility reviewers, whose reports are published openly on OSF ( https://osf.io/6k5fh/). These papers are promoted with badges on the proceedings’ website (https://agile-giss.copernicus.org/articles/1/index.html). Compared to previous years’ submissions (cf. Nüst et al. 2018), the guidelines and increased community awareness markedly improved reproducibility. However, the reproduction attempts also revealed problems, most importantly insufficient documentation. This was partly mitigated by the non-blind reproducibility review, conducted after paper acceptance, where interaction between reviewers and authors can provide the input and attention needed to increase reproducibility. However, the reviews also showed that anonymisation and public repositories, when properly documented, can enable a successful reproduction without interaction, as was the case with one manuscript. Individual and organisational challenges due to the COVID-19 pandemic and the conference’s eventual cancellation increased the teething problems. Nevertheless, also under normal circumstances, future iterations will have to reduce the reviewer’s efforts to be sustainable, ideally by more readily executable workflows and a larger ...