An Introduction to Open Radar Science

What is Open Science?¶

Image based on Ramachandran et al. 2021

What Does this look like in the Weather Radar Community?¶

• Since the late 2000s (and even before) there has been a number of major open source projects released (see e.g. https://openradarscience.org).

• Some of them are in a mature stage and are widely used in an academic (mostly) but also operational environment

• Most make use of modern tools (e.g. github, conda, docker) and practices (e.g. Continuous Integration, automatic tests) that make them easy to evolve and deploy

• Most are backed by major weather services or academic institutions Projects are not competing among them but collaborating : Best practices and inter-operability are discussed regularly and joint open source courses have been organized for years at major radar conferences (AMS, ERAD)

The Conceptual Idea of the Open Radar Stack¶

Common weather radar processing workflows¶

The Open Radar Tools¶

Additional Tools that Can be used with/for Weather Radar Data¶

• MetPy

  • a collection of tools in Python for reading, visualizing, and performing calculations with weather data

• tobac

  • a Python package for rapidly identify, track and analyze clouds in different types of gridded datasets, such as 3D model output from cloud-resolving model simulations or 2D data from satellite retrievals.

Introductions of the Core Open Radar Stack¶

Wradlib (Keep the magic to the minimum (let the user decide))¶

• https://wradlib.org

• One of the oldest packages (2011)

• Open platform for collaborative development of algorithms

• Python-based

• Linux/Windows/Mac

• Flat data model that allows maximum flexibility to interact with the data.

• xarray readers available (now in xradar as of 2.0)

• Comprehensively addresses the full radar processing chain

• Mainly geared to interactive use in research but used in operations too

• Easy to install (PyPI, conda, Docker Hub)

Functionality¶

Sample Image¶

Py-ART (It’s all about the data model)¶

• https://arm-doe.github.io/pyart/

• Created in the context of the ARM program (2013)

• Open platform for collaborative development of algorithms

• Mostly Python-based (some modules in C, Cython and FORTRAN)

• Linux/Windows/Mac

• Core : Radar object that structures the radar data and metadata mirroring the C/F Radial standard

• Limited scope. Base block to built upon

• Rich ecosystem of packages: ART-VIEW, PyTDA, PyDDA, TINT, Pyrad...

• Easy to install (PyPI, conda)

Functionality¶

Sample Image¶

Pyrad (Flexible and replicable data processing chains with no programming)¶

• https://github.com/MeteoSwiss/pyrad

• Initially developed at MeteoSwiss. Now shared development between MeteoSwiss and Météo-France

• Python-based weather radar data processing framework capable of operating in real time or off-line

• Core based on ARM-DOE Py-ART (Pyrad major contributor)

• Easy to install (PyPI, conda)

Functionality¶

Sample Image¶

LROSE (High quality building blocks forcomplex workflows)¶

• http://lrose.net/

• Based on legacy of NCAR and CSU tools

• Fast native cross-platform applications

• Mostly C++

• Linux/Mac/partially Windows

• Many stand-alone tools

• Stores data in CF/Radial

Functionality¶

Sample Image¶

BALTRAD (Advanced Weather Radar Network)¶

• Heritage from the Nordic Network NORDRAD. Partly funded by the EU. BALTRAD and BALTRAD+ projects (2009-2014). 13 partners in 10 countries

• Real-time data exchange and data processing

• Sub-packages written in different languages

  • Data exchange: JAVA

  • Data processing: C and Python

• Linux/Mac

• Distributed networking, partners exchange polar data and process them using a common toolbox

• Uses ODIM-H5

• Documentation: https://baltrad.github.io/

• Code: https://github.com/baltrad

Functionality¶

Sample Image¶

Conclusions on the Open Radar Stack¶

• Acknowledge software when you use it – treat it like a paper

  • Most projects have Digital Object Identifiers (DOIs)

• Contribute back to projects

  • Report bugs, suggest enhancements, share feedback!

• Do not be afraid to contribute

  • We all started somewhere – your code will be reviewed + tests ensure things do not break

• Open Source does not end at making things open

  • Support, documentation, consistency, etc. are required

• Take a look at existing projects to see if you can collaborate/coordinate!

The Open Radar Community¶

More on Open Science: Beyond the Tools¶

• Open Science = Open Data + Open Algorithms + Open Software + Open Peer Review + Open Access Publication

• Thereby, science process and outputs can be:

  • ✅ Transparent

  • ✅ Reproducible

  • ✅ Transferrable

  • ✅ Collaborative

  • ✅ Credible!

  • ✅ Durable, sustainable

Recommendations for Open Scientists¶

• Use Git for collaborative change/version code management

  • GitHub: github.com

  • GitLab: gitlab.com

• Linux virtual machine on your local computer

  • Cost-effective near-replication of official environments

  • Good for development, even while traveling

  • VM is the starting point, a vehicle, for creating transferrable computational environments

• Today’s cloud instance is an “elaborated” VM (Jupyterhub/Binderhub)

• If you have a choice, use an open programming language

• Avoid proprietary algorithms like Numerical Recipes

• Publish code

• Publish data

• Publish code & data associated with a study/paper

• Publish in Open Access journals

The Open Radar Forum - Establishing a Community of Practice¶

https://openradar.discourse.group/

Conclusions¶

https://openradarscience.org

• There is no “silver bullet” for open science

• Try to use open languages where possible

• Share your work!

• Join the conversation on the forum

• You can find more posts relevant to the Open Radar Community on our website (at the top) and on the forum

• We have monthly meetings! Join!