parallel execution – Kai Lochbihler

A gridded data set of precipitation types for Germany

In my previous analysis of the event catalog of precipitation events in Germany, I used all available events. And, I already pointed out that the results might be influenced by the type of precipitation event. In this article, I will create a data set that allows us to distinguish between convective and stratiform precipitation events.

This post is part of the germanRADARanalysis project.

Apply rain cell tracking to 20 years of rain radar data over Germany

So, I downloaded the 20-year radar data set (Mar to Nov) from the German meteorological service (DWD). It contains about 1.5 million individual radar-derived precipitation fields – one every 5 minutes. And, I was very excited to run a rain cell tracking on it. After handing in my PhD thesis, I was going for a short trip to visit family and friends in Germany. Fortunately, I have a home server and I decided to let it do the heavy work while I am on holidays. It took the full ten days of my absence plus two extra days for the 8-core Intel CPU machine to process all data. So, let’s see how I did it.

This post is part of the germanRADARanalysis project.

Download German rain radar data and convert it to NetCDF (or what they should provide in the first place)

For a private project, I wanted to have a look at the German radar archive. I found that they have quasi-calibrated 5-minute precipitation data on a 1km² grid available in the public domain. So, let’s download and process the data set, so that it is easier to use for further data analysis.

This post is part of the germanRADARanalysis project. All code is available on G itHub.

How to run multiple instances of a program with different input data in parallel with xargs

Sometimes, we might want to run a single program with many input files. For example, if we want to resize or crop a large number of photos to the same dimensions. The long way would be to open the first file in a photo editing program and manually resize it, then open the next one, do it again and so on. But, of course we are faster if we automate the task and, most important, use multiple processor cores for parallel processing. Certainly, there are specialized software tools that will do a batch resizing of images for you (most likely not in parallel, though). However, here I will demonstrate a structured and flexible way of applying a program to multiple files in parallel on the command line in a Linux environment using xargs.