Adding GOES 16 / 17 / Himawari 8 / NWS to Kiwiweather.com
Leith (ZL1LAC) has described in two great posts how he managed to start receiving data from the GOES 17 satellite. With him kindly agreeing to let me take a feed straight off his Raspberry Pi and display the output here on this website.
So there was the matter of exactly how to do this. I was thinking of writing something with the Linux rsync functionality, where some code would run on a scheduled basis, look to see what new images had been received and sync them up to one of my VMs for processing.
But Leith had the great idea that goesrecv makes it’s output available via HTTP, so all we had to do was for him to put in a port forwarding rule on his router and I just had to point goesproc at his feed. This was going to be a lot easier by far!
As to how easy this was, I had to:
- Spin up a new VM using ProxMox
- Install Debian and do my usual security hardening
- Install goestools
- Update the config to point at Leigh’s Pi
The last step was as simple as:
And a few seconds later I saw, it was connected and starting to process the data:
And a few seconds later, the first image:
Viewing this on my PC, it was a mesoscale image of California and surrounding areas:
GOES 17 also rebroadcasts data from:
- GOES 16 – which is stationed to the east of GOES 17, over the Americas
- Himawari 8 – which is a Japanese satellite and stationed to get good views over Japan, but also it covers Australia and New Zealand
- NWS – the US National Weather Service with forecast graphics for the areas around the US
The next task was to get the data that arrives to be displayed on the website, so I took the find_files.py that I’d written to process GK-2A data as the starting point and then had to modify it extensively as the directory structure used for each satellite is totally different to GK-2A, with GOES 16 / 17 using one structure and Himawari a slightly different one.
The NWS directory structure really confused me as it was creating files in directories with unexpected dates, since I was doing this in November 2020:
drwxr-xr-x 2 pi pi 4096 Nov 7 22:00 1970-01-01 drwxr-xr-x 2 pi pi 4096 Nov 7 14:00 2020-03-10
The first directory name used a date which is the start of Unix Epoch Time, but the other was from multiple months ago, with over the next few days additional semi-random dated directories being created. So I raised an issue with the goesproc code and then implemented a work around to move all files into a single directory for processing.
The code then just had to scan through the directory structures, identify the latest image for each satellite / sensor / data type combination. Then rsync would copy the images over to the web server.
Next was to add some additional pages to this site using the WordPress functionality and then to enjoy the images, with my all time favourites being the full colour, full disc images from GOES 17.
There is incredible beauty in this image which is almost fully illuminated, other than a bit of Siberia, due to the earth’s axis being tilted slightly.
It is also so obvious how complex our weather is from the massive range of cloud structures, with many measuring hundreds and some thousands of kilometers across, like the ones at the top / bottom of this image.
Now thinking I’d got this cracked, I was dismayed to find that sometimes I was getting no images which turned out to be from goesproc crashing. So I wrote a watchdog script to check if it had crashed and if so, to restart it.
Then I found that overnight goesproc would still be running, but wasn’t creating any images, so I modified it again to check for this condition and if so, to also restart it. With all this code on GitHub.
So the next tasks are:
- Validate that the watchdog now works perfectly
- Add some animations of the images, especially the GOES 17 full disk images
- Use Sanchez to stitch multiple images together
All of this was done without Leith or I having met in person, yet, with Discord being an amazing way to work collaboratively and the results show how well this works for real.
