A Practical Guide for Working with Weather Datasets
Author
Patrick Wiese, ASA
Description
This series of papers is intended to serve as a practical guide for actuaries and researchers who wish to analyze weather datasets. The first paper provides an overview of the main types of weather datasets. The second paper describes computer programming strategies for processing large weather datasets using a standard personal computer. Subsequent papers will describe key weather datasets, accompanied by free, open-source computer programs (developed by SOA staff and volunteers) for analyzing the data. The computer programs will reduce the upfront time and effort required to begin working with weather datasets.
A wide range of datasets will be covered in this series of papers, including (1) data collected by weather stations; (2) data estimated using Doppler radar and/or sensors on satellites; (3) “reanalysis” datasets generated by weather models that assimilate historical data from many sources (land-based stations, ships, planes, weather balloons, buoys, satellites, and radar) and produce, as an output, spatially and temporally complete historical records; (4) short and medium term forecasts, (5) sub-seasonal and seasonal forecasts and (6) long-range climate projections.
Released Papers
Topic #1: The Main Types of Weather Datasets
This report categorizes weather datasets into 8 main types, of which 5 are historical data and 3 are forecast data. Each of the 8 dataset types is described, and an illustrative example of each is provided.
Topic #2: Strategies for Processing Large Weather Datasets
Many weather datasets exceed 100 gigabytes and some are much larger. While most climate scientists have access to servers that can store and process massive weather datasets, other types of researchers may wish to perform weather analyses on a standard personal computer. A personal computer rarely offers more than 1000 gigabytes of storage space and 16 to 32 gigabytes of RAM (active memory for running applications and programs). Given these constraints, a researcher will need a clever approach for working with large weather datasets. This paper describes techniques for running analyses of a large weather dataset despite the storage and memory limitations imposed by a personal computer. This paper is accompanied by 6 small computer programs written in VBA, Python, R, and C++, zipped together into a single file to simplify the download process. These programs illustrate techniques for processing large weather datasets using the limited memory (RAM) available on a personal computer.
Datasets to be Covered in Future Reports
-
GHCN Daily Station Data
-
ERA5 Hourly and Monthly Reanalysis Data
-
NOAA’s Storm Events Database
-
IBTrACS Hurricane Tracks Database
-
CMIP6 Climate Projections
Questions or Comments?
Give us your feedback! Take a short survey on this report. Take Survey
If you have comments or questions, please send an email to research@soa.org.