How to process data in NetCDF and GRIB2 formats

This article provides basic information about the 2 most commonly used weather data formats, and it will describe the steps needed to process the NetCDF and GRIB2 data. It also addresses a few issues that might occur during the processing. There are also a couple of references to publically available weather data in NetCDF and GRIB2 formats.

Format specifications

NetCDF

NetCDF (Network Common Data Form) is designed to facilitate access to array-oriented scientific data. NetCDF is a portable, "self-describing" format. This means that there is a header that describes the layout of the rest of the file, in particular the data arrays, as well as arbitrary file metadata in the form of name/value attributes. The additional information about a file or variable is commonly called "metadata" (information about the data).

Data examples: Pacific climate impacts consortium.

GRIB2

GRIB is a file format for the storage and transport of gridded meteorological data, such as Numerical Weather Prediction model output. It is designed to be self-describing, compact, and portable across computer architectures. The GRIB standard was designed and is maintained by the World Meteorological Organization.

Data examples: US Weather Research and Forecasting (WRF) model

Pre-processing

The GRIB2 doesn’t need any pre-processing and can be processed directly in MapTiler Engine. NetCDF often has a problem with the definition of CRS or is not specified at all. This can be checked by GDAL Documentation.

gdalsrsinfo NETCDF:PNWNAmet_wind.nc.nc

If the file contains CRS, you can continue. For this specific dataset response message will be:

ERROR 1: ERROR - failed to load SRS definition from NETCDF:PNWNAmet_wind.nc.nc

This still doesn’t mean the CRS info is not included in the dataset. Usually, the projection is mentioned in the metadata of the dataset. Using GDAL documentation we can get all the metadata stored in the file.

gdalinfo NETCDF:PNWNAmet_wind.nc.nc

Metadata from the input file (shorted):

Driver: netCDF/Network Common Data Format
Files: none associated
Size is 1088, 512
Origin = (-169.000000000000000,72.000000000000000)
Pixel Size = (0.062500000000000,-0.062500000000000)
Band 1 Block=1088x1 Type=Float32, ColorInterp=Undefined
  NoData Value=-9.96920996838686905e+36
  Unit Type: m/s
  Metadata:
    long_name=Surface (10m) wind speed
    missing_value=-9.96921e+36
    NETCDF_DIM_time=24836
    NETCDF_VARNAME=wind
    projection=+proj=longlat +datum=WGS84 +ellps=WGS84 +towgs84=0,0,0
    projection_format=PROJ.4
    standard_name=wind
    units=m/s
    _FillValue=-9.96921e+36

In the metadata text on the 14th line, we can see the definition of the projection, and on the 15th line projection format. The CRS can be defined using the GDAL-translate documentation.

gdal_translate -a_srs "+proj=longlat +datum=WGS84 +ellps=WGS84 +towgs84=0,0,0" NETCDF:PNWNAmet_wind.nc.nc PNWNAmet_wind_CRS.nc

Sometimes netCDF contains “subbands”, which MapTiler Engine is unable to process. The solution is to create a “virtual” file with GDAL-buildvrt documentation and process the virtual file.

Processing

The NetCDF and GRIB2 were developed mainly for raw scientific data. Formats always contain raw meteorological data.

MapTiler Engine allows choosing bands for RGB(A) color models from multiple map channels. In these formats can be stored hundreds of bands with different variables. Only three classical bands are used for rendering via MapTiler Engine, see MapTiler Engine Manual documentation.

For the MapTiler Weather library is important to scale the band of each variable. Ideally, min and max values should copy the color ramp (legend) values in the map.

NetCDF

maptiler
 -f png32 -store mbtiles -o PNWNAmet_wind_CRS.mbtiles \
  PNWNAmet_wind_CRS.nc \
   # u wind component
   -b 1 \
    -data_scale -30 30 \
   # v wind component
   -b 2 \
    -data_scale -30 30 \
   # wind speed
   -b 3 \
    - data_scale 0 200

GRIB2

maptiler
 -f png32 -store mbtiles -o forecast.mbtiles \
  XXX.grib2 \
   -b 1 \
    -data_scale -30 40

Output tileset can be uploaded on MapTiler Cloud using MapTiler Cloud Admin API or be directly used in MapTiler Server.

Sources

NetCDF Overview - Climate Data Guide
Canadian Weather data

Conclusion

NetCDF and GRIB2 data formats are used by many national and global environment- and weather-focused orgnaizations for storing and facilitating facilitate access to array-oriented scientific data. MapTiler Engine and MapTiler Cloud, together with the MapTiler Weather library, offer an easy way how to process and visualize weather data and incorporate it into your maps.

Useful links

How to make weather maps