Please note: the ideas in the this package were taken forward in tidync and ncmeta which are probably the packages you should use instead.
The ncdump package aims to simplify the way we can approach NetCDF in R.
Currently the only real functionality is to return the complete file metadata in tidy form. There is some experimental function to return specific entities, but in my experience the existing tools ncdf4, raster, rgdal, RNetCDF and rhdf5 are sufficient for easily accessing almost everything. Where they fall down is in providing easy and complete information about what is available in the files, so thatâs where ncdump::NetCDF comes in.
Create an object that has a complete description of the file so that we can easily see the available variables and dimensions and attributes, and perform queries that find the details we need in a form we can used, rather than just printed out on the screen.
library(dplyr)
library(ncdump)
ifile <- system.file("extdata", "S2008001.L3m_DAY_CHL_chlor_a_9km.nc", package = "ncdump")
con <- NetCDF(ifile)Note that this is just the metadata in the file, but weâve organized it into a tidy list of tables so we can use it.
It looks good when we print those tables out, and itâs pretty clear how we can use this information to drive the reader functions in the API packages.
for (i in seq_along(con)) {
print(names(con)[i])
print(con[[i]])
}
#> [1] "dimension"
#> # A tibble: 4 Ã 9
#> name len unlim group_index group_id id create_dimvar .dimension_ .group_
#> <chr> <int> <lgl> <int> <int> <int> <lgl> <int> <int>
#> 1 lat 2160 FALSE 1 65536 0 TRUE 0 65536
#> 2 lon 4320 FALSE 1 65536 1 TRUE 1 65536
#> 3 rgb 3 FALSE 1 65536 2 FALSE 2 65536
#> 4 eigh⦠256 FALSE 1 65536 3 FALSE 3 65536
#> [1] "unlimdims"
#> NULL
#> [1] "dimension_values"
#> # A tibble: 6,739 Ã 3
#> id vals .dimension_
#> <int> <dbl> <int>
#> 1 0 90.0 0
#> 2 0 89.9 0
#> 3 0 89.8 0
#> 4 0 89.7 0
#> 5 0 89.6 0
#> 6 0 89.5 0
#> 7 0 89.5 0
#> 8 0 89.4 0
#> 9 0 89.3 0
#> 10 0 89.2 0
#> # â¹ 6,729 more rows
#> [1] "group"
#> # A tibble: 3 Ã 7
#> id name ndims nvars natts fqgn .group_
#> <int> <chr> <int> <int> <int> <chr> <int>
#> 1 65536 "" 4 4 65 "" 65536
#> 2 65537 "processing_control" 0 0 4 "processing_control" 65537
#> 3 65538 "input_parameters" 0 0 21 "processing_control/inpu⦠65538
#> [1] "file"
#> # A tibble: 1 Ã 12
#> filename writable id error safemode format is_GMT ndims natts unlimdimid
#> <chr> <lgl> <int> <lgl> <lgl> <chr> <lgl> <dbl> <dbl> <dbl>
#> 1 /perm_stor⦠FALSE 65536 FALSE FALSE NC_FO⦠FALSE 4 90 -1
#> # â¹ 2 more variables: nvars <dbl>, .file_ <int>
#> [1] "variable"
#> # A tibble: 2 Ã 18
#> name ndims natts prec units longname group_index storage shuffle compression
#> <chr> <int> <int> <chr> <chr> <chr> <int> <int> <int> <int>
#> 1 chlo⦠2 12 float "mg ⦠Chlorop⦠1 2 0 4
#> 2 pale⦠2 0 unsi⦠"" palette 1 1 0 NA
#> # â¹ 8 more variables: unlim <lgl>, make_missing_value <lgl>,
#> # hasAddOffset <lgl>, addOffset <dbl>, hasScaleFact <lgl>, scaleFact <dbl>,
#> # .variable_ <dbl>, .group_ <int>
#> [1] "vardim"
#> # A tibble: 4 Ã 2
#> .variable_ .dimension_
#> <dbl> <int>
#> 1 0 1
#> 2 0 0
#> 3 3 3
#> 4 3 2
#> [1] "attribute"
#> [1] "NetCDF attributes:"
#> [1] "Global"
#> [1] "\n"
#> # A tibble: 1 Ã 65
#> product_name instrument title project platform temporal_range
#> <chr> <chr> <chr> <chr> <chr> <chr>
#> 1 S2008001.L3m_DAY_CHL_chlor_a⦠SeaWiFS SeaW⦠Ocean ⦠Orbview⦠day
#> # â¹ 59 more variables: processing_version <chr>, date_created <chr>,
#> # history <chr>, l2_flag_names <chr>, time_coverage_start <chr>,
#> # time_coverage_end <chr>, start_orbit_number <int>, end_orbit_number <int>,
#> # map_projection <chr>, latitude_units <chr>, longitude_units <chr>,
#> # northernmost_latitude <dbl>, southernmost_latitude <dbl>,
#> # westernmost_longitude <dbl>, easternmost_longitude <dbl>,
#> # geospatial_lat_max <dbl>, geospatial_lat_min <dbl>, â¦
#> [1] "\n"
#> [1] "Variable attributes:"
#> character(0)The main problems with the existing packages are that they are either too high-level or too low-level and itâs this twilight zone in the middle where ncdump comes in, where the user needs to reach down a bit deeper to make things work. Using the wrappers around the raw API ncdf4 and RNetCDF is very raw, and for the most part is analogous to the NetCDF API itself, though you donât have to compile the code. These packages arenât the same, and neither can read all kinds of NetCDF files. Both do support API features like NetCDF 4, server interfaces (OpenDAP and Thredds and the like), support for HDF, and parallel processing but it depends on how you build the underlying libraries. The details on this are not on the agenda here.
The worker for most low-level NetCDF access is the command line utility ncdump -h - if not called with the header argument you can actually dump the data contents in various ways, including binary forms to other formats. The header is usually what you are expected to read to figure out how to program against the contents. Itâs not completely automatable, because itâs so general itâs unlikely that your generalized code will solve any particular practical task and thereâs no general libary for NetCDF that will give you practical outcomes for all types of data. Itâs a non-virtuous cycle.
The raster package is excellent for NetCDF, but it only works âoff-the-shelfâ for regularly xy-gridded data in 2D, 3D or 4D variables. The mapping of dimensions to these variables is patchy, because the format is so general and many data variables have rectilinear or curvilinear coordinates, which are either ignored by raster or cause it to stop without reading anything. If you know what you want you can get it to work with any 2D or higher variable. Raster is a high-level domain-specific library, for GIS-y rasters, or grids that use an affine transform to define their coordinate system in xy.
GDAL, available in rgdal has similar limitations to raster for the same affine/GIS reason, but also makes different choices when approaching higher dimensions. Essentially all higher dimensions get unrolled as extra bands for a GDAL data set. They are completely independently implemented, GDAL has an in-built NetCDF driver (also a GMT variant) and raster uses ncdf4 directly. (Raster will use rgdal when it can for many formats, but you cannot actually override rasterâs used of ncdf4 without uninstalling that package. Itâs all quite complicated and poorly understood unfortunately, and it keeps changing).
There is another package rhdf5 which provides excellent modern facilities for HDF5 and NetCDF4, and notably this can read NetCDF files with compound types that no other package can do. This would be perfect if it could also read NetCDF âclassic formatâ, i.e. version 3.x prior to NetCDF 4.
I use ncdump in conjunction with angstroms and various other projects. I can perform joins and select/filter tasks on the tables it returns, but itâs not designed in any high-level consistent way, it just gives all the information it can get.
A format method for the object above could recreate the print out you see from ncdump itself.
A future package may wrap the various NetCDF R packages to do the actual read, but each is useful in different ways. The main ones are RNetCDF, ncdf4 and rhdf5 (Bioconductor).
This project was split out of the âR and NetCDF interface developmentâ project. https://github.com/hypertidy/rancid
Code of ConductPlease note that the ncdump project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4