Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://github.com/EBVcube/EBVCube-format below:

EBVcube/EBVCube-format: Definition of the EBVCube netCDF format.

This document defines the EBVCube format. This specification was developed at the Biodiversity and Conservation core group at iDiv: Luise Quoß , Christian Langer , Lina Estupinan Suarez , Miguel Fernandez , Andres Marmol , Emmanuel Oceguera , Jose Valdez , Nestor Fernandez and Henrique M. Pereira .

The files are based on the Network Common Data Form (netCDF). Additionally, it follows the Climate and Forecast Conventions (CF, version 1.8) and the Attribute Convention for Data Discovery (ACDD, version 1.3). This data format complements the Essential Biodiversity Variables framework (EBV).

The EBVCube netCDF file structure supports multiple data cubes. These cubes have four dimensions: longitude, latitude, time and entity, whereby the last dimension can, e.g., encompass various biological or ecological categories, such as species, species groups, ecosystem types, or other groupings. Visualizing a 4D cube can be a challenge. To facilitate its abstrasction, think of it as a data set where each pixel contains four pieces of information: spatial position (latitude and longitude), date (time), and the entity it represents (e.g., a species) (see figure 1).

The use of hierarchical groups allows multiple data cubes to coexist, with common dimensions across all cubes. The first hierarchical level (netCDF group) represents scenarios, such as various Shared Socioeconomic Pathways (SSP) scenarios used in modeling. The second hierarchical level (netCDF group) represents metrics, such as the percentage of protected area per pixel or the proportional loss over a certain time span per pixel. While the scenario-level is optional (no mandatory), each EBVCube netCDF must include at least one metric. If scenarios are included, all metrics must be repeated for each scenario (see figure 2). The number of scenarios and metrics included in the data sets can and will vary.

The EBV data cubes (netCDF variables named 'ebv_cube') are defined as four dimensional arrays: longitude, latitude, time and entity. These dimensions are defined at the root level of the netCDF file, ensuring the consistency of the data cubes. The longitude and latitude spatial dimensions determinate the geographical extent and resolution of the data, while time dimension is the only unlimited dimension. This design enables the data sets to be updated over time by incorporing new temporal EBV monitoring information. Each of these three dimensions (longitude, latitude, and time) is accompanied by a corresponding coordinate variable at the root level, following the CF convention. The fourth dimension, the entity, can encompass a range of biological and ecological categories such as individual species, species groups, ecosystem types included in any EBV measurement. This information is stored in a character array named ‘entity’ at the root level and is referred to in CF terminology as an auxiliary coordinate variable.

Summary:

• Two possible nested sub-groups: scenario and metric
• Metric is a mandatory group, scenario is optional
• The scenario group is always higher than the metric
• If several metrics are present, they need to be repeated for all scenarios
• Hence several 4D data cubes (one per scenario-metric-path or metric-path) are possible
• The dimensions of the 4D data cubes are: longitude, latitude, time and entity

This is a schematic, rather technical representation of the netCDF structure of EBVCube data that incorporates the optional scenarios (note: more scenarios and/or metrics are possible). In contrast to the figure above it covers all components of the netCDF including the dimensions, coordinate variables and georeferencing components. There are ATTRIBUTES at various levels and components. These are listed in the tables below in the Metadata section.

If you have modeled your data for different scenarios, e.g. for the SSP scenarios, the Global trends in biodiversity (BES-SIM PREDICTS) data set by Samantha Hill is a good example to follow. FYI: this data set only has one entity: Alltaxa.

┌── root level
├── GLOBAL ATTRIBUTES
├── Dimensions: entity, time, lat, lon, nchar
├── crs [0]
|   └── ATTRIBUTES
├── lat [lat]
|   └── ATTRIBUTES
├── lon [lon]
|   └── ATTRIBUTES
├── time [time]
|   └── ATTRIBUTES
├── entity [entity, nchar]
|   └── ATTRIBUTES
├── scenario_1
|   ├── ATTRIBUTES
│   ├── metric_1 
|   |   ├── ATTRIBUTES
|   |   └── ebv_cube [entity, time, lat, lon] 
|   |       └── ATTRIBUTES
|   └── metric_2
|       ├── ATTRIBUTES
|       └── ebv_cube [entity, time, lat, lon] 
|           └── ATTRIBUTES
└── scenario_2
|   ├── ATTRIBUTES
│   ├── metric_1 
|   |   ├── ATTRIBUTES
|   |   └── ebv_cube [entity, time, lat, lon] 
|   |       └── ATTRIBUTES
|   └── metric_2
|       ├── ATTRIBUTES
|       └── ebv_cube [entity, time, lat, lon] 
|           └── ATTRIBUTES
...

The following representation follows the same style as example 1 above. The difference is that this is the minimum EBVCube data set you can create: no scenarios and only one metric. Of course, an EBVCube data set can also contain no scenarios, but several metrics.

If your data set follows this or a similar structure, the Habitat availability for African great apes data set by Jessica Junker is a good example to follow. FYI: this data set has seven entities – one per great apes species.

┌── root level
├── GLOBAL ATTRIBUTES
├── DIMENSIONS: entity, time, lat, lon, nchar
├── crs [0]
|   └── ATTRIBUTES
├── lat [lat]
|   └── ATTRIBUTES
├── lon [lon]
|   └── ATTRIBUTES
├── time [time]
|   └── ATTRIBUTES
├── entity [entity, nchar]
|   └── ATTRIBUTES
└── metric_1 
    ├── ATTRIBUTES
    └── ebv_cube [entity, time, lat, lon] 
        └── ATTRIBUTES

In contrast to the first two examples, this one includes the variables that are added to the netCDF when taxonomic information is given. For a detailed description see the taxonomy section.

┌── root level
├── GLOBAL ATTRIBUTES
├── Dimensions: entity, time, lat, lon, nchar, taxonlevel, nchar_taxonlist, nchar_taxonid
├── crs [0]
|   └── ATTRIBUTES
├── lat [lat]
|   └── ATTRIBUTES
├── lon [lon]
|   └── ATTRIBUTES
├── time [time]
|   └── ATTRIBUTES
├── entity [entity, nchar]
|   └── ATTRIBUTES
├── taxonomy_levels [nchar_taxonlist, taxonlevel]
|   └── ATTRIBUTES
├── taxonomy_table [nchar, entity, taxonlevel]
|   └── ATTRIBUTES
├── taxonomy_key [nchar_taxonid, entity]
|   └── ATTRIBUTES
└── metric_1 
    ├── ATTRIBUTES
    └── ebv_cube [entity, time, lat, lon] 
        └── ATTRIBUTES

The following tables describe the attributes in the EBV netCDF files. Each table corresponds to a different component in the netCDF. The descriptions of the attributes that are derived from the ACDD, are directly cited from the ACDD 1.3 documentation. The fifth column (User Input) marks all the attributes that need to be defined by the publisher at the upload form at the EBV Data Portal. The sixth column (Mandatory) shows wether this input by the publisher is mandatory.
Note: These attributes differ in part from those in the metadata files (XML, JSON) in the EBV Data Portal, as they also include the netCDF-specific, often more technical attributes. The How-To of the EBV Data Portal explains the terms of the upload form and maps them to the netCDF attributes described below.

The global netCDF attributes are those that can be found at the root (global) level of the netCDF file. The table follows the order of the attributes in the JSON files of the EBV Data Portal. The additional terms are then listed.

The scenario and the metric are netCDF groups. They are nested. The scenario is the higher level, but unlike the metric, it is not mandatory. The metrics are repeated in all scenarios (if applicable). The following attribtues are found at the metric- and scenario-level. These attributes can also be found in the JSON files.

The data cubes (ebv_cube) are netCDF variables. There is one data cube per (scenario-) metric-path in the netCDFs. Hence, multiple data cubes can be found in one EBVCube netCDF. Only the coverage_content_type attribute is present in the JSON files. The other attributes repeat metric-information and cover technical aspects. For example, the grid_mapping attribute points to the variable in the netCDF that holds the coordiante reference related attributes (see section Coordinate reference system attributes). The coordinates attribute is pointing to the auxiliary coordinate variable

The lat and lon are coordinate variables. The lon and lat dimensions are the basis for these two one-dimensional vectors, which contain the lon/lat values of the CRS of the data set. The lat and lon attributes follow the CF convention for Horizontal Coordinate Reference Systems. These attributes cannot be found in the JSON files.

The time is a coordinate variables. This one-dimensional vector is based on the time dimension. It holds the 'days since 1860' as integer values. These attributes cannot be found in the JSON files.

The entity variable is an auxiliary coordinate variable and stores all entity names as a character array. The ebv_entity_* attributes are also included in the JSON files.

All attributes regarding the georeferencing can be found at the 'crs' variable in the EBVCube netCDFs. The georeferencing is following the grip mappings by the CF convention. Therefore the attributes differ based on the coordinate reference system. Read the CF convention section for more information. Additionally, the GeoTransform and spatial_ref attributes are added based on the netCDF definitions by GDAL. These attributes cannot be found in the JSON files.

FYI: In principle, you can assign all CRSs available in the PROJ library to an EBVCube netCDF. The only restriction is currently the visualization in the map of the EBV Data Portal by the company GeoEngine, which only works for EPSG-based CRSs.

If the dataset encompasses taxonomic information at least two new variables are added. The taxonomy_key variable is only added if the user also provides the taxonomic keys. If an EBVCube netCDF holds taxonomic information, the ebv_entity_classification_name and ebv_entity_classification_url, associated to the entity variable, point to the taxonomic backbone and its URL.

If the entities in your dataset follow a taxonomy, you can also add this information to an EBVCube dataset. The taxonomy can cover species as well as habitat types and more.

For example the Occurrence Metrics for Invasive Alien Species of Union Concern in EU27: A 10 km prototype using GBIF occurrence cubes dataset hold species data following the GBIF taxonomic backbone. The taxonomic information is directly added during the creation of the netCDF file. You can follow the code here.

The ebvcube R package retrieves the taxonomic information for you (see below), and it is also displayed per dataset on the EBV Data Portal detail pages. Further developments are currently underway. This encompasses for example the display of the taxonomy in the EBVCubeVisualizer and the integration into the EBV Data Portal API.

To store the taxonomic information two netCDF variables (character arrays) are added to the netCDF. The 'entity_taxonomy_levels' and the 'entity_taxonomy_table'.

The 'entity_taxonomy_levels' is a 2D array (dimensions: nchar_taxonlist, taxonlevel) that holds the names of the different taxonomy levels, e.g. 'species', 'genus', 'family', 'order', 'class', 'phylum' and 'kingdom'. The 'entity_taxonomy_table' is a 3D array (dimensions: nchar, entity, taxonlevel) that holds the values of all taxonomy levels per entity, e.g. for one entity 'Accipiter brevipes', 'Accipiter', 'Accipitridae', 'Accipitriformes', 'Aves', 'Chordata' and 'Animalia'. A schematic representation can be found in example 3.

FYI: these variables were called 'entity_levels' and 'entity_list' until R package version 0.4.0. The renaming was done in the beginning of April in 2025. The later R package version are able to handle both namings.

To get the taxonomic information with R run the following code:

#import packages
library(ebvcube)

#download the EBVCube file
dir <- tempdir()
filepath <- ebv_download(id = 83,
                         outputdir = dir)

#read properties
prop <- ebv_properties(filepath, verbose=F)

#get the taxonomy
taxonomy <- prop@general$taxonomy
  
#print the first line
print(taxonomy[1,])
#         species  genus   family   order         class       phylum kingdom
#1 Acacia saligna Acacia Fabaceae Fabales Magnoliopsida Tracheophyta Plantae

To get the taxonomic information with Python run the following code:

#import packages
import netCDF4 as nc
import numpy as np
import tempfile
import wget

#download the EBVCube file
url = 'https://portal.geobon.org/data/upload/82/public/suarez_spepop_id82_20240820_v1.nc'
temp_dir = tempfile.TemporaryDirectory()
filepath = wget.download(url, out = temp_dir.name)

#open netCDF file read-only
rootgrp = nc.Dataset(filepath,"r")

#get the taxonomy levels
tax_level = np.transpose(np.array(rootgrp['entity_levels'])) #new naming: taxonomy_levels
tax_level_list = [tax_level[i,:].tobytes().decode('UTF-8').strip() for i in range(tax_level.shape[0])]
#print the taxon levels
print(tax_level_list)
# Out: ['species', 'genus', 'family', 'order', 'class', 'phylum', 'kingdom']

#get the taxonomy list data and transform it into a dictionary
tax_list = np.transpose(np.array(rootgrp['entity_list'])) #new naming: taxonomy_table
#collect taxonomy of this dataset
result_tax = dict()
result_tax['entity_id'] = tax_level_list
for entity in range(tax_list.shape[1]):
    row = [tax_list[:,entity,:][i,:].tobytes().decode('UTF-8').strip() for i in range(tax_list[:,entity,:].shape[0])]
    result_tax[(entity+1)] = row
#print the first row
print(result_tax[1])
# Out: ['Acacia saligna', 'Acacia', 'Fabaceae', 'Fabales', 'Magnoliopsida', 'Tracheophyta', 'Plantae']

#close netCDF file
rootgrp.close()

To discover the EBV netCDFs in full detail, we recommend Panoply. This is a software developed by NASA to generally open HDF5/netCDF files. This tool allows you to see all components of the netCDF including the internal hierarchy and all attribues. Besides, it has a plot function. This can be a bit overwhelming as you also see all the distributed technical components and attributes. But it is a very nice way to deeply understand the EBV netCDFs without code.

You can also download and open the EBVCube netCDFs directly in your R code with the ebvcube R package. This package bundles all the important metadata for you and hides all ‘unnecessary’ technical stuff. Also, you can directly start working with the data. It provides some easy high-level functions to, e.g., directly visualize or subset the data as you like. As the data slices are (also) available as terra SpatRasters, you can include the ebvcube code snippets in any geospatial analysis you are performing with the terra tools.

In addition, we have developed a QGIS plugin called EBVCubeVisualizer. This plugin allows the user to explore the full metadata and hierarchical structure of EBVCube datasets, similar to the Panoply software - but directly in QGIS. It strikes a good balance between displaying the full structure and metadata while hiding technical components and attributes (in contrast to Panoply) for improved user-friendliness. It allows users to extract and visualize specific slices from EBV data cubes, with flexible selection by time, entity, scenario and metric. As with all other layers in QGIS, you can use all geospatial tools directly.

The creation of the EBVCube netCDF is supported via the ebvcube R package. You can find the most recent development version here on GitHub. The package is also published on CRAN. The Readme of the GitHub repository explains shortly the workflow for the creation. The How-To on the EBV Data Portal has a section ('8. Training resources') summarizing different ressources including code examples for the creation of EBVCube netCDFs.

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4