I have 78 netcdf files each around 17MB, with shape (time=1, x=2048, y=2048) to be merged spatially. The single timestep is shared for all 78 files. The collgrid merge command below was able to produce the output netcdf of size 1.3GB, but the memory usage during the merge was 5GB.
INPUT_DIR="/nctiles"
OUTPUT_FILE="/merged.nc"
/usr/bin/time -v cdo --single -r collgrid "$INPUT_DIR/Crop18_??_??.nc" "$OUTPUT_FILE"
Logs from the command: cdo collgrid: Processed 327155712 values from 78 variables over 78 timesteps [3.86s 5195MB].
An individual netcdf is a regular grid and has the following structure:
<xarray.Dataset> Size: 17MB
Dimensions: (x: 2048, y: 2048, time: 1)
Coordinates:
* x (x) float64 16kB 143.3 143.3 143.3 143.3 ... 143.5 143.5 143.5
* y (y) float64 16kB -34.8 -34.8 -34.8 -34.8 ... -35.0 -35.0 -35.0
* time (time) datetime64[ns] 8B 2018-01-01
Data variables:
var0 (time, y, x) float32 17MB ...
Each netcdf is EPSG:4326 at 10m resolution. Opening all files at once in the operation should just require 78 x 17MB = 1.3GB memory. I wonder if the significant operational memory usage (5GB) was the result of creating the output grid for all input netcdfs.
It would be greatly if there is a way to reduce the memory usage somehow. Thanks in advance for any idea!
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4