RetroSearch Browse

Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://github.com/sozip/sozip-spec below:

sozip/sozip-spec: Specification of seek-optimized zip file profile

This repository contains the specification for the SOZip (Seek-Optimized Zip) profile to the ZIP file format.

A Seek-Optimized ZIP file (SOZip) is a ZIP file that contains one or several Deflate-compressed files that are organized and annotated such that a SOZip-aware reader can perform very fast random access (seek) within a compressed file.

SOZip makes it possible to access large compressed files directly from a .zip file without prior decompression. It is not a new file format, but a profile of the existing ZIP format, done in a fully backward compatible way. ZIP readers that are non-SOZip aware can read a SOZip-enabled file normally and ignore the extended features that support efficient seek capability.

: C/C++ open source geospatial library. Try out its sozip branch, pending future inclusion in its master branch.
Python sozipfile module: drop-in replacement for standard zipfile module, creating and reading SOZip-enabled files.

See Annex A: Software implementations for more details.

Examples of SOZip-enabled files can be found in the sozip-examples repository.

Other ZIP related specification

This GitHub organization also hosts the KeyValuePairs extra-field specification, to be able to encode arbitrary key-value pairs of metadata associated with a file within a ZIP. For example to store the Content-Type of a file.

Done with GDAL sozip branch, on a laptop running a Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz (6 cores / 12 virtual CPUs).

ZIP generation:

Timing Action 6.1 s Multithreaded (12 vCPUs) generation of 489 MB SOZip-enabled file from a 1.6 GB uncompresssed GeoPackage file with
sozip nz-building-outlines.gpkg.zip nz-building-outlines.gpkg 36 s Single threaded compression of same file to 480 MB with regular zip utility with
zip nz-building-outlines-regular.gpkg.zip nz-building-outlines.gpkg

Bulk reading: Multithreaded extraction (4 vCPUs) of 3.2 million features with Arrow Array interface

Timing Action 1.2 s from SOZip-compressed GeoPackage file with
bench_ogr_batch nz-building-outlines.gpkg.zip 0.7 s from uncompressed GeoPackage file with
bench_ogr_batch nz-building-outlines.gpkg

Subsetting: Extraction of 66,377 features with a spatial filter

Timing Action 1.2 s from SOZip-compressed GeoPackage file with
ogr2ogr out.gpkg nz-building-outlines.gpkg.zip -spat 1740000 5910000 1750000 5920000 1.1 s from uncompressed GeoPackage file with
ogr2ogr out.gpkg nz-building-outlines.gpkg -spat 1740000 5910000 1750000 5920000

Extraction of one feature from its identifier:

Timing Action 45 ms from SOZip-compressed GeoPackage file with
ogr2ogr out.gpkg nz-building-outlines.gpkg.zip -fid 1000000 44 ms from uncompressed GeoPackage file with
ogr2ogr out.gpkg nz-building-outlines.gpkg -fid 1000000

We welcome contributions to this specification as issues, pull requests or discussions.

If you use SOZip or plan to use it for your data delivery, or consider doing a SOZip implementation, etc., let us know!

Datasets available as SOZip

TUDelft3D delivers the 3DBAG data, an open 3D building dataset, for the whole of the Netherlands, 108 GB uncompressed, as a 18.8 GB large GeoPackage: https://3dbag.nl/en/download
Raw Data API, from Humanitarian OpenStreetMap Team (HOTOSM), now uses SOZip when you download openstreetmap data .zip files through it.

Find me on

The SOZip specification and its GDAL implementation have been developed by Spatialys, with support from Safe Software

RetroSearch is an open source project built by @garambo | Open a GitHub Issue

Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo

HTML: 3.2 | Encoding: UTF-8 | Version: 0.7.4