Paket: python-kerchunk-doc (0.2.9-2)

[ Källkod: kerchunk  ]

Cloud-friendly access to archival data (documentation)

Kerchunk is a library that provides a unified way to represent a variety of chunked, compressed data formats (e.g. NetCDF, HDF5, GRIB), allowing efficient access to the data from traditional file systems or cloud object storage. It also provides a flexible way to create virtual datasets from multiple files. It does this by extracting the byte ranges, compression information and other information about the data and storing this metadata in a new, separate object. This means that you can create a virtual aggregate dataset over potentially many source files, for efficient, parallel and cloud-friendly *in-situ* access without having to copy or translate the originals. It is a gateway to in-the-cloud massive data processing while the data providers still insist on using legacy formats for archival storage.

Features:

 * completely serverless architecture
 * metadata consolidation, so you can understand a many-file dataset
   (metadata plus physical storage) in a single read
 * read from all of the storage backends supported by fsspec,
   including object storage (s3, gcs, abfs, alibaba), http, cloud user
   storage (dropbox, gdrive) and network protocols (ftp, ssh, hdfs,
   smb...)
 * loading of various file types (currently netcdf4/HDF, grib2, tiff,
   fits, zarr), potentially heterogeneous within a single dataset,
   without a need to go via the specific driver (e.g., no need for
   h5py)
 * asynchronous concurrent fetch of many data chunks in one go,
   amortizing the cost of latency
 * parallel access with a library like zarr without any locks
 * logical datasets viewing many (>~millions) data files, and direct
   access/subselection to them via coordinate indexing across an
   arbitrary number of dimensions

This package provides the HTML documentation for kerchunk.

Andra paket besläktade med python-kerchunk-doc

• dep: libjs-jquery (>= 3.6.0)

  JavaScript library for dynamic web applications

• dep: libjs-sphinxdoc (>= 8.2)

  JavaScript support for Sphinx documentation

• dep: sphinx-rtd-theme-common (>= 3.1.0+dfsg)

  sphinx theme from readthedocs.org (common files)

• sug: www-browser

  virtuellt paket som tillhandahålls av chromium, dillo, edbrowse, elinks, epiphany-browser, falkon, firefox, firefox-esr, konqueror, links, links2, luakit, lynx, morph-browser, netrik, netsurf-fb, netsurf-gtk, privacybrowser, qutebrowser, sugar-browse-activity, surf, vimb, w3m, xemacs21-gnome-mule, xemacs21-gnome-mule-canna-wnn, xemacs21-gnome-nomule, xemacs21-mule, xemacs21-mule-canna-wnn, xemacs21-nomule

Hämta python-kerchunk-doc

Hämtningar för alla tillgängliga arkitekturer

Arkitektur	Paketstorlek	Installerad storlek	Filer
all	695,8 kbyte	2.474,0 kbyte	[filförteckning]