套件：python3-kerchunk（0.2.9-2）
Cloud-friendly access to archival data
Kerchunk is a library that provides a unified way to represent a variety of chunked, compressed data formats (e.g. NetCDF, HDF5, GRIB), allowing efficient access to the data from traditional file systems or cloud object storage. It also provides a flexible way to create virtual datasets from multiple files. It does this by extracting the byte ranges, compression information and other information about the data and storing this metadata in a new, separate object. This means that you can create a virtual aggregate dataset over potentially many source files, for efficient, parallel and cloud-friendly *in-situ* access without having to copy or translate the originals. It is a gateway to in-the-cloud massive data processing while the data providers still insist on using legacy formats for archival storage.
Features:
* completely serverless architecture * metadata consolidation, so you can understand a many-file dataset (metadata plus physical storage) in a single read * read from all of the storage backends supported by fsspec, including object storage (s3, gcs, abfs, alibaba), http, cloud user storage (dropbox, gdrive) and network protocols (ftp, ssh, hdfs, smb...) * loading of various file types (currently netcdf4/HDF, grib2, tiff, fits, zarr), potentially heterogeneous within a single dataset, without a need to go via the specific driver (e.g., no need for h5py) * asynchronous concurrent fetch of many data chunks in one go, amortizing the cost of latency * parallel access with a library like zarr without any locks * logical datasets viewing many (>~millions) data files, and direct access/subselection to them via coordinate indexing across an arbitrary number of dimensions
其他與 python3-kerchunk 有關的套件
|
|
|
|
-
- dep: python3
- interactive high-level object-oriented language (default python3 version)
-
- dep: python3-fsspec
- specification that Python filesystems should adhere to (Python 3)
-
- dep: python3-numcodecs
- buffer compression and transformation codecs for Python
-
- dep: python3-numpy
- Python library for numerical computations (Python 3)
-
- dep: python3-ujson
- ultra fast JSON encoder and decoder for Python 3
-
- dep: python3-zarr
- chunked, compressed, N-dimensional arrays for Python
-
- rec: python3-cfgrib
- Python 3 module supporting the CF convention in GRIB files
-
- rec: python3-cftime
- Time-handling functionality from netcdf4-python (Python 3)
-
- rec: python3-h5py
- general-purpose Python interface to hdf5
-
- rec: python3-scipy
- scientific tools for Python 3
-
- rec: python3-xarray
- N-D labeled arrays and datasets in Python 3
-
- sug: python3-aiohttp
- http client/server for asyncio
-
- sug: python3-dask
- Minimal task scheduling abstraction for Python 3
-
- sug: python3-netcdf4
- Python 3 interface to the netCDF4 (network Common Data Form) library