Paquet : python3-kerchunk (0.2.9-2)

[ Paquet source : kerchunk  ]

Cloud-friendly access to archival data

Kerchunk is a library that provides a unified way to represent a variety of chunked, compressed data formats (e.g. NetCDF, HDF5, GRIB), allowing efficient access to the data from traditional file systems or cloud object storage. It also provides a flexible way to create virtual datasets from multiple files. It does this by extracting the byte ranges, compression information and other information about the data and storing this metadata in a new, separate object. This means that you can create a virtual aggregate dataset over potentially many source files, for efficient, parallel and cloud-friendly *in-situ* access without having to copy or translate the originals. It is a gateway to in-the-cloud massive data processing while the data providers still insist on using legacy formats for archival storage.

Features:

 * completely serverless architecture
 * metadata consolidation, so you can understand a many-file dataset
   (metadata plus physical storage) in a single read
 * read from all of the storage backends supported by fsspec,
   including object storage (s3, gcs, abfs, alibaba), http, cloud user
   storage (dropbox, gdrive) and network protocols (ftp, ssh, hdfs,
   smb...)
 * loading of various file types (currently netcdf4/HDF, grib2, tiff,
   fits, zarr), potentially heterogeneous within a single dataset,
   without a need to go via the specific driver (e.g., no need for
   h5py)
 * asynchronous concurrent fetch of many data chunks in one go,
   amortizing the cost of latency
 * parallel access with a library like zarr without any locks
 * logical datasets viewing many (>~millions) data files, and direct
   access/subselection to them via coordinate indexing across an
   arbitrary number of dimensions

Autres paquets associés à python3-kerchunk

• dep: python3

  langage orienté objet interactif de haut niveau – version par défaut de Python 3

• dep: python3-fsspec

  specification that Python filesystems should adhere to (Python 3)

• dep: python3-numcodecs

  compression en tampon et codecs de transformation pour Python

• dep: python3-numpy

  bibliothèque de Python pour des calculs numériques – Python 3

• dep: python3-ujson

  ultra fast JSON encoder and decoder for Python 3

• dep: python3-zarr

  tableaux à N dimensions fragmentés, compressés avec Python

• rec: python3-cfgrib

  Python 3 module supporting the CF convention in GRIB files

• rec: python3-cftime

  fonctionnalité de gestion de temps de netcdf4-python – Python 3

• rec: python3-h5py

  interface généraliste de Python pour hdf5

• rec: python3-scipy

  outils scientifiques pour Python 3

• rec: python3-xarray

  tableaux étiquetés et ensembles de données à N dimensions en Python 3

• sug: python3-aiohttp

  client/serveur HTTP pour asyncio

• sug: python3-dask

  abstraction minimale de planification de tâches pour Python 3

• sug: python3-netcdf4

  interface de Python 3 pour la bibliothèque netCDF4 (network Common Data Form)

Télécharger python3-kerchunk

Télécharger pour toutes les architectures proposées

Architecture	Taille du paquet	Espace occupé une fois installé	Fichiers
all	456,1 ko	3 972,0 ko	[liste des fichiers]