输入和输出numpy数组到h5py

问题:输入和输出numpy数组到h5py

我有一个Python代码,其输出为大小矩阵,其条目均为type float。如果使用扩展名保存,.dat则文件大小约为500 MB。我读到使用h5py会大大减少文件大小。因此,假设我有一个名为的2D numpy数组A。如何将其保存到h5py文件?另外,由于需要对数组进行操作,如何读取相同文件并将其作为numpy数组放入不同的代码中?

I have a Python code whose output is a sized matrix, whose entries are all of the type float. If I save it with the extension .dat the file size is of the order of 500 MB. I read that using h5py reduces the file size considerably. So, let’s say I have the 2D numpy array named A. How do I save it to an h5py file? Also, how do I read the same file and put it as a numpy array in a different code, as I need to do manipulations with the array?


回答 0

h5py提供了一个数据集模型。前者基本上是数组,而后者可以视为目录。每个都被命名。您应该查看API的文档和示例:

http://docs.h5py.org/en/latest/quick.html

一个简单的示例,其中您要先创建所有数据,然后只想将其保存到hdf5文件,如下所示:

In [1]: import numpy as np
In [2]: import h5py
In [3]: a = np.random.random(size=(100,20))
In [4]: h5f = h5py.File('data.h5', 'w')
In [5]: h5f.create_dataset('dataset_1', data=a)
Out[5]: <HDF5 dataset "dataset_1": shape (100, 20), type "<f8">

In [6]: h5f.close()

然后,您可以使用以下命令将数据加载回:

In [10]: h5f = h5py.File('data.h5','r')
In [11]: b = h5f['dataset_1'][:]
In [12]: h5f.close()

In [13]: np.allclose(a,b)
Out[13]: True

绝对看看文档:

http://docs.h5py.org

写入hdf5文件取决于h5py或pytables(每个文件都具有位于hdf5文件规范之上的不同python API)。您还应该看看numpy本机提供的其他简单二进制格式,例如np.savenp.savez等等:

http://docs.scipy.org/doc/numpy/reference/routines.io.html

h5py provides a model of datasets and groups. The former is basically arrays and the latter you can think of as directories. Each is named. You should look at the documentation for the API and examples:

http://docs.h5py.org/en/latest/quick.html

A simple example where you are creating all of the data upfront and just want to save it to an hdf5 file would look something like:

In [1]: import numpy as np
In [2]: import h5py
In [3]: a = np.random.random(size=(100,20))
In [4]: h5f = h5py.File('data.h5', 'w')
In [5]: h5f.create_dataset('dataset_1', data=a)
Out[5]: <HDF5 dataset "dataset_1": shape (100, 20), type "<f8">

In [6]: h5f.close()

You can then load that data back in using: ‘

In [10]: h5f = h5py.File('data.h5','r')
In [11]: b = h5f['dataset_1'][:]
In [12]: h5f.close()

In [13]: np.allclose(a,b)
Out[13]: True

Definitely check out the docs:

http://docs.h5py.org

Writing to hdf5 file depends either on h5py or pytables (each has a different python API that sits on top of the hdf5 file specification). You should also take a look at other simple binary formats provided by numpy natively such as np.save, np.savez etc:

http://docs.scipy.org/doc/numpy/reference/routines.io.html


回答 1

处理文件打开/关闭并避免内存泄漏的更干净的方法

准备:

import numpy as np
import h5py

data_to_write = np.random.random(size=(100,20)) # or some such

写:

with h5py.File('name-of-file.h5', 'w') as hf:
    hf.create_dataset("name-of-dataset",  data=data_to_write)

读:

with h5py.File('name-of-file.h5', 'r') as hf:
    data = hf['name-of-dataset'][:]

A cleaner way to handle file open/close and avoid memory leaks:

Prep:

import numpy as np
import h5py

data_to_write = np.random.random(size=(100,20)) # or some such

Write:

with h5py.File('name-of-file.h5', 'w') as hf:
    hf.create_dataset("name-of-dataset",  data=data_to_write)

Read:

with h5py.File('name-of-file.h5', 'r') as hf:
    data = hf['name-of-dataset'][:]