与常规Python列表相比,NumPy有什么优势?

问题:与常规Python列表相比,NumPy有什么优势?

与常规Python列表相比,NumPy有什么优势?

我大约有100个金融市场系列,我将创建一个100x100x100 = 1百万个单元的多维数据集数组。我将每个x与y和z回归(3变量),以用标准误差填充数组。

我听说对于“大型矩阵”,出于性能和可伸缩性的原因,我应该使用NumPy而不是Python列表。事实是,我知道Python列表,它们似乎对我有用。

如果我转到NumPy,会有什么好处?

如果我有1000个序列(即立方体中有10亿个浮点单元)怎么办?

What are the advantages of NumPy over regular Python lists?

I have approximately 100 financial markets series, and I am going to create a cube array of 100x100x100 = 1 million cells. I will be regressing (3-variable) each x with each y and z, to fill the array with standard errors.

I have heard that for “large matrices” I should use NumPy as opposed to Python lists, for performance and scalability reasons. Thing is, I know Python lists and they seem to work for me.

What will the benefits be if I move to NumPy?

What if I had 1000 series (that is, 1 billion floating point cells in the cube)?


回答 0

NumPy的数组比Python列表更紧凑-您在Python中描述的列表列表至少需要20 MB左右,而单元格中具有单精度浮点数的NumPy 3D数组则需要4 MB。使用NumPy可以更快地读取和写入项目。

也许您只关心一百万个单元就不会那么在意,但是您肯定会关心十亿个单元-两种方法都不适合32位体系结构,但是使用64位版本,NumPy可以节省约4 GB ,仅Python一项就至少需要12 GB(很多指针的大小加倍),这是一个昂贵得多的硬件!

差异主要是由于“间接性”造成的-Python列表是指向Python对象的指针的数组,每个指针至少4个字节,对于最小的Python对象也至少包含16个字节(类型指针为4,引用计数为4,类型为4值-内存分配器舍入为16)。NumPy数组是统一值的数组-单精度数字每个占用4个字节,双精度数字每个占用8个字节。灵活性较差,但您要为标准Python列表的灵活性付出高昂的代价!

NumPy’s arrays are more compact than Python lists — a list of lists as you describe, in Python, would take at least 20 MB or so, while a NumPy 3D array with single-precision floats in the cells would fit in 4 MB. Access in reading and writing items is also faster with NumPy.

Maybe you don’t care that much for just a million cells, but you definitely would for a billion cells — neither approach would fit in a 32-bit architecture, but with 64-bit builds NumPy would get away with 4 GB or so, Python alone would need at least about 12 GB (lots of pointers which double in size) — a much costlier piece of hardware!

The difference is mostly due to “indirectness” — a Python list is an array of pointers to Python objects, at least 4 bytes per pointer plus 16 bytes for even the smallest Python object (4 for type pointer, 4 for reference count, 4 for value — and the memory allocators rounds up to 16). A NumPy array is an array of uniform values — single-precision numbers takes 4 bytes each, double-precision ones, 8 bytes. Less flexible, but you pay substantially for the flexibility of standard Python lists!


回答 1

NumPy不仅效率更高;这也更加方便。您可以免费获得许多矢量和矩阵运算,有时这可以避免不必要的工作。而且它们也得到有效实施。

例如,您可以将多维数据集直接从文件读取到数组中:

x = numpy.fromfile(file=open("data"), dtype=float).reshape((100, 100, 100))

沿第二维求和:

s = x.sum(axis=1)

查找哪些单元格高于阈值:

(x > 0.5).nonzero()

删除沿第三维的每个偶数索引切片:

x[:, :, ::2]

同样,许多有用的库都可以与NumPy数组一起使用。例如,统计分析和可视化库。

即使您没有性能问题,学习NumPy也是值得的。

NumPy is not just more efficient; it is also more convenient. You get a lot of vector and matrix operations for free, which sometimes allow one to avoid unnecessary work. And they are also efficiently implemented.

For example, you could read your cube directly from a file into an array:

x = numpy.fromfile(file=open("data"), dtype=float).reshape((100, 100, 100))

Sum along the second dimension:

s = x.sum(axis=1)

Find which cells are above a threshold:

(x > 0.5).nonzero()

Remove every even-indexed slice along the third dimension:

x[:, :, ::2]

Also, many useful libraries work with NumPy arrays. For example, statistical analysis and visualization libraries.

Even if you don’t have performance problems, learning NumPy is worth the effort.


回答 2

Alex提到了内存效率,Roberto提到了便利性,这些都是不错的地方。对于其他一些想法,我将提到速度功能

功能性:NumPy,FFT,卷积,快速搜索,基本统计信息,线性代数,直方图等都内置了很多功能。实际上,没有FFT谁能活下去?

速度:这是一项对列表和NumPy数组求和的测试,表明NumPy数组的求和速度快10倍(在此测试中,里程可能会有所不同)。

from numpy import arange
from timeit import Timer

Nelements = 10000
Ntimeits = 10000

x = arange(Nelements)
y = range(Nelements)

t_numpy = Timer("x.sum()", "from __main__ import x")
t_list = Timer("sum(y)", "from __main__ import y")
print("numpy: %.3e" % (t_numpy.timeit(Ntimeits)/Ntimeits,))
print("list:  %.3e" % (t_list.timeit(Ntimeits)/Ntimeits,))

在我的系统上(运行备份时),它会给出:

numpy: 3.004e-05
list:  5.363e-04

Alex mentioned memory efficiency, and Roberto mentions convenience, and these are both good points. For a few more ideas, I’ll mention speed and functionality.

Functionality: You get a lot built in with NumPy, FFTs, convolutions, fast searching, basic statistics, linear algebra, histograms, etc. And really, who can live without FFTs?

Speed: Here’s a test on doing a sum over a list and a NumPy array, showing that the sum on the NumPy array is 10x faster (in this test — mileage may vary).

from numpy import arange
from timeit import Timer

Nelements = 10000
Ntimeits = 10000

x = arange(Nelements)
y = range(Nelements)

t_numpy = Timer("x.sum()", "from __main__ import x")
t_list = Timer("sum(y)", "from __main__ import y")
print("numpy: %.3e" % (t_numpy.timeit(Ntimeits)/Ntimeits,))
print("list:  %.3e" % (t_list.timeit(Ntimeits)/Ntimeits,))

which on my systems (while I’m running a backup) gives:

numpy: 3.004e-05
list:  5.363e-04

回答 3

这是scipy.org网站上的常见问题解答中的一个很好的答案:

与(嵌套)Python列表相比,NumPy数组有什么优势?

Python的列表是有效的通用容器。它们支持(相当)高效的插入,删除,附加和连接,并且Python的列表理解使它们易于构造和操作。但是,它们有一定的局限性:它们不支持“向量化”操作,例如逐元素加法和乘法,并且它们可以包含不同类型的对象这一事实意味着Python必须为每个元素存储类型信息,并且必须执行类型分派代码在每个元素上操作时。这也意味着有效的C循环几乎无法执行列表操作-每次迭代都需要类型检查和其他Python API簿记。

Here’s a nice answer from the FAQ on the scipy.org website:

What advantages do NumPy arrays offer over (nested) Python lists?

Python’s lists are efficient general-purpose containers. They support (fairly) efficient insertion, deletion, appending, and concatenation, and Python’s list comprehensions make them easy to construct and manipulate. However, they have certain limitations: they don’t support “vectorized” operations like elementwise addition and multiplication, and the fact that they can contain objects of differing types mean that Python must store type information for every element, and must execute type dispatching code when operating on each element. This also means that very few list operations can be carried out by efficient C loops – each iteration would require type checks and other Python API bookkeeping.


回答 4

所有人都强调了numpy数组和python列表之间的几乎所有主要区别,在这里我将向大家简单介绍一下:

  1. Numpy数组在创建时具有固定的大小,这与python列表(可以动态增长)不同。更改ndarray的大小将创建一个新数组并删除原始数组。

  2. Numpy数组中的所有元素都必须具有相同的数据类型(我们也可以具有异构类型,但这将不允许您进行数学运算),因此在内存中的大小将相同

  3. Numpy数组有助于对大量数据进行数学运算和其他类型的运算。通常,与使用python顺序构建相比,此类操作执行效率更高且代码更少

All have highlighted almost all major differences between numpy array and python list, I will just brief them out here:

  1. Numpy arrays have a fixed size at creation, unlike python lists (which can grow dynamically). Changing the size of ndarray will create a new array and delete the original.

  2. The elements in a Numpy array are all required to be of the same data type (we can have the heterogeneous type as well but that will not gonna permit you mathematical operations) and thus will be the same size in memory

  3. Numpy arrays are facilitated advances mathematical and other types of operations on large numbers of data. Typically such operations are executed more efficiently and with less code than is possible using pythons build in sequences