标签归档：numpy

在numpy数组中查找最接近的值

2021年7月27日 Python实用宝典

问题：在numpy数组中查找最接近的值

是否有numpy-thonic方法（例如函数）在数组中查找最接近的值？

例：

np.find_nearest( array, value )

Is there a numpy-thonic way, e.g. function, to find the nearest value in an array?

Example:

np.find_nearest( array, value )

回答 0

import numpy as np
def find_nearest(array, value):
    array = np.asarray(array)
    idx = (np.abs(array - value)).argmin()
    return array[idx]

array = np.random.random(10)
print(array)
# [ 0.21069679  0.61290182  0.63425412  0.84635244  0.91599191  0.00213826
#   0.17104965  0.56874386  0.57319379  0.28719469]

value = 0.5

print(find_nearest(array, value))
# 0.568743859261

import numpy as np
def find_nearest(array, value):
    array = np.asarray(array)
    idx = (np.abs(array - value)).argmin()
    return array[idx]

array = np.random.random(10)
print(array)
# [ 0.21069679  0.61290182  0.63425412  0.84635244  0.91599191  0.00213826
#   0.17104965  0.56874386  0.57319379  0.28719469]

value = 0.5

print(find_nearest(array, value))
# 0.568743859261

回答 1

如果您的数组已排序并且非常大，则这是一个更快的解决方案：

def find_nearest(array,value):
    idx = np.searchsorted(array, value, side="left")
    if idx > 0 and (idx == len(array) or math.fabs(value - array[idx-1]) < math.fabs(value - array[idx])):
        return array[idx-1]
    else:
        return array[idx]

这可以扩展到非常大的阵列。如果您不能假定数组已经排序，则可以轻松修改上面的内容以对方法进行排序。对于小型阵列而言，这是过大的杀伤力，但是一旦阵列变大，速度就会更快。

IF your array is sorted and is very large, this is a much faster solution:

def find_nearest(array,value):
    idx = np.searchsorted(array, value, side="left")
    if idx > 0 and (idx == len(array) or math.fabs(value - array[idx-1]) < math.fabs(value - array[idx])):
        return array[idx-1]
    else:
        return array[idx]

This scales to very large arrays. You can easily modify the above to sort in the method if you can’t assume that the array is already sorted. It’s overkill for small arrays, but once they get large this is much faster.

回答 2

稍作修改，上面的答案就可以用于任意维数（1d，2d，3d等）的数组：

def find_nearest(a, a0):
    "Element in nd array `a` closest to the scalar value `a0`"
    idx = np.abs(a - a0).argmin()
    return a.flat[idx]

或者，写成一行：

a.flat[np.abs(a - a0).argmin()]

With slight modification, the answer above works with arrays of arbitrary dimension (1d, 2d, 3d, …):

def find_nearest(a, a0):
    "Element in nd array `a` closest to the scalar value `a0`"
    idx = np.abs(a - a0).argmin()
    return a.flat[idx]

Or, written as a single line:

a.flat[np.abs(a - a0).argmin()]

回答 3

答案摘要：如果已排序，array则二等分代码（如下所示）执行最快。大型阵列的速度要快〜100-1000倍，小型阵列的速度要快〜2-100倍。它也不需要numpy。如果您有一个未排序的，array则如果array为大，则应首先考虑使用O（n logn）排序，然后再按等分；如果array为小，则方法2似乎是最快的。

首先，您应该弄清最近值的含义。通常人们想要一个横坐标的间隔，例如array = [0,0.7,2.1]，value = 1.95，答案将是idx = 1。我怀疑您是这种情况（否则，一旦找到间隔，可以使用后续条件语句很容易地修改以下内容）。我将注意到，执行此操作的最佳方法是使用二分法（我将首先提供它-请注意，它根本不需要numpy，并且比使用numpy函数要快，因为它们执行冗余操作）。然后，我将与其他用户在此处介绍的其他项目进行时间比较。

二等分：

def bisection(array,value):
    '''Given an ``array`` , and given a ``value`` , returns an index j such that ``value`` is between array[j]
    and array[j+1]. ``array`` must be monotonic increasing. j=-1 or j=len(array) is returned
    to indicate that ``value`` is out of range below and above respectively.'''
    n = len(array)
    if (value < array[0]):
        return -1
    elif (value > array[n-1]):
        return n
    jl = 0# Initialize lower
    ju = n-1# and upper limits.
    while (ju-jl > 1):# If we are not yet done,
        jm=(ju+jl) >> 1# compute a midpoint with a bitshift
        if (value >= array[jm]):
            jl=jm# and replace either the lower limit
        else:
            ju=jm# or the upper limit, as appropriate.
        # Repeat until the test condition is satisfied.
    if (value == array[0]):# edge cases at bottom
        return 0
    elif (value == array[n-1]):# and top
        return n-1
    else:
        return jl

现在，我将从其他答案中定义代码，它们每个都返回一个索引：

import math
import numpy as np

def find_nearest1(array,value):
    idx,val = min(enumerate(array), key=lambda x: abs(x[1]-value))
    return idx

def find_nearest2(array, values):
    indices = np.abs(np.subtract.outer(array, values)).argmin(0)
    return indices

def find_nearest3(array, values):
    values = np.atleast_1d(values)
    indices = np.abs(np.int64(np.subtract.outer(array, values))).argmin(0)
    out = array[indices]
    return indices

def find_nearest4(array,value):
    idx = (np.abs(array-value)).argmin()
    return idx


def find_nearest5(array, value):
    idx_sorted = np.argsort(array)
    sorted_array = np.array(array[idx_sorted])
    idx = np.searchsorted(sorted_array, value, side="left")
    if idx >= len(array):
        idx_nearest = idx_sorted[len(array)-1]
    elif idx == 0:
        idx_nearest = idx_sorted[0]
    else:
        if abs(value - sorted_array[idx-1]) < abs(value - sorted_array[idx]):
            idx_nearest = idx_sorted[idx-1]
        else:
            idx_nearest = idx_sorted[idx]
    return idx_nearest

def find_nearest6(array,value):
    xi = np.argmin(np.abs(np.ceil(array[None].T - value)),axis=0)
    return xi

现在，我将对代码进行计时：注意方法1,2,4,5没有正确给出间隔。方法1,2,4舍入到数组中的最近点（例如> = 1.5-> 2），方法5始终舍入（例如1.45-> 2）。只有方法3和6，当然还有二等分，才能正确给出间隔。

array = np.arange(100000)
val = array[50000]+0.55
print( bisection(array,val))
%timeit bisection(array,val)
print( find_nearest1(array,val))
%timeit find_nearest1(array,val)
print( find_nearest2(array,val))
%timeit find_nearest2(array,val)
print( find_nearest3(array,val))
%timeit find_nearest3(array,val)
print( find_nearest4(array,val))
%timeit find_nearest4(array,val)
print( find_nearest5(array,val))
%timeit find_nearest5(array,val)
print( find_nearest6(array,val))
%timeit find_nearest6(array,val)

(50000, 50000)
100000 loops, best of 3: 4.4 µs per loop
50001
1 loop, best of 3: 180 ms per loop
50001
1000 loops, best of 3: 267 µs per loop
[50000]
1000 loops, best of 3: 390 µs per loop
50001
1000 loops, best of 3: 259 µs per loop
50001
1000 loops, best of 3: 1.21 ms per loop
[50000]
1000 loops, best of 3: 746 µs per loop

对于大型阵列，二等分与次优的180us和最长的1.21ms相比较给出4us（约快100-1000倍）。对于较小的阵列，速度要快2到100倍。

Summary of answer: If one has a sorted array then the bisection code (given below) performs the fastest. ~100-1000 times faster for large arrays, and ~2-100 times faster for small arrays. It does not require numpy either. If you have an unsorted array then if array is large, one should consider first using an O(n logn) sort and then bisection, and if array is small then method 2 seems the fastest.

First you should clarify what you mean by nearest value. Often one wants the interval in an abscissa, e.g. array=[0,0.7,2.1], value=1.95, answer would be idx=1. This is the case that I suspect you need (otherwise the following can be modified very easily with a followup conditional statement once you find the interval). I will note that the optimal way to perform this is with bisection (which I will provide first – note it does not require numpy at all and is faster than using numpy functions because they perform redundant operations). Then I will provide a timing comparison against the others presented here by other users.

Bisection:

def bisection(array,value):
    '''Given an ``array`` , and given a ``value`` , returns an index j such that ``value`` is between array[j]
    and array[j+1]. ``array`` must be monotonic increasing. j=-1 or j=len(array) is returned
    to indicate that ``value`` is out of range below and above respectively.'''
    n = len(array)
    if (value < array[0]):
        return -1
    elif (value > array[n-1]):
        return n
    jl = 0# Initialize lower
    ju = n-1# and upper limits.
    while (ju-jl > 1):# If we are not yet done,
        jm=(ju+jl) >> 1# compute a midpoint with a bitshift
        if (value >= array[jm]):
            jl=jm# and replace either the lower limit
        else:
            ju=jm# or the upper limit, as appropriate.
        # Repeat until the test condition is satisfied.
    if (value == array[0]):# edge cases at bottom
        return 0
    elif (value == array[n-1]):# and top
        return n-1
    else:
        return jl

Now I’ll define the code from the other answers, they each return an index:

import math
import numpy as np

def find_nearest1(array,value):
    idx,val = min(enumerate(array), key=lambda x: abs(x[1]-value))
    return idx

def find_nearest2(array, values):
    indices = np.abs(np.subtract.outer(array, values)).argmin(0)
    return indices

def find_nearest3(array, values):
    values = np.atleast_1d(values)
    indices = np.abs(np.int64(np.subtract.outer(array, values))).argmin(0)
    out = array[indices]
    return indices

def find_nearest4(array,value):
    idx = (np.abs(array-value)).argmin()
    return idx


def find_nearest5(array, value):
    idx_sorted = np.argsort(array)
    sorted_array = np.array(array[idx_sorted])
    idx = np.searchsorted(sorted_array, value, side="left")
    if idx >= len(array):
        idx_nearest = idx_sorted[len(array)-1]
    elif idx == 0:
        idx_nearest = idx_sorted[0]
    else:
        if abs(value - sorted_array[idx-1]) < abs(value - sorted_array[idx]):
            idx_nearest = idx_sorted[idx-1]
        else:
            idx_nearest = idx_sorted[idx]
    return idx_nearest

def find_nearest6(array,value):
    xi = np.argmin(np.abs(np.ceil(array[None].T - value)),axis=0)
    return xi

Now I’ll time the codes: Note methods 1,2,4,5 don’t correctly give the interval. Methods 1,2,4 round to nearest point in array (e.g. >=1.5 -> 2), and method 5 always rounds up (e.g. 1.45 -> 2). Only methods 3, and 6, and of course bisection give the interval properly.

array = np.arange(100000)
val = array[50000]+0.55
print( bisection(array,val))
%timeit bisection(array,val)
print( find_nearest1(array,val))
%timeit find_nearest1(array,val)
print( find_nearest2(array,val))
%timeit find_nearest2(array,val)
print( find_nearest3(array,val))
%timeit find_nearest3(array,val)
print( find_nearest4(array,val))
%timeit find_nearest4(array,val)
print( find_nearest5(array,val))
%timeit find_nearest5(array,val)
print( find_nearest6(array,val))
%timeit find_nearest6(array,val)

(50000, 50000)
100000 loops, best of 3: 4.4 µs per loop
50001
1 loop, best of 3: 180 ms per loop
50001
1000 loops, best of 3: 267 µs per loop
[50000]
1000 loops, best of 3: 390 µs per loop
50001
1000 loops, best of 3: 259 µs per loop
50001
1000 loops, best of 3: 1.21 ms per loop
[50000]
1000 loops, best of 3: 746 µs per loop

For a large array bisection gives 4us compared to next best 180us and longest 1.21ms (~100 – 1000 times faster). For smaller arrays it’s ~2-100 times faster.

回答 4

这是在向量数组中查找最近的向量的扩展。

import numpy as np

def find_nearest_vector(array, value):
  idx = np.array([np.linalg.norm(x+y) for (x,y) in array-value]).argmin()
  return array[idx]

A = np.random.random((10,2))*100
""" A = array([[ 34.19762933,  43.14534123],
   [ 48.79558706,  47.79243283],
   [ 38.42774411,  84.87155478],
   [ 63.64371943,  50.7722317 ],
   [ 73.56362857,  27.87895698],
   [ 96.67790593,  77.76150486],
   [ 68.86202147,  21.38735169],
   [  5.21796467,  59.17051276],
   [ 82.92389467,  99.90387851],
   [  6.76626539,  30.50661753]])"""
pt = [6, 30]  
print find_nearest_vector(A,pt)
# array([  6.76626539,  30.50661753])

Here’s an extension to find the nearest vector in an array of vectors.

import numpy as np

def find_nearest_vector(array, value):
  idx = np.array([np.linalg.norm(x+y) for (x,y) in array-value]).argmin()
  return array[idx]

A = np.random.random((10,2))*100
""" A = array([[ 34.19762933,  43.14534123],
   [ 48.79558706,  47.79243283],
   [ 38.42774411,  84.87155478],
   [ 63.64371943,  50.7722317 ],
   [ 73.56362857,  27.87895698],
   [ 96.67790593,  77.76150486],
   [ 68.86202147,  21.38735169],
   [  5.21796467,  59.17051276],
   [ 82.92389467,  99.90387851],
   [  6.76626539,  30.50661753]])"""
pt = [6, 30]  
print find_nearest_vector(A,pt)
# array([  6.76626539,  30.50661753])

回答 5

如果您不想使用numpy，可以这样做：

def find_nearest(array, value):
    n = [abs(i-value) for i in array]
    idx = n.index(min(n))
    return array[idx]

If you don’t want to use numpy this will do it:

def find_nearest(array, value):
    n = [abs(i-value) for i in array]
    idx = n.index(min(n))
    return array[idx]

回答 6

这是将处理非标量“值”数组的版本：

import numpy as np

def find_nearest(array, values):
    indices = np.abs(np.subtract.outer(array, values)).argmin(0)
    return array[indices]

如果输入是标量，则返回一个数字类型（例如，int，float）的版本：

def find_nearest(array, values):
    values = np.atleast_1d(values)
    indices = np.abs(np.subtract.outer(array, values)).argmin(0)
    out = array[indices]
    return out if len(out) > 1 else out[0]

Here’s a version that will handle a non-scalar “values” array:

import numpy as np

def find_nearest(array, values):
    indices = np.abs(np.subtract.outer(array, values)).argmin(0)
    return array[indices]

Or a version that returns a numeric type (e.g. int, float) if the input is scalar:

def find_nearest(array, values):
    values = np.atleast_1d(values)
    indices = np.abs(np.subtract.outer(array, values)).argmin(0)
    out = array[indices]
    return out if len(out) > 1 else out[0]

回答 7

这是@Ari Onasafari的scipy版本，请回答“ 在向量数组中查找最近的向量 ”

In [1]: from scipy import spatial

In [2]: import numpy as np

In [3]: A = np.random.random((10,2))*100

In [4]: A
Out[4]:
array([[ 68.83402637,  38.07632221],
       [ 76.84704074,  24.9395109 ],
       [ 16.26715795,  98.52763827],
       [ 70.99411985,  67.31740151],
       [ 71.72452181,  24.13516764],
       [ 17.22707611,  20.65425362],
       [ 43.85122458,  21.50624882],
       [ 76.71987125,  44.95031274],
       [ 63.77341073,  78.87417774],
       [  8.45828909,  30.18426696]])

In [5]: pt = [6, 30]  # <-- the point to find

In [6]: A[spatial.KDTree(A).query(pt)[1]] # <-- the nearest point 
Out[6]: array([  8.45828909,  30.18426696])

#how it works!
In [7]: distance,index = spatial.KDTree(A).query(pt)

In [8]: distance # <-- The distances to the nearest neighbors
Out[8]: 2.4651855048258393

In [9]: index # <-- The locations of the neighbors
Out[9]: 9

#then 
In [10]: A[index]
Out[10]: array([  8.45828909,  30.18426696])

Here is a version with scipy for @Ari Onasafari, answer “to find the nearest vector in an array of vectors“

In [1]: from scipy import spatial

In [2]: import numpy as np

In [3]: A = np.random.random((10,2))*100

In [4]: A
Out[4]:
array([[ 68.83402637,  38.07632221],
       [ 76.84704074,  24.9395109 ],
       [ 16.26715795,  98.52763827],
       [ 70.99411985,  67.31740151],
       [ 71.72452181,  24.13516764],
       [ 17.22707611,  20.65425362],
       [ 43.85122458,  21.50624882],
       [ 76.71987125,  44.95031274],
       [ 63.77341073,  78.87417774],
       [  8.45828909,  30.18426696]])

In [5]: pt = [6, 30]  # <-- the point to find

In [6]: A[spatial.KDTree(A).query(pt)[1]] # <-- the nearest point 
Out[6]: array([  8.45828909,  30.18426696])

#how it works!
In [7]: distance,index = spatial.KDTree(A).query(pt)

In [8]: distance # <-- The distances to the nearest neighbors
Out[8]: 2.4651855048258393

In [9]: index # <-- The locations of the neighbors
Out[9]: 9

#then 
In [10]: A[index]
Out[10]: array([  8.45828909,  30.18426696])

回答 8

如果您有很多values要搜索的东西，这是@Dimitri解决方案的快速向量化版本（values可以是多维数组）：

#`values` should be sorted
def get_closest(array, values):
    #make sure array is a numpy array
    array = np.array(array)

    # get insert positions
    idxs = np.searchsorted(array, values, side="left")

    # find indexes where previous index is closer
    prev_idx_is_less = ((idxs == len(array))|(np.fabs(values - array[np.maximum(idxs-1, 0)]) < np.fabs(values - array[np.minimum(idxs, len(array)-1)])))
    idxs[prev_idx_is_less] -= 1

    return array[idxs]

基准测试

比使用for@Demitri解决方案的循环快100倍以上

>>> %timeit ar=get_closest(np.linspace(1, 1000, 100), np.random.randint(0, 1050, (1000, 1000)))
139 ms ± 4.04 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

>>> %timeit ar=[find_nearest(np.linspace(1, 1000, 100), value) for value in np.random.randint(0, 1050, 1000*1000)]
took 21.4 seconds

Here is a fast vectorized version of @Dimitri’s solution if you have many values to search for (values can be multi-dimensional array):

#`values` should be sorted
def get_closest(array, values):
    #make sure array is a numpy array
    array = np.array(array)

    # get insert positions
    idxs = np.searchsorted(array, values, side="left")

    # find indexes where previous index is closer
    prev_idx_is_less = ((idxs == len(array))|(np.fabs(values - array[np.maximum(idxs-1, 0)]) < np.fabs(values - array[np.minimum(idxs, len(array)-1)])))
    idxs[prev_idx_is_less] -= 1

    return array[idxs]

Benchmarks

> 100 times faster than using a for loop with @Demitri’s solution`

>>> %timeit ar=get_closest(np.linspace(1, 1000, 100), np.random.randint(0, 1050, (1000, 1000)))
139 ms ± 4.04 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

>>> %timeit ar=[find_nearest(np.linspace(1, 1000, 100), value) for value in np.random.randint(0, 1050, 1000*1000)]
took 21.4 seconds

回答 9

对于大型数组，@ Demitri给出的（出色）答案远远快于当前标记为最佳的答案。我通过以下两种方式调整了他的确切算法：

无论输入数组是否已排序，下面的函数均有效。
下面的函数返回与最接近的值相对应的输入数组的索引，该值更为通用。

请注意，下面的函数还处理特定的边缘情况，这会导致@Demitri编写的原始函数存在错误。否则，我的算法与他的算法相同。

def find_idx_nearest_val(array, value):
    idx_sorted = np.argsort(array)
    sorted_array = np.array(array[idx_sorted])
    idx = np.searchsorted(sorted_array, value, side="left")
    if idx >= len(array):
        idx_nearest = idx_sorted[len(array)-1]
    elif idx == 0:
        idx_nearest = idx_sorted[0]
    else:
        if abs(value - sorted_array[idx-1]) < abs(value - sorted_array[idx]):
            idx_nearest = idx_sorted[idx-1]
        else:
            idx_nearest = idx_sorted[idx]
    return idx_nearest

For large arrays, the (excellent) answer given by @Demitri is far faster than the answer currently marked as best. I’ve adapted his exact algorithm in the following two ways:

The function below works whether or not the input array is sorted.
The function below returns the index of the input array corresponding to the closest value, which is somewhat more general.

Note that the function below also handles a specific edge case that would lead to a bug in the original function written by @Demitri. Otherwise, my algorithm is identical to his.

def find_idx_nearest_val(array, value):
    idx_sorted = np.argsort(array)
    sorted_array = np.array(array[idx_sorted])
    idx = np.searchsorted(sorted_array, value, side="left")
    if idx >= len(array):
        idx_nearest = idx_sorted[len(array)-1]
    elif idx == 0:
        idx_nearest = idx_sorted[0]
    else:
        if abs(value - sorted_array[idx-1]) < abs(value - sorted_array[idx]):
            idx_nearest = idx_sorted[idx-1]
        else:
            idx_nearest = idx_sorted[idx]
    return idx_nearest

回答 10

这是unutbu答案的矢量化版本：

def find_nearest(array, values):
    array = np.asarray(array)

    # the last dim must be 1 to broadcast in (array - values) below.
    values = np.expand_dims(values, axis=-1) 

    indices = np.abs(array - values).argmin(axis=-1)

    return array[indices]


image = plt.imread('example_3_band_image.jpg')

print(image.shape) # should be (nrows, ncols, 3)

quantiles = np.linspace(0, 255, num=2 ** 2, dtype=np.uint8)

quantiled_image = find_nearest(quantiles, image)

print(quantiled_image.shape) # should be (nrows, ncols, 3)

This is a vectorized version of unutbu’s answer:

def find_nearest(array, values):
    array = np.asarray(array)

    # the last dim must be 1 to broadcast in (array - values) below.
    values = np.expand_dims(values, axis=-1) 

    indices = np.abs(array - values).argmin(axis=-1)

    return array[indices]


image = plt.imread('example_3_band_image.jpg')

print(image.shape) # should be (nrows, ncols, 3)

quantiles = np.linspace(0, 255, num=2 ** 2, dtype=np.uint8)

quantiled_image = find_nearest(quantiles, image)

print(quantiled_image.shape) # should be (nrows, ncols, 3)

回答 11

我认为最Python化的方式是：

 num = 65 # Input number
 array = n.random.random((10))*100 # Given array 
 nearest_idx = n.where(abs(array-num)==abs(array-num).min())[0] # If you want the index of the element of array (array) nearest to the the given number (num)
 nearest_val = array[abs(array-num)==abs(array-num).min()] # If you directly want the element of array (array) nearest to the given number (num)

这是基本代码。您可以根据需要将其用作功能

I think the most pythonic way would be:

 num = 65 # Input number
 array = n.random.random((10))*100 # Given array 
 nearest_idx = n.where(abs(array-num)==abs(array-num).min())[0] # If you want the index of the element of array (array) nearest to the the given number (num)
 nearest_val = array[abs(array-num)==abs(array-num).min()] # If you directly want the element of array (array) nearest to the given number (num)

This is the basic code. You can use it as a function if you want

回答 12

所有答案都有助于收集信息以编写高效的代码。但是，我编写了一个小的Python脚本来针对各种情况进行优化。如果对提供的数组进行了排序，那将是最好的情况。如果搜索指定值最近点的索引，则bisect模块是最省时的。当一个搜索索引对应于一个数组时，numpy searchsorted效率最高。

import numpy as np
import bisect
xarr = np.random.rand(int(1e7))

srt_ind = xarr.argsort()
xar = xarr.copy()[srt_ind]
xlist = xar.tolist()
bisect.bisect_left(xlist, 0.3)

在[63]中：％time bisect.bisect_left（xlist，0.3）CPU时间：用户0 ns，sys：0 ns，总计：0 ns墙壁时间：22.2 µs

np.searchsorted(xar, 0.3, side="left")

在[64]中：％time np.searchsorted（xar，0.3，side =“ left”）CPU时间：用户0 ns，sys：0 ns，总计：0 ns挂墙时间：98.9 µs

randpts = np.random.rand(1000)
np.searchsorted(xar, randpts, side="left")

％time np.searchsorted（xar，randpts，side =“ left”）CPU时间：用户4 ms，sys：0 ns，总计：4 ms挂墙时间：1.2 ms

如果我们遵循乘法规则，那么numpy应该花费〜100毫秒，这意味着〜83X更快。

All the answers are beneficial to gather the information to write efficient code. However, I have written a small Python script to optimize for various cases. It will be the best case if the provided array is sorted. If one searches the index of the nearest point of a specified value, then bisect module is the most time efficient. When one search the indices correspond to an array, the numpy searchsorted is most efficient.

import numpy as np
import bisect
xarr = np.random.rand(int(1e7))

srt_ind = xarr.argsort()
xar = xarr.copy()[srt_ind]
xlist = xar.tolist()
bisect.bisect_left(xlist, 0.3)

In [63]: %time bisect.bisect_left(xlist, 0.3) CPU times: user 0 ns, sys: 0 ns, total: 0 ns Wall time: 22.2 µs

np.searchsorted(xar, 0.3, side="left")

In [64]: %time np.searchsorted(xar, 0.3, side=”left”) CPU times: user 0 ns, sys: 0 ns, total: 0 ns Wall time: 98.9 µs

randpts = np.random.rand(1000)
np.searchsorted(xar, randpts, side="left")

%time np.searchsorted(xar, randpts, side=”left”) CPU times: user 4 ms, sys: 0 ns, total: 4 ms Wall time: 1.2 ms

If we follow the multiplicative rule, then numpy should take ~100 ms which implies ~83X faster.

回答 13

对于2d数组，确定最近元素的i，j位置：

import numpy as np
def find_nearest(a, a0):
    idx = (np.abs(a - a0)).argmin()
    w = a.shape[1]
    i = idx // w
    j = idx - i * w
    return a[i,j], i, j

For 2d array, to determine the i, j position of nearest element:

import numpy as np
def find_nearest(a, a0):
    idx = (np.abs(a - a0)).argmin()
    w = a.shape[1]
    i = idx // w
    j = idx - i * w
    return a[i,j], i, j

回答 14

import numpy as np
def find_nearest(array, value):
    array = np.array(array)
    z=np.abs(array-value)
    y= np.where(z == z.min())
    m=np.array(y)
    x=m[0,0]
    y=m[1,0]
    near_value=array[x,y]

    return near_value

array =np.array([[60,200,30],[3,30,50],[20,1,-50],[20,-500,11]])
print(array)
value = 0
print(find_nearest(array, value))

import numpy as np
def find_nearest(array, value):
    array = np.array(array)
    z=np.abs(array-value)
    y= np.where(z == z.min())
    m=np.array(y)
    x=m[0,0]
    y=m[1,0]
    near_value=array[x,y]

    return near_value

array =np.array([[60,200,30],[3,30,50],[20,1,-50],[20,-500,11]])
print(array)
value = 0
print(find_nearest(array, value))

回答 15

可能对ndarrays：

def find_nearest(X, value):
    return X[np.unravel_index(np.argmin(np.abs(X - value)), X.shape)]

Maybe helpful for ndarrays:

def find_nearest(X, value):
    return X[np.unravel_index(np.argmin(np.abs(X - value)), X.shape)]

知识问答

如何在没有科学符号和给定精度的情况下漂亮地打印numpy.array？

2021年7月27日 Python实用宝典

问题：如何在没有科学符号和给定精度的情况下漂亮地打印numpy.array？

我很好奇，是否有任何打印格式化的方法numpy.arrays，例如，类似于以下方式：

x = 1.23456
print '%.3f' % x

如果我想打印numpy.array浮点数，它会以“科学”格式打印几位小数，即使对于低维数组也很难阅读。但是，numpy.array显然必须将其打印为字符串，即使用%s。有解决方案吗？

I’m curious, whether there is any way to print formatted numpy.arrays, e.g., in a way similar to this:

x = 1.23456
print '%.3f' % x

If I want to print the numpy.array of floats, it prints several decimals, often in ‘scientific’ format, which is rather hard to read even for low-dimensional arrays. However, numpy.array apparently has to be printed as a string, i.e., with %s. Is there a solution for this?

回答 0

您可以set_printoptions用来设置输出的精度：

import numpy as np
x=np.random.random(10)
print(x)
# [ 0.07837821  0.48002108  0.41274116  0.82993414  0.77610352  0.1023732
#   0.51303098  0.4617183   0.33487207  0.71162095]

np.set_printoptions(precision=3)
print(x)
# [ 0.078  0.48   0.413  0.83   0.776  0.102  0.513  0.462  0.335  0.712]

并suppress禁止对小数使用科学计数法：

y=np.array([1.5e-10,1.5,1500])
print(y)
# [  1.500e-10   1.500e+00   1.500e+03]
np.set_printoptions(suppress=True)
print(y)
# [    0.      1.5  1500. ]

有关其他选项，请参见文档中的set_printoptions。

要使用NumPy 1.15.0或更高版本在本地应用打印选项，可以使用numpy.printoptions上下文管理器。例如，在with-suite precision=3和suppress=True中设置：

x = np.random.random(10)
with np.printoptions(precision=3, suppress=True):
    print(x)
    # [ 0.073  0.461  0.689  0.754  0.624  0.901  0.049  0.582  0.557  0.348]

但是在with-suite打印选项之外，将恢复为默认设置：

print(x)    
# [ 0.07334334  0.46132615  0.68935231  0.75379645  0.62424021  0.90115836
#   0.04879837  0.58207504  0.55694118  0.34768638]

如果您使用的是NumPy的早期版本，则可以自己创建上下文管理器。例如，

import numpy as np
import contextlib

@contextlib.contextmanager
def printoptions(*args, **kwargs):
    original = np.get_printoptions()
    np.set_printoptions(*args, **kwargs)
    try:
        yield
    finally: 
        np.set_printoptions(**original)

x = np.random.random(10)
with printoptions(precision=3, suppress=True):
    print(x)
    # [ 0.073  0.461  0.689  0.754  0.624  0.901  0.049  0.582  0.557  0.348]

为防止浮点数结尾处的零被剥离：

np.set_printoptions现在有一个formatter参数，可让您为每种类型指定格式功能。

np.set_printoptions(formatter={'float': '{: 0.3f}'.format})
print(x)

哪个打印

[ 0.078  0.480  0.413  0.830  0.776  0.102  0.513  0.462  0.335  0.712]

代替

[ 0.078  0.48   0.413  0.83   0.776  0.102  0.513  0.462  0.335  0.712]

You can use set_printoptions to set the precision of the output:

import numpy as np
x=np.random.random(10)
print(x)
# [ 0.07837821  0.48002108  0.41274116  0.82993414  0.77610352  0.1023732
#   0.51303098  0.4617183   0.33487207  0.71162095]

np.set_printoptions(precision=3)
print(x)
# [ 0.078  0.48   0.413  0.83   0.776  0.102  0.513  0.462  0.335  0.712]

And suppress suppresses the use of scientific notation for small numbers:

y=np.array([1.5e-10,1.5,1500])
print(y)
# [  1.500e-10   1.500e+00   1.500e+03]
np.set_printoptions(suppress=True)
print(y)
# [    0.      1.5  1500. ]

See the docs for set_printoptions for other options.

To apply print options locally, using NumPy 1.15.0 or later, you could use the numpy.printoptions context manager. For example, inside the with-suite precision=3 and suppress=True are set:

x = np.random.random(10)
with np.printoptions(precision=3, suppress=True):
    print(x)
    # [ 0.073  0.461  0.689  0.754  0.624  0.901  0.049  0.582  0.557  0.348]

But outside the with-suite the print options are back to default settings:

print(x)    
# [ 0.07334334  0.46132615  0.68935231  0.75379645  0.62424021  0.90115836
#   0.04879837  0.58207504  0.55694118  0.34768638]

If you are using an earlier version of NumPy, you can create the context manager yourself. For example,

import numpy as np
import contextlib

@contextlib.contextmanager
def printoptions(*args, **kwargs):
    original = np.get_printoptions()
    np.set_printoptions(*args, **kwargs)
    try:
        yield
    finally: 
        np.set_printoptions(**original)

x = np.random.random(10)
with printoptions(precision=3, suppress=True):
    print(x)
    # [ 0.073  0.461  0.689  0.754  0.624  0.901  0.049  0.582  0.557  0.348]

To prevent zeros from being stripped from the end of floats:

np.set_printoptions now has a formatter parameter which allows you to specify a format function for each type.

np.set_printoptions(formatter={'float': '{: 0.3f}'.format})
print(x)

which prints

[ 0.078  0.480  0.413  0.830  0.776  0.102  0.513  0.462  0.335  0.712]

instead of

[ 0.078  0.48   0.413  0.83   0.776  0.102  0.513  0.462  0.335  0.712]

回答 1

您可以np.set_printoptions从np.array_str命令中获得功能的子集，该命令仅适用于单个打印语句。

http://docs.scipy.org/doc/numpy/reference/generated/numpy.array_str.html

例如：

In [27]: x = np.array([[1.1, 0.9, 1e-6]]*3)

In [28]: print x
[[  1.10000000e+00   9.00000000e-01   1.00000000e-06]
 [  1.10000000e+00   9.00000000e-01   1.00000000e-06]
 [  1.10000000e+00   9.00000000e-01   1.00000000e-06]]

In [29]: print np.array_str(x, precision=2)
[[  1.10e+00   9.00e-01   1.00e-06]
 [  1.10e+00   9.00e-01   1.00e-06]
 [  1.10e+00   9.00e-01   1.00e-06]]

In [30]: print np.array_str(x, precision=2, suppress_small=True)
[[ 1.1  0.9  0. ]
 [ 1.1  0.9  0. ]
 [ 1.1  0.9  0. ]]

You can get a subset of the np.set_printoptions functionality from the np.array_str command, which applies only to a single print statement.

http://docs.scipy.org/doc/numpy/reference/generated/numpy.array_str.html

For example:

In [27]: x = np.array([[1.1, 0.9, 1e-6]]*3)

In [28]: print x
[[  1.10000000e+00   9.00000000e-01   1.00000000e-06]
 [  1.10000000e+00   9.00000000e-01   1.00000000e-06]
 [  1.10000000e+00   9.00000000e-01   1.00000000e-06]]

In [29]: print np.array_str(x, precision=2)
[[  1.10e+00   9.00e-01   1.00e-06]
 [  1.10e+00   9.00e-01   1.00e-06]
 [  1.10e+00   9.00e-01   1.00e-06]]

In [30]: print np.array_str(x, precision=2, suppress_small=True)
[[ 1.1  0.9  0. ]
 [ 1.1  0.9  0. ]
 [ 1.1  0.9  0. ]]

回答 2

Unutbu给出了一个非常完整的答案（他们也从我这里得到了+1），但这是一种高科技的替代方法：

>>> x=np.random.randn(5)
>>> x
array([ 0.25276524,  2.28334499, -1.88221637,  0.69949927,  1.0285625 ])
>>> ['{:.2f}'.format(i) for i in x]
['0.25', '2.28', '-1.88', '0.70', '1.03']

作为一项功能（使用format()语法进行格式化）：

def ndprint(a, format_string ='{0:.2f}'):
    print [format_string.format(v,i) for i,v in enumerate(a)]

用法：

>>> ndprint(x)
['0.25', '2.28', '-1.88', '0.70', '1.03']

>>> ndprint(x, '{:10.4e}')
['2.5277e-01', '2.2833e+00', '-1.8822e+00', '6.9950e-01', '1.0286e+00']

>>> ndprint(x, '{:.8g}')
['0.25276524', '2.283345', '-1.8822164', '0.69949927', '1.0285625']

可以使用以下格式的字符串访问数组的索引：

>>> ndprint(x, 'Element[{1:d}]={0:.2f}')
['Element[0]=0.25', 'Element[1]=2.28', 'Element[2]=-1.88', 'Element[3]=0.70', 'Element[4]=1.03']

Unutbu gave a really complete answer (they got a +1 from me too), but here is a lo-tech alternative:

>>> x=np.random.randn(5)
>>> x
array([ 0.25276524,  2.28334499, -1.88221637,  0.69949927,  1.0285625 ])
>>> ['{:.2f}'.format(i) for i in x]
['0.25', '2.28', '-1.88', '0.70', '1.03']

As a function (using the format() syntax for formatting):

def ndprint(a, format_string ='{0:.2f}'):
    print [format_string.format(v,i) for i,v in enumerate(a)]

Usage:

>>> ndprint(x)
['0.25', '2.28', '-1.88', '0.70', '1.03']

>>> ndprint(x, '{:10.4e}')
['2.5277e-01', '2.2833e+00', '-1.8822e+00', '6.9950e-01', '1.0286e+00']

>>> ndprint(x, '{:.8g}')
['0.25276524', '2.283345', '-1.8822164', '0.69949927', '1.0285625']

The index of the array is accessible in the format string:

>>> ndprint(x, 'Element[{1:d}]={0:.2f}')
['Element[0]=0.25', 'Element[1]=2.28', 'Element[2]=-1.88', 'Element[3]=0.70', 'Element[4]=1.03']

回答 3

FYI Numpy 1.15（发布日期待定）将包括一个上下文管理器，用于在本地设置打印选项。这意味着以下内容将与接受的答案（由unutbu和Neil G撰写）中的相应示例相同，而无需编写您自己的上下文管理器。例如，使用他们的示例：

x = np.random.random(10)
with np.printoptions(precision=3, suppress=True):
    print(x)
    # [ 0.073  0.461  0.689  0.754  0.624  0.901  0.049  0.582  0.557  0.348]

FYI Numpy 1.15 (release date pending) will include a context manager for setting print options locally. This means that the following will work the same as the corresponding example in the accepted answer (by unutbu and Neil G) without having to write your own context manager. E.g., using their example:

x = np.random.random(10)
with np.printoptions(precision=3, suppress=True):
    print(x)
    # [ 0.073  0.461  0.689  0.754  0.624  0.901  0.049  0.582  0.557  0.348]

回答 4

在denis答案中隐藏了使它很容易以字符串形式获得结果的gem（在当今的numpy版本中）： np.array2string

>>> import numpy as np
>>> x=np.random.random(10)
>>> np.array2string(x, formatter={'float_kind':'{0:.3f}'.format})
'[0.599 0.847 0.513 0.155 0.844 0.753 0.920 0.797 0.427 0.420]'

The gem that makes it all too easy to obtain the result as a string (in today’s numpy versions) is hidden in denis answer: np.array2string

>>> import numpy as np
>>> x=np.random.random(10)
>>> np.array2string(x, formatter={'float_kind':'{0:.3f}'.format})
'[0.599 0.847 0.513 0.155 0.844 0.753 0.920 0.797 0.427 0.420]'

回答 5

几年后，下面是另一个。但是对于日常使用，我只是

np.set_printoptions( threshold=20, edgeitems=10, linewidth=140,
    formatter = dict( float = lambda x: "%.3g" % x ))  # float arrays %.3g

''' printf( "... %.3g ... %.1f  ...", arg, arg ... ) for numpy arrays too

Example:
    printf( """ x: %.3g   A: %.1f   s: %s   B: %s """,
                   x,        A,        "str",  B )

If `x` and `A` are numbers, this is like `"format" % (x, A, "str", B)` in python.
If they're numpy arrays, each element is printed in its own format:
    `x`: e.g. [ 1.23 1.23e-6 ... ]  3 digits
    `A`: [ [ 1 digit after the decimal point ... ] ... ]
with the current `np.set_printoptions()`. For example, with
    np.set_printoptions( threshold=100, edgeitems=3, suppress=True )
only the edges of big `x` and `A` are printed.
`B` is printed as `str(B)`, for any `B` -- a number, a list, a numpy object ...

`printf()` tries to handle too few or too many arguments sensibly,
but this is iffy and subject to change.

How it works:
numpy has a function `np.array2string( A, "%.3g" )` (simplifying a bit).
`printf()` splits the format string, and for format / arg pairs
    format: % d e f g
    arg: try `np.asanyarray()`
-->  %s  np.array2string( arg, format )
Other formats and non-ndarray args are left alone, formatted as usual.

Notes:

`printf( ... end= file= )` are passed on to the python `print()` function.

Only formats `% [optional width . precision] d e f g` are implemented,
not `%(varname)format` .

%d truncates floats, e.g. 0.9 and -0.9 to 0; %.0f rounds, 0.9 to 1 .
%g is the same as %.6g, 6 digits.
%% is a single "%" character.

The function `sprintf()` returns a long string. For example,
    title = sprintf( "%s  m %g  n %g  X %.3g",
                    __file__, m, n, X )
    print( title )
    ...
    pl.title( title )

Module globals:
_fmt = "%.3g"  # default for extra args
_squeeze = np.squeeze  # (n,1) (1,n) -> (n,) print in 1 line not n

See also:
http://docs.scipy.org/doc/numpy/reference/generated/numpy.set_printoptions.html
http://docs.python.org/2.7/library/stdtypes.html#string-formatting

'''
# http://stackoverflow.com/questions/2891790/pretty-printing-of-numpy-array


#...............................................................................
from __future__ import division, print_function
import re
import numpy as np

__version__ = "2014-02-03 feb denis"

_splitformat = re.compile( r'''(
    %
    (?<! %% )  # not %%
    -? [ \d . ]*  # optional width.precision
    \w
    )''', re.X )
    # ... %3.0f  ... %g  ... %-10s ...
    # -> ['...' '%3.0f' '...' '%g' '...' '%-10s' '...']
    # odd len, first or last may be ""

_fmt = "%.3g"  # default for extra args
_squeeze = np.squeeze  # (n,1) (1,n) -> (n,) print in 1 line not n

#...............................................................................
def printf( format, *args, **kwargs ):
    print( sprintf( format, *args ), **kwargs )  # end= file=

printf.__doc__ = __doc__


def sprintf( format, *args ):
    """ sprintf( "text %.3g text %4.1f ... %s ... ", numpy arrays or ... )
        %[defg] array -> np.array2string( formatter= )
    """
    args = list(args)
    if not isinstance( format, basestring ):
        args = [format] + args
        format = ""

    tf = _splitformat.split( format )  # [ text %e text %f ... ]
    nfmt = len(tf) // 2
    nargs = len(args)
    if nargs < nfmt:
        args += (nfmt - nargs) * ["?arg?"]
    elif nargs > nfmt:
        tf += (nargs - nfmt) * [_fmt, " "]  # default _fmt

    for j, arg in enumerate( args ):
        fmt = tf[ 2*j + 1 ]
        if arg is None \
        or isinstance( arg, basestring ) \
        or (hasattr( arg, "__iter__" ) and len(arg) == 0):
            tf[ 2*j + 1 ] = "%s"  # %f -> %s, not error
            continue
        args[j], isarray = _tonumpyarray(arg)
        if isarray  and fmt[-1] in "defgEFG":
            tf[ 2*j + 1 ] = "%s"
            fmtfunc = (lambda x: fmt % x)
            formatter = dict( float_kind=fmtfunc, int=fmtfunc )
            args[j] = np.array2string( args[j], formatter=formatter )
    try:
        return "".join(tf) % tuple(args)
    except TypeError:  # shouldn't happen
        print( "error: tf %s  types %s" % (tf, map( type, args )))
        raise


def _tonumpyarray( a ):
    """ a, isarray = _tonumpyarray( a )
        ->  scalar, False
            np.asanyarray(a), float or int
            a, False
    """
    a = getattr( a, "value", a )  # cvxpy
    if np.isscalar(a):
        return a, False
    if hasattr( a, "__iter__" )  and len(a) == 0:
        return a, False
    try:
        # map .value ?
        a = np.asanyarray( a )
    except ValueError:
        return a, False
    if hasattr( a, "dtype" )  and a.dtype.kind in "fi":  # complex ?
        if callable( _squeeze ):
            a = _squeeze( a )  # np.squeeze
        return a, True
    else:
        return a, False


#...............................................................................
if __name__ == "__main__":
    import sys

    n = 5
    seed = 0
        # run this.py n= ...  in sh or ipython
    for arg in sys.argv[1:]:
        exec( arg )
    np.set_printoptions( 1, threshold=4, edgeitems=2, linewidth=80, suppress=True )
    np.random.seed(seed)

    A = np.random.exponential( size=(n,n) ) ** 10
    x = A[0]

    printf( "x: %.3g  \nA: %.1f  \ns: %s  \nB: %s ",
                x,         A,         "str",   A )
    printf( "x %%d: %d", x )
    printf( "x %%.0f: %.0f", x )
    printf( "x %%.1e: %.1e", x )
    printf( "x %%g: %g", x )
    printf( "x %%s uses np printoptions: %s", x )

    printf( "x with default _fmt: ", x )
    printf( "no args" )
    printf( "too few args: %g %g", x )
    printf( x )
    printf( x, x )
    printf( None )
    printf( "[]:", [] )
    printf( "[3]:", [3] )
    printf( np.array( [] ))
    printf( [[]] )  # squeeze

Years later, another one is below. But for everyday use I just

np.set_printoptions( threshold=20, edgeitems=10, linewidth=140,
    formatter = dict( float = lambda x: "%.3g" % x ))  # float arrays %.3g

''' printf( "... %.3g ... %.1f  ...", arg, arg ... ) for numpy arrays too

Example:
    printf( """ x: %.3g   A: %.1f   s: %s   B: %s """,
                   x,        A,        "str",  B )

If `x` and `A` are numbers, this is like `"format" % (x, A, "str", B)` in python.
If they're numpy arrays, each element is printed in its own format:
    `x`: e.g. [ 1.23 1.23e-6 ... ]  3 digits
    `A`: [ [ 1 digit after the decimal point ... ] ... ]
with the current `np.set_printoptions()`. For example, with
    np.set_printoptions( threshold=100, edgeitems=3, suppress=True )
only the edges of big `x` and `A` are printed.
`B` is printed as `str(B)`, for any `B` -- a number, a list, a numpy object ...

`printf()` tries to handle too few or too many arguments sensibly,
but this is iffy and subject to change.

How it works:
numpy has a function `np.array2string( A, "%.3g" )` (simplifying a bit).
`printf()` splits the format string, and for format / arg pairs
    format: % d e f g
    arg: try `np.asanyarray()`
-->  %s  np.array2string( arg, format )
Other formats and non-ndarray args are left alone, formatted as usual.

Notes:

`printf( ... end= file= )` are passed on to the python `print()` function.

Only formats `% [optional width . precision] d e f g` are implemented,
not `%(varname)format` .

%d truncates floats, e.g. 0.9 and -0.9 to 0; %.0f rounds, 0.9 to 1 .
%g is the same as %.6g, 6 digits.
%% is a single "%" character.

The function `sprintf()` returns a long string. For example,
    title = sprintf( "%s  m %g  n %g  X %.3g",
                    __file__, m, n, X )
    print( title )
    ...
    pl.title( title )

Module globals:
_fmt = "%.3g"  # default for extra args
_squeeze = np.squeeze  # (n,1) (1,n) -> (n,) print in 1 line not n

See also:
http://docs.scipy.org/doc/numpy/reference/generated/numpy.set_printoptions.html
http://docs.python.org/2.7/library/stdtypes.html#string-formatting

'''
# http://stackoverflow.com/questions/2891790/pretty-printing-of-numpy-array


#...............................................................................
from __future__ import division, print_function
import re
import numpy as np

__version__ = "2014-02-03 feb denis"

_splitformat = re.compile( r'''(
    %
    (?<! %% )  # not %%
    -? [ \d . ]*  # optional width.precision
    \w
    )''', re.X )
    # ... %3.0f  ... %g  ... %-10s ...
    # -> ['...' '%3.0f' '...' '%g' '...' '%-10s' '...']
    # odd len, first or last may be ""

_fmt = "%.3g"  # default for extra args
_squeeze = np.squeeze  # (n,1) (1,n) -> (n,) print in 1 line not n

#...............................................................................
def printf( format, *args, **kwargs ):
    print( sprintf( format, *args ), **kwargs )  # end= file=

printf.__doc__ = __doc__


def sprintf( format, *args ):
    """ sprintf( "text %.3g text %4.1f ... %s ... ", numpy arrays or ... )
        %[defg] array -> np.array2string( formatter= )
    """
    args = list(args)
    if not isinstance( format, basestring ):
        args = [format] + args
        format = ""

    tf = _splitformat.split( format )  # [ text %e text %f ... ]
    nfmt = len(tf) // 2
    nargs = len(args)
    if nargs < nfmt:
        args += (nfmt - nargs) * ["?arg?"]
    elif nargs > nfmt:
        tf += (nargs - nfmt) * [_fmt, " "]  # default _fmt

    for j, arg in enumerate( args ):
        fmt = tf[ 2*j + 1 ]
        if arg is None \
        or isinstance( arg, basestring ) \
        or (hasattr( arg, "__iter__" ) and len(arg) == 0):
            tf[ 2*j + 1 ] = "%s"  # %f -> %s, not error
            continue
        args[j], isarray = _tonumpyarray(arg)
        if isarray  and fmt[-1] in "defgEFG":
            tf[ 2*j + 1 ] = "%s"
            fmtfunc = (lambda x: fmt % x)
            formatter = dict( float_kind=fmtfunc, int=fmtfunc )
            args[j] = np.array2string( args[j], formatter=formatter )
    try:
        return "".join(tf) % tuple(args)
    except TypeError:  # shouldn't happen
        print( "error: tf %s  types %s" % (tf, map( type, args )))
        raise


def _tonumpyarray( a ):
    """ a, isarray = _tonumpyarray( a )
        ->  scalar, False
            np.asanyarray(a), float or int
            a, False
    """
    a = getattr( a, "value", a )  # cvxpy
    if np.isscalar(a):
        return a, False
    if hasattr( a, "__iter__" )  and len(a) == 0:
        return a, False
    try:
        # map .value ?
        a = np.asanyarray( a )
    except ValueError:
        return a, False
    if hasattr( a, "dtype" )  and a.dtype.kind in "fi":  # complex ?
        if callable( _squeeze ):
            a = _squeeze( a )  # np.squeeze
        return a, True
    else:
        return a, False


#...............................................................................
if __name__ == "__main__":
    import sys

    n = 5
    seed = 0
        # run this.py n= ...  in sh or ipython
    for arg in sys.argv[1:]:
        exec( arg )
    np.set_printoptions( 1, threshold=4, edgeitems=2, linewidth=80, suppress=True )
    np.random.seed(seed)

    A = np.random.exponential( size=(n,n) ) ** 10
    x = A[0]

    printf( "x: %.3g  \nA: %.1f  \ns: %s  \nB: %s ",
                x,         A,         "str",   A )
    printf( "x %%d: %d", x )
    printf( "x %%.0f: %.0f", x )
    printf( "x %%.1e: %.1e", x )
    printf( "x %%g: %g", x )
    printf( "x %%s uses np printoptions: %s", x )

    printf( "x with default _fmt: ", x )
    printf( "no args" )
    printf( "too few args: %g %g", x )
    printf( x )
    printf( x, x )
    printf( None )
    printf( "[]:", [] )
    printf( "[3]:", [3] )
    printf( np.array( [] ))
    printf( [[]] )  # squeeze

回答 6

这是我所使用的，并且非常简单：

print(np.vectorize("%.2f".__mod__)(sparse))

And here is what I use, and it’s pretty uncomplicated:

print(np.vectorize("%.2f".__mod__)(sparse))

回答 7

惊讶的是没有看到around提到的方法-意味着不会弄乱打印选项。

import numpy as np

x = np.random.random([5,5])
print(np.around(x,decimals=3))

Output:
[[0.475 0.239 0.183 0.991 0.171]
 [0.231 0.188 0.235 0.335 0.049]
 [0.87  0.212 0.219 0.9   0.3  ]
 [0.628 0.791 0.409 0.5   0.319]
 [0.614 0.84  0.812 0.4   0.307]]

Was surprised to not see around method mentioned – means no messing with print options.

import numpy as np

x = np.random.random([5,5])
print(np.around(x,decimals=3))

Output:
[[0.475 0.239 0.183 0.991 0.171]
 [0.231 0.188 0.235 0.335 0.049]
 [0.87  0.212 0.219 0.9   0.3  ]
 [0.628 0.791 0.409 0.5   0.319]
 [0.614 0.84  0.812 0.4   0.307]]

回答 8

我经常希望不同的列具有不同的格式。这是我通过将NumPy数组（的片段）转换为元组来使用格式多样的简单2D数组的方式：

import numpy as np
dat = np.random.random((10,11))*100  # Array of random values between 0 and 100
print(dat)                           # Lines get truncated and are hard to read
for i in range(10):
    print((4*"%6.2f"+7*"%9.4f") % tuple(dat[i,:]))

I often want different columns to have different formats. Here is how I print a simple 2D array using some variety in the formatting by converting (slices of) my NumPy array to a tuple:

import numpy as np
dat = np.random.random((10,11))*100  # Array of random values between 0 and 100
print(dat)                           # Lines get truncated and are hard to read
for i in range(10):
    print((4*"%6.2f"+7*"%9.4f") % tuple(dat[i,:]))

回答 9

numpy.char.mod根据您应用程序的详细信息，它可能也很有用，例如：numpy.char.mod('Value=%4.2f', numpy.arange(5, 10, 0.1))将返回一个包含元素“ Value = 5.00”，“ Value = 5.10”等的字符串数组（作为一个人为的示例）。

numpy.char.mod may also be useful, depending on the details of your application e.g.:numpy.char.mod('Value=%4.2f', numpy.arange(5, 10, 0.1)) will return a string array with elements “Value=5.00”, “Value=5.10” etc. (as a somewhat contrived example).

回答 10

numpy数组具有round(precision)返回一个新的numpy数组的方法，该数组具有相应的舍入元素。

import numpy as np

x = np.random.random([5,5])
print(x.round(3))

The numpy arrays have the method round(precision) which return a new numpy array with elements rounded accordingly.

import numpy as np

x = np.random.random([5,5])
print(x.round(3))

回答 11

我发现使用循环显示列表或数组时，通常的浮点格式{：9.5f}可以正常工作-抑制小数值电子注释。但是，当格式化程序在单个print语句中有多个项目时，该格式有时无法抑制其电子注释。例如：

import numpy as np
np.set_printoptions(suppress=True)
a3 = 4E-3
a4 = 4E-4
a5 = 4E-5
a6 = 4E-6
a7 = 4E-7
a8 = 4E-8
#--first, display separate numbers-----------
print('Case 3:  a3, a4, a5:             {:9.5f}{:9.5f}{:9.5f}'.format(a3,a4,a5))
print('Case 4:  a3, a4, a5, a6:         {:9.5f}{:9.5f}{:9.5f}{:9.5}'.format(a3,a4,a5,a6))
print('Case 5:  a3, a4, a5, a6, a7:     {:9.5f}{:9.5f}{:9.5f}{:9.5}{:9.5f}'.format(a3,a4,a5,a6,a7))
print('Case 6:  a3, a4, a5, a6, a7, a8: {:9.5f}{:9.5f}{:9.5f}{:9.5f}{:9.5}{:9.5f}'.format(a3,a4,a5,a6,a7,a8))
#---second, display a list using a loop----------
myList = [a3,a4,a5,a6,a7,a8]
print('List 6:  a3, a4, a5, a6, a7, a8: ', end='')
for x in myList: 
    print('{:9.5f}'.format(x), end='')
print()
#---third, display a numpy array using a loop------------
myArray = np.array(myList)
print('Array 6: a3, a4, a5, a6, a7, a8: ', end='')
for x in myArray:
    print('{:9.5f}'.format(x), end='')
print()

我的结果显示了情况4、5和6中的错误：

Case 3:  a3, a4, a5:               0.00400  0.00040  0.00004
Case 4:  a3, a4, a5, a6:           0.00400  0.00040  0.00004    4e-06
Case 5:  a3, a4, a5, a6, a7:       0.00400  0.00040  0.00004    4e-06  0.00000
Case 6:  a3, a4, a5, a6, a7, a8:   0.00400  0.00040  0.00004  0.00000    4e-07  0.00000
List 6:  a3, a4, a5, a6, a7, a8:   0.00400  0.00040  0.00004  0.00000  0.00000  0.00000
Array 6: a3, a4, a5, a6, a7, a8:   0.00400  0.00040  0.00004  0.00000  0.00000  0.00000

我对此没有任何解释，因此我总是使用循环来浮动多个值的输出。

I find that the usual float format {:9.5f} works properly — suppressing small-value e-notations — when displaying a list or an array using a loop. But that format sometimes fails to suppress its e-notation when a formatter has several items in a single print statement. For example:

import numpy as np
np.set_printoptions(suppress=True)
a3 = 4E-3
a4 = 4E-4
a5 = 4E-5
a6 = 4E-6
a7 = 4E-7
a8 = 4E-8
#--first, display separate numbers-----------
print('Case 3:  a3, a4, a5:             {:9.5f}{:9.5f}{:9.5f}'.format(a3,a4,a5))
print('Case 4:  a3, a4, a5, a6:         {:9.5f}{:9.5f}{:9.5f}{:9.5}'.format(a3,a4,a5,a6))
print('Case 5:  a3, a4, a5, a6, a7:     {:9.5f}{:9.5f}{:9.5f}{:9.5}{:9.5f}'.format(a3,a4,a5,a6,a7))
print('Case 6:  a3, a4, a5, a6, a7, a8: {:9.5f}{:9.5f}{:9.5f}{:9.5f}{:9.5}{:9.5f}'.format(a3,a4,a5,a6,a7,a8))
#---second, display a list using a loop----------
myList = [a3,a4,a5,a6,a7,a8]
print('List 6:  a3, a4, a5, a6, a7, a8: ', end='')
for x in myList: 
    print('{:9.5f}'.format(x), end='')
print()
#---third, display a numpy array using a loop------------
myArray = np.array(myList)
print('Array 6: a3, a4, a5, a6, a7, a8: ', end='')
for x in myArray:
    print('{:9.5f}'.format(x), end='')
print()

My results show the bug in cases 4, 5, and 6:

Case 3:  a3, a4, a5:               0.00400  0.00040  0.00004
Case 4:  a3, a4, a5, a6:           0.00400  0.00040  0.00004    4e-06
Case 5:  a3, a4, a5, a6, a7:       0.00400  0.00040  0.00004    4e-06  0.00000
Case 6:  a3, a4, a5, a6, a7, a8:   0.00400  0.00040  0.00004  0.00000    4e-07  0.00000
List 6:  a3, a4, a5, a6, a7, a8:   0.00400  0.00040  0.00004  0.00000  0.00000  0.00000
Array 6: a3, a4, a5, a6, a7, a8:   0.00400  0.00040  0.00004  0.00000  0.00000  0.00000

I have no explanation for this, and therefore I always use a loop for floating output of multiple values.

回答 12

我用

def np_print(array,fmt="10.5f"):
    print (array.size*("{:"+fmt+"}")).format(*array)

修改多维数组并不难。

I use

def np_print(array,fmt="10.5f"):
    print (array.size*("{:"+fmt+"}")).format(*array)

It’s not difficult to modify it for multi-dimensional arrays.

回答 13

另一个选择是使用decimal模块：

import numpy as np
from decimal import *

arr = np.array([  56.83,  385.3 ,    6.65,  126.63,   85.76,  192.72,  112.81, 10.55])
arr2 = [str(Decimal(i).quantize(Decimal('.01'))) for i in arr]

# ['56.83', '385.30', '6.65', '126.63', '85.76', '192.72', '112.81', '10.55']

Yet another option is to use the decimal module:

import numpy as np
from decimal import *

arr = np.array([  56.83,  385.3 ,    6.65,  126.63,   85.76,  192.72,  112.81, 10.55])
arr2 = [str(Decimal(i).quantize(Decimal('.01'))) for i in arr]

# ['56.83', '385.30', '6.65', '126.63', '85.76', '192.72', '112.81', '10.55']

知识问答

如何在不截断的情况下打印完整的NumPy数组？

2021年7月25日 Python实用宝典

问题：如何在不截断的情况下打印完整的NumPy数组？

当我打印一个numpy数组时，我得到一个截断的表示形式，但是我想要完整的数组。

有什么办法吗？

例子：

>>> numpy.arange(10000)
array([   0,    1,    2, ..., 9997, 9998, 9999])

>>> numpy.arange(10000).reshape(250,40)
array([[   0,    1,    2, ...,   37,   38,   39],
       [  40,   41,   42, ...,   77,   78,   79],
       [  80,   81,   82, ...,  117,  118,  119],
       ..., 
       [9880, 9881, 9882, ..., 9917, 9918, 9919],
       [9920, 9921, 9922, ..., 9957, 9958, 9959],
       [9960, 9961, 9962, ..., 9997, 9998, 9999]])

When I print a numpy array, I get a truncated representation, but I want the full array.

Is there any way to do this?

Examples:

>>> numpy.arange(10000)
array([   0,    1,    2, ..., 9997, 9998, 9999])

>>> numpy.arange(10000).reshape(250,40)
array([[   0,    1,    2, ...,   37,   38,   39],
       [  40,   41,   42, ...,   77,   78,   79],
       [  80,   81,   82, ...,  117,  118,  119],
       ..., 
       [9880, 9881, 9882, ..., 9917, 9918, 9919],
       [9920, 9921, 9922, ..., 9957, 9958, 9959],
       [9960, 9961, 9962, ..., 9997, 9998, 9999]])

回答 0

用途numpy.set_printoptions：

import sys
import numpy
numpy.set_printoptions(threshold=sys.maxsize)

Use numpy.set_printoptions:

import sys
import numpy
numpy.set_printoptions(threshold=sys.maxsize)

回答 1

import numpy as np
np.set_printoptions(threshold=np.inf)

我建议使用，np.inf而不是np.nan别人建议的。它们都为您的目的而工作，但是通过将阈值设置为“无穷大”，对于每个阅读您的代码的人来说都是显而易见的。对我来说，达到“没有数字”的门槛似乎有点模糊。

import numpy as np
np.set_printoptions(threshold=np.inf)

I suggest using np.inf instead of np.nan which is suggested by others. They both work for your purpose, but by setting the threshold to “infinity” it is obvious to everybody reading your code what you mean. Having a threshold of “not a number” seems a little vague to me.

回答 2

先前的答案是正确的，但是作为较弱的选择，您可以转换为列表：

>>> numpy.arange(100).reshape(25,4).tolist()

[[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11], [12, 13, 14, 15], [16, 17, 18, 19], [20, 21,
22, 23], [24, 25, 26, 27], [28, 29, 30, 31], [32, 33, 34, 35], [36, 37, 38, 39], [40, 41,
42, 43], [44, 45, 46, 47], [48, 49, 50, 51], [52, 53, 54, 55], [56, 57, 58, 59], [60, 61,
62, 63], [64, 65, 66, 67], [68, 69, 70, 71], [72, 73, 74, 75], [76, 77, 78, 79], [80, 81,
82, 83], [84, 85, 86, 87], [88, 89, 90, 91], [92, 93, 94, 95], [96, 97, 98, 99]]

The previous answers are the correct ones, but as a weaker alternative you can transform into a list:

>>> numpy.arange(100).reshape(25,4).tolist()

[[0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11], [12, 13, 14, 15], [16, 17, 18, 19], [20, 21,
22, 23], [24, 25, 26, 27], [28, 29, 30, 31], [32, 33, 34, 35], [36, 37, 38, 39], [40, 41,
42, 43], [44, 45, 46, 47], [48, 49, 50, 51], [52, 53, 54, 55], [56, 57, 58, 59], [60, 61,
62, 63], [64, 65, 66, 67], [68, 69, 70, 71], [72, 73, 74, 75], [76, 77, 78, 79], [80, 81,
82, 83], [84, 85, 86, 87], [88, 89, 90, 91], [92, 93, 94, 95], [96, 97, 98, 99]]

回答 3

NumPy 1.15或更高版本

如果您使用NumPy 1.15（2018年7月23日发行）或更高版本，则可以使用printoptions上下文管理器：

with numpy.printoptions(threshold=numpy.inf):
    print(arr)

（当然，如果您导入的方式是，请替换numpy为）npnumpy

使用上下文管理器（with-block）可确保在上下文管理器完成后，打印选项将恢复为块启动之前的状态。它确保设置是临时的，并且仅应用于块内的代码。

有关上下文管理器及其支持的其他参数的详细信息，请参见numpy.printoptions文档。

NumPy 1.15 or newer

If you use NumPy 1.15 (released 2018-07-23) or newer, you can use the printoptions context manager:

with numpy.printoptions(threshold=numpy.inf):
    print(arr)

(of course, replace numpy by np if that’s how you imported numpy)

The use of a context manager (the with-block) ensures that after the context manager is finished, the print options will revert to whatever they were before the block started. It ensures the setting is temporary, and only applied to code within the block.

See numpy.printoptions documentation for details on the context manager and what other arguments it supports.

回答 4

听起来您正在使用numpy。

如果是这样，您可以添加：

import numpy as np
np.set_printoptions(threshold=np.nan)

这将禁用边角打印。有关更多信息，请参见此NumPy教程。

This sounds like you’re using numpy.

If that’s the case, you can add:

import numpy as np
np.set_printoptions(threshold=np.nan)

That will disable the corner printing. For more information, see this NumPy Tutorial.

回答 5

这是一种一次性的方法，如果您不想更改默认设置，这将非常有用：

def fullprint(*args, **kwargs):
  from pprint import pprint
  import numpy
  opt = numpy.get_printoptions()
  numpy.set_printoptions(threshold=numpy.inf)
  pprint(*args, **kwargs)
  numpy.set_printoptions(**opt)

Here is a one-off way to do this, which is useful if you don’t want to change your default settings:

def fullprint(*args, **kwargs):
  from pprint import pprint
  import numpy
  opt = numpy.get_printoptions()
  numpy.set_printoptions(threshold=numpy.inf)
  pprint(*args, **kwargs)
  numpy.set_printoptions(**opt)

回答 6

使用上下文管理作为保价 sugggested

import numpy as np


class fullprint:
    'context manager for printing full numpy arrays'

    def __init__(self, **kwargs):
        kwargs.setdefault('threshold', np.inf)
        self.opt = kwargs

    def __enter__(self):
        self._opt = np.get_printoptions()
        np.set_printoptions(**self.opt)

    def __exit__(self, type, value, traceback):
        np.set_printoptions(**self._opt)


if __name__ == '__main__': 
    a = np.arange(1001)

    with fullprint():
        print(a)

    print(a)

    with fullprint(threshold=None, edgeitems=10):
        print(a)

Using a context manager as Paul Price sugggested

import numpy as np


class fullprint:
    'context manager for printing full numpy arrays'

    def __init__(self, **kwargs):
        kwargs.setdefault('threshold', np.inf)
        self.opt = kwargs

    def __enter__(self):
        self._opt = np.get_printoptions()
        np.set_printoptions(**self.opt)

    def __exit__(self, type, value, traceback):
        np.set_printoptions(**self._opt)


if __name__ == '__main__': 
    a = np.arange(1001)

    with fullprint():
        print(a)

    print(a)

    with fullprint(threshold=None, edgeitems=10):
        print(a)

回答 7

numpy.savetxt

numpy.savetxt(sys.stdout, numpy.arange(10000))

或者如果您需要一个字符串：

import StringIO
sio = StringIO.StringIO()
numpy.savetxt(sio, numpy.arange(10000))
s = sio.getvalue()
print s

默认输出格式为：

0.000000000000000000e+00
1.000000000000000000e+00
2.000000000000000000e+00
3.000000000000000000e+00
...

并可以使用其他参数进行配置。

特别要注意的是，它也不会显示方括号，并允许进行大量自定义，如以下内容所述：如何打印不带括号的Numpy数组？

在python 2.7.12，numpy 1.11.1上测试。

numpy.savetxt

numpy.savetxt(sys.stdout, numpy.arange(10000))

or if you need a string:

import StringIO
sio = StringIO.StringIO()
numpy.savetxt(sio, numpy.arange(10000))
s = sio.getvalue()
print s

The default output format is:

0.000000000000000000e+00
1.000000000000000000e+00
2.000000000000000000e+00
3.000000000000000000e+00
...

and it can be configured with further arguments.

Note in particular how this also not shows the square brackets, and allows for a lot of customization, as mentioned at: How to print a Numpy array without brackets?

Tested on Python 2.7.12, numpy 1.11.1.

回答 8

这是一个微小的修饰（除去传递额外的参数选项set_printoptions)的neok的回答。

它显示了如何使用contextlib.contextmanager更少的代码行轻松地创建这样的contextmanager：

import numpy as np
from contextlib import contextmanager

@contextmanager
def show_complete_array():
    oldoptions = np.get_printoptions()
    np.set_printoptions(threshold=np.inf)
    try:
        yield
    finally:
        np.set_printoptions(**oldoptions)

在您的代码中，可以这样使用它：

a = np.arange(1001)

print(a)      # shows the truncated array

with show_complete_array():
    print(a)  # shows the complete array

print(a)      # shows the truncated array (again)

This is a slight modification (removed the option to pass additional arguments to set_printoptions)of neoks answer.

It shows how you can use contextlib.contextmanager to easily create such a contextmanager with fewer lines of code:

import numpy as np
from contextlib import contextmanager

@contextmanager
def show_complete_array():
    oldoptions = np.get_printoptions()
    np.set_printoptions(threshold=np.inf)
    try:
        yield
    finally:
        np.set_printoptions(**oldoptions)

In your code it can be used like this:

a = np.arange(1001)

print(a)      # shows the truncated array

with show_complete_array():
    print(a)  # shows the complete array

print(a)      # shows the truncated array (again)

回答 9

除了最大列数（以固定）之外，此答案numpy.set_printoptions(threshold=numpy.nan)还可以显示一定数量的字符。在某些环境中，例如从bash调用python（而不是交互式会话）时，可以通过如下设置参数来解决此问题linewidth。

import numpy as np
np.set_printoptions(linewidth=2000)    # default = 75
Mat = np.arange(20000,20150).reshape(2,75)    # 150 elements (75 columns)
print(Mat)

在这种情况下，您的窗口应限制换行符的字符数。

对于那些使用sublime文本并希望在输出窗口中查看结果的用户，应将build选项添加"word_wrap": false到sublime-build文件[ source ]中。

Complementary to this answer from the maximum number of columns (fixed with numpy.set_printoptions(threshold=numpy.nan)), there is also a limit of characters to be displayed. In some environments like when calling python from bash (rather than the interactive session), this can be fixed by setting the parameter linewidth as following.

import numpy as np
np.set_printoptions(linewidth=2000)    # default = 75
Mat = np.arange(20000,20150).reshape(2,75)    # 150 elements (75 columns)
print(Mat)

In this case, your window should limit the number of characters to wrap the line.

For those out there using sublime text and wanting to see results within the output window, you should add the build option "word_wrap": false to the sublime-build file [source] .

回答 10

从NumPy 1.16版本开始，有关更多详细信息，请参见GitHub票证12251。

from sys import maxsize
from numpy import set_printoptions

set_printoptions(threshold=maxsize)

Since NumPy version 1.16, for more details see GitHub ticket 12251.

from sys import maxsize
from numpy import set_printoptions

set_printoptions(threshold=maxsize)

回答 11

要关闭它并返回正常模式

np.set_printoptions(threshold=False)

To turn it off and return to the normal mode

np.set_printoptions(threshold=False)

回答 12

假设您有一个numpy数组

 arr = numpy.arange(10000).reshape(250,40)

如果要一次性打印整个数组（不切换np.set_printoptions），但是想要比上下文管理器更简单（更少的代码）的方法，那就做

for row in arr:
     print row

Suppose you have a numpy array

 arr = numpy.arange(10000).reshape(250,40)

If you want to print the full array in a one-off way (without toggling np.set_printoptions), but want something simpler (less code) than the context manager, just do

for row in arr:
     print row

回答 13

稍作修改：（因为您要打印大量列表）

import numpy as np
np.set_printoptions(threshold=np.inf, linewidth=200)

x = np.arange(1000)
print(x)

这将增加每行的字符数（默认线宽为75）。使用任何您喜欢的值作为适合您的编码环境的线宽。通过每行添加更多字符，这将使您不必遍历大量输出行。

A slight modification: (since you are going to print a huge list)

import numpy as np
np.set_printoptions(threshold=np.inf, linewidth=200)

x = np.arange(1000)
print(x)

This will increase the number of characters per line (default linewidth of 75). Use any value you like for the linewidth which suits your coding environment. This will save you from having to go through huge number of output lines by adding more characters per line.

回答 14

您可以使用array2string功能-docs。

a = numpy.arange(10000).reshape(250,40)
print(numpy.array2string(a, threshold=numpy.nan, max_line_width=numpy.nan))
# [Big output]

You can use the array2string function – docs.

a = numpy.arange(10000).reshape(250,40)
print(numpy.array2string(a, threshold=numpy.nan, max_line_width=numpy.nan))
# [Big output]

回答 15

您不会总是希望打印所有项目，尤其是对于大型阵列。

一种显示更多项目的简单方法：

In [349]: ar
Out[349]: array([1, 1, 1, ..., 0, 0, 0])

In [350]: ar[:100]
Out[350]:
array([1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1,
       1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1])

默认情况下，当切片的数组<1000时，它可以正常工作。

You won’t always want all items printed, especially for large arrays.

A simple way to show more items:

In [349]: ar
Out[349]: array([1, 1, 1, ..., 0, 0, 0])

In [350]: ar[:100]
Out[350]:
array([1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1,
       1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1])

It works fine when sliced array < 1000 by default.

回答 16

如果有熊猫

    numpy.arange(10000).reshape(250,40)
    print(pandas.DataFrame(a).to_string(header=False, index=False))

避免了需要重新设置的副作用，numpy.set_printoptions(threshold=sys.maxsize)并且您没有得到numpy.array和方括号。我发现这很方便将大量数组转储到日志文件中

If you have pandas available,

    numpy.arange(10000).reshape(250,40)
    print(pandas.DataFrame(a).to_string(header=False, index=False))

avoids the side effect of requiring a reset of numpy.set_printoptions(threshold=sys.maxsize) and you don’t get the numpy.array and brackets. I find this convenient for dumping a wide array into a log file

回答 17

如果一个数组太大而无法打印，NumPy会自动跳过该数组的中央部分而仅打印角点：要禁用此行为并强制NumPy打印整个数组，可以使用更改打印选项set_printoptions。

>>> np.set_printoptions(threshold='nan')

要么

>>> np.set_printoptions(edgeitems=3,infstr='inf',
... linewidth=75, nanstr='nan', precision=8,
... suppress=False, threshold=1000, formatter=None)

您也可以参考numpy文档 numpy文档中的“或部分”以获取更多帮助。

If an array is too large to be printed, NumPy automatically skips the central part of the array and only prints the corners: To disable this behaviour and force NumPy to print the entire array, you can change the printing options using set_printoptions.

>>> np.set_printoptions(threshold='nan')

>>> np.set_printoptions(edgeitems=3,infstr='inf',
... linewidth=75, nanstr='nan', precision=8,
... suppress=False, threshold=1000, formatter=None)

You can also refer to the numpy documentation numpy documentation for “or part” for more help.

知识问答

将NumPy数组转储到csv文件中

2021年7月25日 Python实用宝典

问题：将NumPy数组转储到csv文件中

有没有办法将NumPy数组转储到CSV文件中？我有一个2D NumPy数组，需要以人类可读的格式转储它。

Is there a way to dump a NumPy array into a CSV file? I have a 2D NumPy array and need to dump it in human-readable format.

回答 0

numpy.savetxt 将数组保存到文本文件。

import numpy
a = numpy.asarray([ [1,2,3], [4,5,6], [7,8,9] ])
numpy.savetxt("foo.csv", a, delimiter=",")

numpy.savetxt saves an array to a text file.

import numpy
a = numpy.asarray([ [1,2,3], [4,5,6], [7,8,9] ])
numpy.savetxt("foo.csv", a, delimiter=",")

回答 1

您可以使用pandas。它确实需要一些额外的内存，因此并不总是可能的，但是它非常快速且易于使用。

import pandas as pd 
pd.DataFrame(np_array).to_csv("path/to/file.csv")

如果您不想要标题或索引，请使用 to_csv("/path/to/file.csv", header=None, index=None)

You can use pandas. It does take some extra memory so it’s not always possible, but it’s very fast and easy to use.

import pandas as pd 
pd.DataFrame(np_array).to_csv("path/to/file.csv")

if you don’t want a header or index, use to_csv("/path/to/file.csv", header=None, index=None)

回答 2

tofile 是执行此操作的便捷功能：

import numpy as np
a = np.asarray([ [1,2,3], [4,5,6], [7,8,9] ])
a.tofile('foo.csv',sep=',',format='%10.5f')

手册页有一些有用的注释：

这是用于快速存储阵列数据的便利功能。有关字节序和精度的信息会丢失，因此对于打算在不同字节序的计算机之间存档数据或传输数据的文件，此方法不是一个好的选择。这些问题中的一些可以通过将数据输出为文本文件来克服，而这是以速度和文件大小为代价的。

注意。此功能不会生成多行的CSV文件，而是将所有内容保存到一行。

tofile is a convenient function to do this:

import numpy as np
a = np.asarray([ [1,2,3], [4,5,6], [7,8,9] ])
a.tofile('foo.csv',sep=',',format='%10.5f')

The man page has some useful notes:

This is a convenience function for quick storage of array data. Information on endianness and precision is lost, so this method is not a good choice for files intended to archive data or transport data between machines with different endianness. Some of these problems can be overcome by outputting the data as text files, at the expense of speed and file size.

Note. This function does not produce multi-line csv files, it saves everything to one line.

回答 3

将记录数组编写为带有标题的CSV文件需要更多的工作。

本示例读取标题为第一行的CSV文件，然后写入相同的文件。

import numpy as np

# Write an example CSV file with headers on first line
with open('example.csv', 'w') as fp:
    fp.write('''\
col1,col2,col3
1,100.1,string1
2,222.2,second string
''')

# Read it as a Numpy record array
ar = np.recfromcsv('example.csv')
print(repr(ar))
# rec.array([(1, 100.1, 'string1'), (2, 222.2, 'second string')], 
#           dtype=[('col1', '<i4'), ('col2', '<f8'), ('col3', 'S13')])

# Write as a CSV file with headers on first line
with open('out.csv', 'w') as fp:
    fp.write(','.join(ar.dtype.names) + '\n')
    np.savetxt(fp, ar, '%s', ',')

请注意，此示例不考虑带逗号的字符串。要考虑非数字数据的引号，请使用以下csv软件包：

import csv

with open('out2.csv', 'wb') as fp:
    writer = csv.writer(fp, quoting=csv.QUOTE_NONNUMERIC)
    writer.writerow(ar.dtype.names)
    writer.writerows(ar.tolist())

Writing record arrays as CSV files with headers requires a bit more work.

This example reads a CSV file with the header on the first line, then writes the same file.

import numpy as np

# Write an example CSV file with headers on first line
with open('example.csv', 'w') as fp:
    fp.write('''\
col1,col2,col3
1,100.1,string1
2,222.2,second string
''')

# Read it as a Numpy record array
ar = np.recfromcsv('example.csv')
print(repr(ar))
# rec.array([(1, 100.1, 'string1'), (2, 222.2, 'second string')], 
#           dtype=[('col1', '<i4'), ('col2', '<f8'), ('col3', 'S13')])

# Write as a CSV file with headers on first line
with open('out.csv', 'w') as fp:
    fp.write(','.join(ar.dtype.names) + '\n')
    np.savetxt(fp, ar, '%s', ',')

Note that this example does not consider strings with commas. To consider quotes for non-numeric data, use the csv package:

import csv

with open('out2.csv', 'wb') as fp:
    writer = csv.writer(fp, quoting=csv.QUOTE_NONNUMERIC)
    writer.writerow(ar.dtype.names)
    writer.writerows(ar.tolist())

回答 4

如前所述，将数组转储为CSV文件的最佳方法是使用.savetxt(...)方法。但是，有些事情我们应该知道如何正确完成。

例如，如果您有一个带dtype = np.int32as 的numpy数组

   narr = np.array([[1,2],
                 [3,4],
                 [5,6]], dtype=np.int32)

并想另存savetxt为

np.savetxt('values.csv', narr, delimiter=",")

它将数据以浮点指数格式存储为

1.000000000000000000e+00,2.000000000000000000e+00
3.000000000000000000e+00,4.000000000000000000e+00
5.000000000000000000e+00,6.000000000000000000e+00

你必须使用一个名为参数更改格式fmt为

np.savetxt('values.csv', narr, fmt="%d", delimiter=",")

以原始格式存储数据

以压缩的gz格式保存数据

此外，savetxt还可用于以.gz压缩格式存储数据，这在通过网络传输数据时可能很有用。

我们只需要更改文件的扩展名，因为.gznumpy会自动处理所有内容

np.savetxt('values.gz', narr, fmt="%d", delimiter=",")

希望能帮助到你

As already discussed, the best way to dump the array into a CSV file is by using .savetxt(...)method. However, there are certain things we should know to do it properly.

For example, if you have a numpy array with dtype = np.int32 as

   narr = np.array([[1,2],
                 [3,4],
                 [5,6]], dtype=np.int32)

and want to save using savetxt as

np.savetxt('values.csv', narr, delimiter=",")

It will store the data in floating point exponential format as

1.000000000000000000e+00,2.000000000000000000e+00
3.000000000000000000e+00,4.000000000000000000e+00
5.000000000000000000e+00,6.000000000000000000e+00

You will have to change the formatting by using a parameter called fmt as

np.savetxt('values.csv', narr, fmt="%d", delimiter=",")

to store data in its original format

Saving Data in Compressed gz format

Also, savetxt can be used for storing data in .gz compressed format which might be useful while transferring data over network.

We just need to change the extension of the file as .gz and numpy will take care of everything automatically

np.savetxt('values.gz', narr, fmt="%d", delimiter=",")

Hope it helps

回答 5

我相信您也可以很简单地完成此操作，如下所示：

将Numpy数组转换为Pandas数据框
另存为CSV

例如＃1：

    # Libraries to import
    import pandas as pd
    import nump as np

    #N x N numpy array (dimensions dont matter)
    corr_mat    #your numpy array
    my_df = pd.DataFrame(corr_mat)  #converting it to a pandas dataframe

例如＃2：

    #save as csv 
    my_df.to_csv('foo.csv', index=False)   # "foo" is the name you want to give
                                           # to csv file. Make sure to add ".csv"
                                           # after whatever name like in the code

I believe you can also accomplish this quite simply as follows:

Convert Numpy array into a Pandas dataframe
Save as CSV

e.g. #1:

    # Libraries to import
    import pandas as pd
    import nump as np

    #N x N numpy array (dimensions dont matter)
    corr_mat    #your numpy array
    my_df = pd.DataFrame(corr_mat)  #converting it to a pandas dataframe

e.g. #2:

    #save as csv 
    my_df.to_csv('foo.csv', index=False)   # "foo" is the name you want to give
                                           # to csv file. Make sure to add ".csv"
                                           # after whatever name like in the code

回答 6

如果要在列中写：

    for x in np.nditer(a.T, order='C'): 
            file.write(str(x))
            file.write("\n")

这里的“ a”是numpy数组的名称，“文件”是要写入文件的变量。

如果要写在行中：

    writer= csv.writer(file, delimiter=',')
    for x in np.nditer(a.T, order='C'): 
            row.append(str(x))
    writer.writerow(row)

if you want to write in column:

    for x in np.nditer(a.T, order='C'): 
            file.write(str(x))
            file.write("\n")

Here ‘a’ is the name of numpy array and ‘file’ is the variable to write in a file.

If you want to write in row:

    writer= csv.writer(file, delimiter=',')
    for x in np.nditer(a.T, order='C'): 
            row.append(str(x))
    writer.writerow(row)

回答 7

如果要将numpy数组（例如your_array = np.array([[1,2],[3,4]])）保存到一个单元格，可以先使用进行转换your_array.tolist()。

然后将其以正常方式保存到一个单元格中，并且delimiter=';' 和，csv文件中的单元格将如下所示[[1, 2], [2, 4]]

然后，您可以像这样恢复阵列： your_array = np.array(ast.literal_eval(cell_string))

If you want to save your numpy array (e.g. your_array = np.array([[1,2],[3,4]])) to one cell, you could convert it first with your_array.tolist().

Then save it the normal way to one cell, with delimiter=';' and the cell in the csv-file will look like this [[1, 2], [2, 4]]

Then you could restore your array like this: your_array = np.array(ast.literal_eval(cell_string))

回答 8

您也可以使用纯python而不使用任何模块来完成此操作。

# format as a block of csv text to do whatever you want
csv_rows = ["{},{}".format(i, j) for i, j in array]
csv_text = "\n".join(csv_rows)

# write it to a file
with open('file.csv', 'w') as f:
    f.write(csv_text)

You can also do it with pure python without using any modules.

# format as a block of csv text to do whatever you want
csv_rows = ["{},{}".format(i, j) for i, j in array]
csv_text = "\n".join(csv_rows)

# write it to a file
with open('file.csv', 'w') as f:
    f.write(csv_text)

回答 9

在Python中，我们使用csv.writer（）模块将数据写入csv文件。该模块类似于csv.reader（）模块。

import csv

person = [['SN', 'Person', 'DOB'],
['1', 'John', '18/1/1997'],
['2', 'Marie','19/2/1998'],
['3', 'Simon','20/3/1999'],
['4', 'Erik', '21/4/2000'],
['5', 'Ana', '22/5/2001']]

csv.register_dialect('myDialect',
delimiter = '|',
quoting=csv.QUOTE_NONE,
skipinitialspace=True)

with open('dob.csv', 'w') as f:
    writer = csv.writer(f, dialect='myDialect')
    for row in person:
       writer.writerow(row)

f.close()

定界符是用于分隔字段的字符串。默认值为comma（，）。

In Python we use csv.writer() module to write data into csv files. This module is similar to the csv.reader() module.

import csv

person = [['SN', 'Person', 'DOB'],
['1', 'John', '18/1/1997'],
['2', 'Marie','19/2/1998'],
['3', 'Simon','20/3/1999'],
['4', 'Erik', '21/4/2000'],
['5', 'Ana', '22/5/2001']]

csv.register_dialect('myDialect',
delimiter = '|',
quoting=csv.QUOTE_NONE,
skipinitialspace=True)

with open('dob.csv', 'w') as f:
    writer = csv.writer(f, dialect='myDialect')
    for row in person:
       writer.writerow(row)

f.close()

A delimiter is a string used to separate fields. The default value is comma(,).

知识问答

如何使用NumPy计算欧几里得距离？

2021年7月25日 Python实用宝典

问题：如何使用NumPy计算欧几里得距离？

我在3D中有两点：

(xa, ya, za)
(xb, yb, zb)

我想计算距离：

dist = sqrt((xa-xb)^2 + (ya-yb)^2 + (za-zb)^2)

使用NumPy或一般使用Python的最佳方法是什么？我有：

import numpy
a = numpy.array((xa ,ya, za))
b = numpy.array((xb, yb, zb))

I have two points in 3D:

(xa, ya, za)
(xb, yb, zb)

And I want to calculate the distance:

dist = sqrt((xa-xb)^2 + (ya-yb)^2 + (za-zb)^2)

What’s the best way to do this with NumPy, or with Python in general? I have:

import numpy
a = numpy.array((xa ,ya, za))
b = numpy.array((xb, yb, zb))

回答 0

用途numpy.linalg.norm：

dist = numpy.linalg.norm(a-b)

您可以在“数据挖掘导论”中找到其背后的理论

这是有效的，因为欧几里得距离为l2范数，并且numpy.linalg.norm 中ord参数的默认值为2。

Use numpy.linalg.norm:

dist = numpy.linalg.norm(a-b)

You can find the theory behind this in Introduction to Data Mining

This works because Euclidean distance is l2 norm and the default value of ord parameter in numpy.linalg.norm is 2.

回答 1

SciPy中有一个功能。称为欧几里得。

例：

from scipy.spatial import distance
a = (1, 2, 3)
b = (4, 5, 6)
dst = distance.euclidean(a, b)

There’s a function for that in SciPy. It’s called Euclidean.

Example:

from scipy.spatial import distance
a = (1, 2, 3)
b = (4, 5, 6)
dst = distance.euclidean(a, b)

回答 2

对于有兴趣一次计算多个距离的任何人，我已经使用perfplot（我的一个小项目）进行了一些比较。

第一个建议是组织数据，使数组具有维(3, n)（并且显然是C连续的）。如果添加在连续的第一个维度发生，事情是更快，它没有太大的关系，如果您使用sqrt-sum与axis=0，linalg.norm与axis=0，或

a_min_b = a - b
numpy.sqrt(numpy.einsum('ij,ij->j', a_min_b, a_min_b))

这是最快的变体。（实际上也只适用于一行。）

您在第二个轴上进行汇总的变体axis=1都慢得多。

复制剧情的代码：

import numpy
import perfplot
from scipy.spatial import distance


def linalg_norm(data):
    a, b = data[0]
    return numpy.linalg.norm(a - b, axis=1)


def linalg_norm_T(data):
    a, b = data[1]
    return numpy.linalg.norm(a - b, axis=0)


def sqrt_sum(data):
    a, b = data[0]
    return numpy.sqrt(numpy.sum((a - b) ** 2, axis=1))


def sqrt_sum_T(data):
    a, b = data[1]
    return numpy.sqrt(numpy.sum((a - b) ** 2, axis=0))


def scipy_distance(data):
    a, b = data[0]
    return list(map(distance.euclidean, a, b))


def sqrt_einsum(data):
    a, b = data[0]
    a_min_b = a - b
    return numpy.sqrt(numpy.einsum("ij,ij->i", a_min_b, a_min_b))


def sqrt_einsum_T(data):
    a, b = data[1]
    a_min_b = a - b
    return numpy.sqrt(numpy.einsum("ij,ij->j", a_min_b, a_min_b))


def setup(n):
    a = numpy.random.rand(n, 3)
    b = numpy.random.rand(n, 3)
    out0 = numpy.array([a, b])
    out1 = numpy.array([a.T, b.T])
    return out0, out1


perfplot.save(
    "norm.png",
    setup=setup,
    n_range=[2 ** k for k in range(22)],
    kernels=[
        linalg_norm,
        linalg_norm_T,
        scipy_distance,
        sqrt_sum,
        sqrt_sum_T,
        sqrt_einsum,
        sqrt_einsum_T,
    ],
    logx=True,
    logy=True,
    xlabel="len(x), len(y)",
)

For anyone interested in computing multiple distances at once, I’ve done a little comparison using perfplot (a small project of mine).

The first advice is to organize your data such that the arrays have dimension (3, n) (and are C-contiguous obviously). If adding happens in the contiguous first dimension, things are faster, and it doesn’t matter too much if you use sqrt-sum with axis=0, linalg.norm with axis=0, or

a_min_b = a - b
numpy.sqrt(numpy.einsum('ij,ij->j', a_min_b, a_min_b))

which is, by a slight margin, the fastest variant. (That actually holds true for just one row as well.)

The variants where you sum up over the second axis, axis=1, are all substantially slower.

Code to reproduce the plot:

import numpy
import perfplot
from scipy.spatial import distance


def linalg_norm(data):
    a, b = data[0]
    return numpy.linalg.norm(a - b, axis=1)


def linalg_norm_T(data):
    a, b = data[1]
    return numpy.linalg.norm(a - b, axis=0)


def sqrt_sum(data):
    a, b = data[0]
    return numpy.sqrt(numpy.sum((a - b) ** 2, axis=1))


def sqrt_sum_T(data):
    a, b = data[1]
    return numpy.sqrt(numpy.sum((a - b) ** 2, axis=0))


def scipy_distance(data):
    a, b = data[0]
    return list(map(distance.euclidean, a, b))


def sqrt_einsum(data):
    a, b = data[0]
    a_min_b = a - b
    return numpy.sqrt(numpy.einsum("ij,ij->i", a_min_b, a_min_b))


def sqrt_einsum_T(data):
    a, b = data[1]
    a_min_b = a - b
    return numpy.sqrt(numpy.einsum("ij,ij->j", a_min_b, a_min_b))


def setup(n):
    a = numpy.random.rand(n, 3)
    b = numpy.random.rand(n, 3)
    out0 = numpy.array([a, b])
    out1 = numpy.array([a.T, b.T])
    return out0, out1


perfplot.save(
    "norm.png",
    setup=setup,
    n_range=[2 ** k for k in range(22)],
    kernels=[
        linalg_norm,
        linalg_norm_T,
        scipy_distance,
        sqrt_sum,
        sqrt_sum_T,
        sqrt_einsum,
        sqrt_einsum_T,
    ],
    logx=True,
    logy=True,
    xlabel="len(x), len(y)",
)

回答 3

我想用各种性能说明来解释简单答案。np.linalg.norm可能会做比您需要的更多的工作：

dist = numpy.linalg.norm(a-b)

首先-该功能的目的是工作在一个列表，并返回所有的值，例如到距离比较pA的点的集合sP：

sP = set(points)
pA = point
distances = np.linalg.norm(sP - pA, ord=2, axis=1.)  # 'distances' is a list

记住几件事：

Python函数调用非常昂贵。
[常规] Python不缓存名称查找。

所以

def distance(pointA, pointB):
    dist = np.linalg.norm(pointA - pointB)
    return dist

没有看起来那么天真。

>>> dis.dis(distance)
  2           0 LOAD_GLOBAL              0 (np)
              2 LOAD_ATTR                1 (linalg)
              4 LOAD_ATTR                2 (norm)
              6 LOAD_FAST                0 (pointA)
              8 LOAD_FAST                1 (pointB)
             10 BINARY_SUBTRACT
             12 CALL_FUNCTION            1
             14 STORE_FAST               2 (dist)

  3          16 LOAD_FAST                2 (dist)
             18 RETURN_VALUE

首先，每次调用时，我们都必须对“ np”进行全局查找，对“ linalg”进行有范围的查找，对“ norm”进行有范围的查找，以及仅调用的开销该函数就相当于数十个python。说明。

最后，我们浪费了两个操作来存储结果并重新加载以返回结果…

改进的第一步：加快查找速度，跳过商店

def distance(pointA, pointB, _norm=np.linalg.norm):
    return _norm(pointA - pointB)

我们得到了更加简化：

>>> dis.dis(distance)
  2           0 LOAD_FAST                2 (_norm)
              2 LOAD_FAST                0 (pointA)
              4 LOAD_FAST                1 (pointB)
              6 BINARY_SUBTRACT
              8 CALL_FUNCTION            1
             10 RETURN_VALUE

但是，函数调用开销仍然需要完成一些工作。而且，您需要进行基准测试以确定您自己做数学是否会更好：

def distance(pointA, pointB):
    return (
        ((pointA.x - pointB.x) ** 2) +
        ((pointA.y - pointB.y) ** 2) +
        ((pointA.z - pointB.z) ** 2)
    ) ** 0.5  # fast sqrt

在某些平台上，**0.5速度比math.sqrt。你的旅费可能会改变。

****高级性能说明。

为什么要计算距离？如果唯一的目的是显示它，

 print("The target is %.2fm away" % (distance(a, b)))

向前走。但是，如果您要比较距离，进行范围检查等，我想添加一些有用的性能观察。

让我们采取两种情况：按距离排序或将列表筛选为满足范围约束的项目。

# Ultra naive implementations. Hold onto your hat.

def sort_things_by_distance(origin, things):
    return things.sort(key=lambda thing: distance(origin, thing))

def in_range(origin, range, things):
    things_in_range = []
    for thing in things:
        if distance(origin, thing) <= range:
            things_in_range.append(thing)

我们需要记住的第一件事是我们正在使用毕达哥拉斯来计算距离（dist = sqrt(x^2 + y^2 + z^2)），因此我们进行了很多sqrt通话。数学101：

dist = root ( x^2 + y^2 + z^2 )
:.
dist^2 = x^2 + y^2 + z^2
and
sq(N) < sq(M) iff M > N
and
sq(N) > sq(M) iff N > M
and
sq(N) = sq(M) iff N == M

简而言之：直到我们实际需要以X而不是X ^ 2为单位的距离，我们才能消除计算中最困难的部分。

# Still naive, but much faster.

def distance_sq(left, right):
    """ Returns the square of the distance between left and right. """
    return (
        ((left.x - right.x) ** 2) +
        ((left.y - right.y) ** 2) +
        ((left.z - right.z) ** 2)
    )

def sort_things_by_distance(origin, things):
    return things.sort(key=lambda thing: distance_sq(origin, thing))

def in_range(origin, range, things):
    things_in_range = []

    # Remember that sqrt(N)**2 == N, so if we square
    # range, we don't need to root the distances.
    range_sq = range**2

    for thing in things:
        if distance_sq(origin, thing) <= range_sq:
            things_in_range.append(thing)

太好了，这两个函数不再需要任何昂贵的平方根。这样会更快。我们还可以通过将in_range转换为生成器来改进它：

def in_range(origin, range, things):
    range_sq = range**2
    yield from (thing for thing in things
                if distance_sq(origin, thing) <= range_sq)

如果您正在执行以下操作，则这尤其有好处：

if any(in_range(origin, max_dist, things)):
    ...

但是，如果接下来要做的事情需要一段距离，

for nearby in in_range(origin, walking_distance, hotdog_stands):
    print("%s %.2fm" % (nearby.name, distance(origin, nearby)))

考虑产生元组：

def in_range_with_dist_sq(origin, range, things):
    range_sq = range**2
    for thing in things:
        dist_sq = distance_sq(origin, thing)
        if dist_sq <= range_sq: yield (thing, dist_sq)

如果您可以进行范围检查（“找到在X附近且在Y Nm之内的东西”，因为您不必再次计算距离），这将特别有用。

但是，如果我们要搜索的清单很大，那things又有很多不值得考虑的地方怎么办？

实际上有一个非常简单的优化：

def in_range_all_the_things(origin, range, things):
    range_sq = range**2
    for thing in things:
        dist_sq = (origin.x - thing.x) ** 2
        if dist_sq <= range_sq:
            dist_sq += (origin.y - thing.y) ** 2
            if dist_sq <= range_sq:
                dist_sq += (origin.z - thing.z) ** 2
                if dist_sq <= range_sq:
                    yield thing

这是否有用将取决于“事物”的大小。

def in_range_all_the_things(origin, range, things):
    range_sq = range**2
    if len(things) >= 4096:
        for thing in things:
            dist_sq = (origin.x - thing.x) ** 2
            if dist_sq <= range_sq:
                dist_sq += (origin.y - thing.y) ** 2
                if dist_sq <= range_sq:
                    dist_sq += (origin.z - thing.z) ** 2
                    if dist_sq <= range_sq:
                        yield thing
    elif len(things) > 32:
        for things in things:
            dist_sq = (origin.x - thing.x) ** 2
            if dist_sq <= range_sq:
                dist_sq += (origin.y - thing.y) ** 2 + (origin.z - thing.z) ** 2
                if dist_sq <= range_sq:
                    yield thing
    else:
        ... just calculate distance and range-check it ...

再次考虑产生dist_sq。然后我们的热狗示例变为：

# Chaining generators
info = in_range_with_dist_sq(origin, walking_distance, hotdog_stands)
info = (stand, dist_sq**0.5 for stand, dist_sq in info)
for stand, dist in info:
    print("%s %.2fm" % (stand, dist))

I want to expound on the simple answer with various performance notes. np.linalg.norm will do perhaps more than you need:

dist = numpy.linalg.norm(a-b)

Firstly – this function is designed to work over a list and return all of the values, e.g. to compare the distance from pA to the set of points sP:

sP = set(points)
pA = point
distances = np.linalg.norm(sP - pA, ord=2, axis=1.)  # 'distances' is a list

Remember several things:

Python function calls are expensive.
[Regular] Python doesn’t cache name lookups.

def distance(pointA, pointB):
    dist = np.linalg.norm(pointA - pointB)
    return dist

isn’t as innocent as it looks.

>>> dis.dis(distance)
  2           0 LOAD_GLOBAL              0 (np)
              2 LOAD_ATTR                1 (linalg)
              4 LOAD_ATTR                2 (norm)
              6 LOAD_FAST                0 (pointA)
              8 LOAD_FAST                1 (pointB)
             10 BINARY_SUBTRACT
             12 CALL_FUNCTION            1
             14 STORE_FAST               2 (dist)

  3          16 LOAD_FAST                2 (dist)
             18 RETURN_VALUE

Firstly – every time we call it, we have to do a global lookup for “np”, a scoped lookup for “linalg” and a scoped lookup for “norm”, and the overhead of merely calling the function can equate to dozens of python instructions.

Lastly, we wasted two operations on to store the result and reload it for return…

First pass at improvement: make the lookup faster, skip the store

def distance(pointA, pointB, _norm=np.linalg.norm):
    return _norm(pointA - pointB)

We get the far more streamlined:

>>> dis.dis(distance)
  2           0 LOAD_FAST                2 (_norm)
              2 LOAD_FAST                0 (pointA)
              4 LOAD_FAST                1 (pointB)
              6 BINARY_SUBTRACT
              8 CALL_FUNCTION            1
             10 RETURN_VALUE

The function call overhead still amounts to some work, though. And you’ll want to do benchmarks to determine whether you might be better doing the math yourself:

def distance(pointA, pointB):
    return (
        ((pointA.x - pointB.x) ** 2) +
        ((pointA.y - pointB.y) ** 2) +
        ((pointA.z - pointB.z) ** 2)
    ) ** 0.5  # fast sqrt

On some platforms, **0.5 is faster than math.sqrt. Your mileage may vary.

**** Advanced performance notes.

Why are you calculating distance? If the sole purpose is to display it,

 print("The target is %.2fm away" % (distance(a, b)))

move along. But if you’re comparing distances, doing range checks, etc., I’d like to add some useful performance observations.

Let’s take two cases: sorting by distance or culling a list to items that meet a range constraint.

# Ultra naive implementations. Hold onto your hat.

def sort_things_by_distance(origin, things):
    return things.sort(key=lambda thing: distance(origin, thing))

def in_range(origin, range, things):
    things_in_range = []
    for thing in things:
        if distance(origin, thing) <= range:
            things_in_range.append(thing)

The first thing we need to remember is that we are using Pythagoras to calculate the distance (dist = sqrt(x^2 + y^2 + z^2)) so we’re making a lot of sqrt calls. Math 101:

dist = root ( x^2 + y^2 + z^2 )
:.
dist^2 = x^2 + y^2 + z^2
and
sq(N) < sq(M) iff M > N
and
sq(N) > sq(M) iff N > M
and
sq(N) = sq(M) iff N == M

In short: until we actually require the distance in a unit of X rather than X^2, we can eliminate the hardest part of the calculations.

# Still naive, but much faster.

def distance_sq(left, right):
    """ Returns the square of the distance between left and right. """
    return (
        ((left.x - right.x) ** 2) +
        ((left.y - right.y) ** 2) +
        ((left.z - right.z) ** 2)
    )

def sort_things_by_distance(origin, things):
    return things.sort(key=lambda thing: distance_sq(origin, thing))

def in_range(origin, range, things):
    things_in_range = []

    # Remember that sqrt(N)**2 == N, so if we square
    # range, we don't need to root the distances.
    range_sq = range**2

    for thing in things:
        if distance_sq(origin, thing) <= range_sq:
            things_in_range.append(thing)

Great, both functions no-longer do any expensive square roots. That’ll be much faster. We can also improve in_range by converting it to a generator:

def in_range(origin, range, things):
    range_sq = range**2
    yield from (thing for thing in things
                if distance_sq(origin, thing) <= range_sq)

This especially has benefits if you are doing something like:

if any(in_range(origin, max_dist, things)):
    ...

But if the very next thing you are going to do requires a distance,

for nearby in in_range(origin, walking_distance, hotdog_stands):
    print("%s %.2fm" % (nearby.name, distance(origin, nearby)))

consider yielding tuples:

def in_range_with_dist_sq(origin, range, things):
    range_sq = range**2
    for thing in things:
        dist_sq = distance_sq(origin, thing)
        if dist_sq <= range_sq: yield (thing, dist_sq)

This can be especially useful if you might chain range checks (‘find things that are near X and within Nm of Y’, since you don’t have to calculate the distance again).

But what about if we’re searching a really large list of things and we anticipate a lot of them not being worth consideration?

There is actually a very simple optimization:

def in_range_all_the_things(origin, range, things):
    range_sq = range**2
    for thing in things:
        dist_sq = (origin.x - thing.x) ** 2
        if dist_sq <= range_sq:
            dist_sq += (origin.y - thing.y) ** 2
            if dist_sq <= range_sq:
                dist_sq += (origin.z - thing.z) ** 2
                if dist_sq <= range_sq:
                    yield thing

Whether this is useful will depend on the size of ‘things’.

def in_range_all_the_things(origin, range, things):
    range_sq = range**2
    if len(things) >= 4096:
        for thing in things:
            dist_sq = (origin.x - thing.x) ** 2
            if dist_sq <= range_sq:
                dist_sq += (origin.y - thing.y) ** 2
                if dist_sq <= range_sq:
                    dist_sq += (origin.z - thing.z) ** 2
                    if dist_sq <= range_sq:
                        yield thing
    elif len(things) > 32:
        for things in things:
            dist_sq = (origin.x - thing.x) ** 2
            if dist_sq <= range_sq:
                dist_sq += (origin.y - thing.y) ** 2 + (origin.z - thing.z) ** 2
                if dist_sq <= range_sq:
                    yield thing
    else:
        ... just calculate distance and range-check it ...

And again, consider yielding the dist_sq. Our hotdog example then becomes:

# Chaining generators
info = in_range_with_dist_sq(origin, walking_distance, hotdog_stands)
info = (stand, dist_sq**0.5 for stand, dist_sq in info)
for stand, dist in info:
    print("%s %.2fm" % (stand, dist))

回答 4

此问题解决方法的另一个实例：

def dist(x,y):   
    return numpy.sqrt(numpy.sum((x-y)**2))

a = numpy.array((xa,ya,za))
b = numpy.array((xb,yb,zb))
dist_a_b = dist(a,b)

Another instance of this problem solving method:

def dist(x,y):   
    return numpy.sqrt(numpy.sum((x-y)**2))

a = numpy.array((xa,ya,za))
b = numpy.array((xb,yb,zb))
dist_a_b = dist(a,b)

回答 5

从开始Python 3.8，math模块直接提供dist函数，该函数返回两点之间的欧几里得距离（以元组或坐标列表形式给出）：

from math import dist

dist((1, 2, 6), (-2, 3, 2)) # 5.0990195135927845

如果您正在使用列表：

dist([1, 2, 6], [-2, 3, 2]) # 5.0990195135927845

Starting Python 3.8, the math module directly provides the dist function, which returns the euclidean distance between two points (given as tuples or lists of coordinates):

from math import dist

dist((1, 2, 6), (-2, 3, 2)) # 5.0990195135927845

And if you’re working with lists:

dist([1, 2, 6], [-2, 3, 2]) # 5.0990195135927845

回答 6

可以像下面这样完成。我不知道它有多快，但是它没有使用NumPy。

from math import sqrt
a = (1, 2, 3) # Data point 1
b = (4, 5, 6) # Data point 2
print sqrt(sum( (a - b)**2 for a, b in zip(a, b)))

It can be done like the following. I don’t know how fast it is, but it’s not using NumPy.

from math import sqrt
a = (1, 2, 3) # Data point 1
b = (4, 5, 6) # Data point 2
print sqrt(sum( (a - b)**2 for a, b in zip(a, b)))

回答 7

我在matplotlib.mlab中找到了一个“ dist”函数，但我认为它并不方便。

我将其发布在这里仅供参考。

import numpy as np
import matplotlib as plt

a = np.array([1, 2, 3])
b = np.array([2, 3, 4])

# Distance between a and b
dis = plt.mlab.dist(a, b)

I find a ‘dist’ function in matplotlib.mlab, but I don’t think it’s handy enough.

I’m posting it here just for reference.

import numpy as np
import matplotlib as plt

a = np.array([1, 2, 3])
b = np.array([2, 3, 4])

# Distance between a and b
dis = plt.mlab.dist(a, b)

回答 8

我喜欢np.dot（点积）：

a = numpy.array((xa,ya,za))
b = numpy.array((xb,yb,zb))

distance = (np.dot(a-b,a-b))**.5

I like np.dot (dot product):

a = numpy.array((xa,ya,za))
b = numpy.array((xb,yb,zb))

distance = (np.dot(a-b,a-b))**.5

回答 9

一个不错的单线：

dist = numpy.linalg.norm(a-b)

但是，如果需要考虑速度，建议您在计算机上进行实验。我发现在我的机器上使用带有操作符的平方math函数库比单行NumPy解决方案快得多。sqrt**

我使用以下简单程序运行了测试：

#!/usr/bin/python
import math
import numpy
from random import uniform

def fastest_calc_dist(p1,p2):
    return math.sqrt((p2[0] - p1[0]) ** 2 +
                     (p2[1] - p1[1]) ** 2 +
                     (p2[2] - p1[2]) ** 2)

def math_calc_dist(p1,p2):
    return math.sqrt(math.pow((p2[0] - p1[0]), 2) +
                     math.pow((p2[1] - p1[1]), 2) +
                     math.pow((p2[2] - p1[2]), 2))

def numpy_calc_dist(p1,p2):
    return numpy.linalg.norm(numpy.array(p1)-numpy.array(p2))

TOTAL_LOCATIONS = 1000

p1 = dict()
p2 = dict()
for i in range(0, TOTAL_LOCATIONS):
    p1[i] = (uniform(0,1000),uniform(0,1000),uniform(0,1000))
    p2[i] = (uniform(0,1000),uniform(0,1000),uniform(0,1000))

total_dist = 0
for i in range(0, TOTAL_LOCATIONS):
    for j in range(0, TOTAL_LOCATIONS):
        dist = fastest_calc_dist(p1[i], p2[j]) #change this line for testing
        total_dist += dist

print total_dist

在我的机器上，math_calc_dist运行速度比numpy_calc_dist：1.5秒和23.5秒。

为了获得与之间的可测量差异fastest_calc_dist，math_calc_dist我必须达到TOTAL_LOCATIONS6000。然后fastest_calc_dist花费〜50 秒，而math_calc_dist花费〜60秒。

您也可以尝试用numpy.sqrt和numpy.square，虽然均高于较慢math我的机器上的替代品。

我的测试是使用Python 2.6.6运行的。

A nice one-liner:

dist = numpy.linalg.norm(a-b)

However, if speed is a concern I would recommend experimenting on your machine. I’ve found that using math library’s sqrt with the ** operator for the square is much faster on my machine than the one-liner NumPy solution.

I ran my tests using this simple program:

#!/usr/bin/python
import math
import numpy
from random import uniform

def fastest_calc_dist(p1,p2):
    return math.sqrt((p2[0] - p1[0]) ** 2 +
                     (p2[1] - p1[1]) ** 2 +
                     (p2[2] - p1[2]) ** 2)

def math_calc_dist(p1,p2):
    return math.sqrt(math.pow((p2[0] - p1[0]), 2) +
                     math.pow((p2[1] - p1[1]), 2) +
                     math.pow((p2[2] - p1[2]), 2))

def numpy_calc_dist(p1,p2):
    return numpy.linalg.norm(numpy.array(p1)-numpy.array(p2))

TOTAL_LOCATIONS = 1000

p1 = dict()
p2 = dict()
for i in range(0, TOTAL_LOCATIONS):
    p1[i] = (uniform(0,1000),uniform(0,1000),uniform(0,1000))
    p2[i] = (uniform(0,1000),uniform(0,1000),uniform(0,1000))

total_dist = 0
for i in range(0, TOTAL_LOCATIONS):
    for j in range(0, TOTAL_LOCATIONS):
        dist = fastest_calc_dist(p1[i], p2[j]) #change this line for testing
        total_dist += dist

print total_dist

On my machine, math_calc_dist runs much faster than numpy_calc_dist: 1.5 seconds versus 23.5 seconds.

To get a measurable difference between fastest_calc_dist and math_calc_dist I had to up TOTAL_LOCATIONS to 6000. Then fastest_calc_dist takes ~50 seconds while math_calc_dist takes ~60 seconds.

You can also experiment with numpy.sqrt and numpy.square though both were slower than the math alternatives on my machine.

My tests were run with Python 2.6.6.

回答 10

您可以先减去向量，然后减去内积。

按照您的示例，

a = numpy.array((xa, ya, za))
b = numpy.array((xb, yb, zb))

tmp = a - b
sum_squared = numpy.dot(tmp.T, tmp)
result = sqrt(sum_squared)

You can just subtract the vectors and then innerproduct.

Following your example,

a = numpy.array((xa, ya, za))
b = numpy.array((xb, yb, zb))

tmp = a - b
sum_squared = numpy.dot(tmp.T, tmp)
result = sqrt(sum_squared)

回答 11

拥有a并b定义它们时，还可以使用：

distance = np.sqrt(np.sum((a-b)**2))

Having a and b as you defined them, you can use also:

distance = np.sqrt(np.sum((a-b)**2))

回答 12

使用Python 3.8，这非常容易。

https://docs.python.org/3/library/math.html#math.dist

math.dist(p, q)

返回两个点p和q之间的欧几里得距离，每个点以坐标序列（或可迭代）给出。这两个点必须具有相同的尺寸。

大致相当于：

sqrt(sum((px - qx) ** 2.0 for px, qx in zip(p, q)))

With Python 3.8, it’s very easy.

https://docs.python.org/3/library/math.html#math.dist

math.dist(p, q)

Return the Euclidean distance between two points p and q, each given as a sequence (or iterable) of coordinates. The two points must have the same dimension.

Roughly equivalent to:

sqrt(sum((px - qx) ** 2.0 for px, qx in zip(p, q)))

回答 13

这是一些Python中的欧几里得距离的简洁代码，给出了用Python列表表示的两个点。

def distance(v1,v2): 
    return sum([(x-y)**2 for (x,y) in zip(v1,v2)])**(0.5)

Here’s some concise code for Euclidean distance in Python given two points represented as lists in Python.

def distance(v1,v2): 
    return sum([(x-y)**2 for (x,y) in zip(v1,v2)])**(0.5)

回答 14

从python 3.8开始

从Python 3.8开始，该math模块包含函数math.dist()。
请参阅https://docs.python.org/3.8/library/math.html#math.dist。

math.dist（p1，p2）
返回两个点p1和p2之间的欧几里得距离，每个点均以坐标序列（或可迭代）给出。

import math
print( math.dist( (0,0),   (1,1)   )) # sqrt(2) -> 1.4142
print( math.dist( (0,0,0), (1,1,1) )) # sqrt(3) -> 1.7321

Since Python 3.8

Since Python 3.8 the math module includes the function math.dist().
See here https://docs.python.org/3.8/library/math.html#math.dist.

math.dist(p1, p2)
Return the Euclidean distance between two points p1 and p2, each given as a sequence (or iterable) of coordinates.

import math
print( math.dist( (0,0),   (1,1)   )) # sqrt(2) -> 1.4142
print( math.dist( (0,0,0), (1,1,1) )) # sqrt(3) -> 1.7321

回答 15

计算多维空间的欧几里得距离：

 import math

 x = [1, 2, 6] 
 y = [-2, 3, 2]

 dist = math.sqrt(sum([(xi-yi)**2 for xi,yi in zip(x, y)]))
 5.0990195135927845

Calculate the Euclidean distance for multidimensional space:

 import math

 x = [1, 2, 6] 
 y = [-2, 3, 2]

 dist = math.sqrt(sum([(xi-yi)**2 for xi,yi in zip(x, y)]))
 5.0990195135927845

回答 16

import numpy as np
from scipy.spatial import distance
input_arr = np.array([[0,3,0],[2,0,0],[0,1,3],[0,1,2],[-1,0,1],[1,1,1]]) 
test_case = np.array([0,0,0])
dst=[]
for i in range(0,6):
    temp = distance.euclidean(test_case,input_arr[i])
    dst.append(temp)
print(dst)

import numpy as np
from scipy.spatial import distance
input_arr = np.array([[0,3,0],[2,0,0],[0,1,3],[0,1,2],[-1,0,1],[1,1,1]]) 
test_case = np.array([0,0,0])
dst=[]
for i in range(0,6):
    temp = distance.euclidean(test_case,input_arr[i])
    dst.append(temp)
print(dst)

回答 17

import math

dist = math.hypot(math.hypot(xa-xb, ya-yb), za-zb)

import math

dist = math.hypot(math.hypot(xa-xb, ya-yb), za-zb)

回答 18

您可以轻松使用公式

distance = np.sqrt(np.sum(np.square(a-b)))

实际上，这无非是使用毕达哥拉斯定理来计算距离，方法是将Δx，Δy和Δz的平方相加并取根。

You can easily use the formula

distance = np.sqrt(np.sum(np.square(a-b)))

which does actually nothing more than using Pythagoras’ theorem to calculate the distance, by adding the squares of Δx, Δy and Δz and rooting the result.

回答 19

首先找到两个矩阵的差。然后，使用numpy的乘法命令应用逐元素乘法。之后，找到逐元素相乘的新矩阵的总和。最后，找到求和的平方根。

def findEuclideanDistance(a, b):
    euclidean_distance = a - b
    euclidean_distance = np.sum(np.multiply(euclidean_distance, euclidean_distance))
    euclidean_distance = np.sqrt(euclidean_distance)
    return euclidean_distance

Find difference of two matrices first. Then, apply element wise multiplication with numpy’s multiply command. After then, find summation of the element wise multiplied new matrix. Finally, find square root of the summation.

def findEuclideanDistance(a, b):
    euclidean_distance = a - b
    euclidean_distance = np.sum(np.multiply(euclidean_distance, euclidean_distance))
    euclidean_distance = np.sqrt(euclidean_distance)
    return euclidean_distance

回答 20

import numpy as np
# any two python array as two points
a = [0, 0]
b = [3, 4]

您首先将列表更改为numpy array并执行以下操作：print(np.linalg.norm(np.array(a) - np.array(b)))。直接从python列表中获取的第二种方法为：print(np.linalg.norm(np.subtract(a,b)))

import numpy as np
# any two python array as two points
a = [0, 0]
b = [3, 4]

You first change list to numpy array and do like this: print(np.linalg.norm(np.array(a) - np.array(b))). Second method directly from python list as: print(np.linalg.norm(np.subtract(a,b)))

知识问答

在pandas / python中的数据框中合并两列文本

2021年7月25日 Python实用宝典

问题：在pandas / python中的数据框中合并两列文本

我在Python中使用熊猫有20 x 4000数据框。其中两列分别命名为Year和quarter。我想创建一个名为periodmake Year = 2000和quarter= q2into 的变量2000q2。

有人可以帮忙吗？

I have a 20 x 4000 dataframe in Python using pandas. Two of these columns are named Year and quarter. I’d like to create a variable called period that makes Year = 2000 and quarter= q2 into 2000q2.

Can anyone help with that?

回答 0

如果两个列都是字符串，则可以直接将它们连接起来：

df["period"] = df["Year"] + df["quarter"]

如果其中一列（或两列）均未输入字符串，则应首先将其转换为字符串，

df["period"] = df["Year"].astype(str) + df["quarter"]

这样做时要小心NaN！

如果需要连接多个字符串列，则可以使用agg：

df['period'] = df[['Year', 'quarter', ...]].agg('-'.join, axis=1)

其中“-”是分隔符。

if both columns are strings, you can concatenate them directly:

df["period"] = df["Year"] + df["quarter"]

If one (or both) of the columns are not string typed, you should convert it (them) first,

df["period"] = df["Year"].astype(str) + df["quarter"]

Beware of NaNs when doing this!

If you need to join multiple string columns, you can use agg:

df['period'] = df[['Year', 'quarter', ...]].agg('-'.join, axis=1)

Where “-” is the separator.

回答 1

df = pd.DataFrame({'Year': ['2014', '2015'], 'quarter': ['q1', 'q2']})
df['period'] = df[['Year', 'quarter']].apply(lambda x: ''.join(x), axis=1)

产生此数据框

   Year quarter  period
0  2014      q1  2014q1
1  2015      q2  2015q2

此方法通过替换df[['Year', 'quarter']]为数据框的任何列切片（例如）将其推广为任意数量的字符串列df.iloc[:,0:2].apply(lambda x: ''.join(x), axis=1)。

您可以在此处查看有关apply（）方法的更多信息

df = pd.DataFrame({'Year': ['2014', '2015'], 'quarter': ['q1', 'q2']})
df['period'] = df[['Year', 'quarter']].apply(lambda x: ''.join(x), axis=1)

Yields this dataframe

   Year quarter  period
0  2014      q1  2014q1
1  2015      q2  2015q2

This method generalizes to an arbitrary number of string columns by replacing df[['Year', 'quarter']] with any column slice of your dataframe, e.g. df.iloc[:,0:2].apply(lambda x: ''.join(x), axis=1).

You can check more information about apply() method here

回答 2

小型数据集（<150行）

[''.join(i) for i in zip(df["Year"].map(str),df["quarter"])]

或稍慢但更紧凑：

df.Year.str.cat(df.quarter)

更大的数据集（> 150行）

df['Year'].astype(str) + df['quarter']

更新：时序图熊猫0.23.4

让我们在200K行DF上进行测试：

In [250]: df
Out[250]:
   Year quarter
0  2014      q1
1  2015      q2

In [251]: df = pd.concat([df] * 10**5)

In [252]: df.shape
Out[252]: (200000, 2)

更新：使用Pandas 0.19.0的新计时

定时不CPU / GPU优化（从排序最快到最慢）：

In [107]: %timeit df['Year'].astype(str) + df['quarter']
10 loops, best of 3: 131 ms per loop

In [106]: %timeit df['Year'].map(str) + df['quarter']
10 loops, best of 3: 161 ms per loop

In [108]: %timeit df.Year.str.cat(df.quarter)
10 loops, best of 3: 189 ms per loop

In [109]: %timeit df.loc[:, ['Year','quarter']].astype(str).sum(axis=1)
1 loop, best of 3: 567 ms per loop

In [110]: %timeit df[['Year','quarter']].astype(str).sum(axis=1)
1 loop, best of 3: 584 ms per loop

In [111]: %timeit df[['Year','quarter']].apply(lambda x : '{}{}'.format(x[0],x[1]), axis=1)
1 loop, best of 3: 24.7 s per loop

时序采用CPU / GPU优化：

In [113]: %timeit df['Year'].astype(str) + df['quarter']
10 loops, best of 3: 53.3 ms per loop

In [114]: %timeit df['Year'].map(str) + df['quarter']
10 loops, best of 3: 65.5 ms per loop

In [115]: %timeit df.Year.str.cat(df.quarter)
10 loops, best of 3: 79.9 ms per loop

In [116]: %timeit df.loc[:, ['Year','quarter']].astype(str).sum(axis=1)
1 loop, best of 3: 230 ms per loop

In [117]: %timeit df[['Year','quarter']].astype(str).sum(axis=1)
1 loop, best of 3: 230 ms per loop

In [118]: %timeit df[['Year','quarter']].apply(lambda x : '{}{}'.format(x[0],x[1]), axis=1)
1 loop, best of 3: 9.38 s per loop

回答@ anton-vbr的贡献

Small data-sets (< 150rows)

[''.join(i) for i in zip(df["Year"].map(str),df["quarter"])]

or slightly slower but more compact:

df.Year.str.cat(df.quarter)

Larger data sets (> 150rows)

df['Year'].astype(str) + df['quarter']

UPDATE: Timing graph Pandas 0.23.4

Let’s test it on 200K rows DF:

In [250]: df
Out[250]:
   Year quarter
0  2014      q1
1  2015      q2

In [251]: df = pd.concat([df] * 10**5)

In [252]: df.shape
Out[252]: (200000, 2)

UPDATE: new timings using Pandas 0.19.0

Timing without CPU/GPU optimization (sorted from fastest to slowest):

In [107]: %timeit df['Year'].astype(str) + df['quarter']
10 loops, best of 3: 131 ms per loop

In [106]: %timeit df['Year'].map(str) + df['quarter']
10 loops, best of 3: 161 ms per loop

In [108]: %timeit df.Year.str.cat(df.quarter)
10 loops, best of 3: 189 ms per loop

In [109]: %timeit df.loc[:, ['Year','quarter']].astype(str).sum(axis=1)
1 loop, best of 3: 567 ms per loop

In [110]: %timeit df[['Year','quarter']].astype(str).sum(axis=1)
1 loop, best of 3: 584 ms per loop

In [111]: %timeit df[['Year','quarter']].apply(lambda x : '{}{}'.format(x[0],x[1]), axis=1)
1 loop, best of 3: 24.7 s per loop

Timing using CPU/GPU optimization:

In [113]: %timeit df['Year'].astype(str) + df['quarter']
10 loops, best of 3: 53.3 ms per loop

In [114]: %timeit df['Year'].map(str) + df['quarter']
10 loops, best of 3: 65.5 ms per loop

In [115]: %timeit df.Year.str.cat(df.quarter)
10 loops, best of 3: 79.9 ms per loop

In [116]: %timeit df.loc[:, ['Year','quarter']].astype(str).sum(axis=1)
1 loop, best of 3: 230 ms per loop

In [117]: %timeit df[['Year','quarter']].astype(str).sum(axis=1)
1 loop, best of 3: 230 ms per loop

In [118]: %timeit df[['Year','quarter']].apply(lambda x : '{}{}'.format(x[0],x[1]), axis=1)
1 loop, best of 3: 9.38 s per loop

Answer contribution by @anton-vbr

回答 3

该方法cat()的的.str访问可以很好地表现这一点：

>>> import pandas as pd
>>> df = pd.DataFrame([["2014", "q1"], 
...                    ["2015", "q3"]],
...                   columns=('Year', 'Quarter'))
>>> print(df)
   Year Quarter
0  2014      q1
1  2015      q3
>>> df['Period'] = df.Year.str.cat(df.Quarter)
>>> print(df)
   Year Quarter  Period
0  2014      q1  2014q1
1  2015      q3  2015q3

cat() 甚至允许您添加分隔符，因此，例如，假设年份和期间只有整数，则可以执行以下操作：

>>> import pandas as pd
>>> df = pd.DataFrame([[2014, 1],
...                    [2015, 3]],
...                   columns=('Year', 'Quarter'))
>>> print(df)
   Year Quarter
0  2014       1
1  2015       3
>>> df['Period'] = df.Year.astype(str).str.cat(df.Quarter.astype(str), sep='q')
>>> print(df)
   Year Quarter  Period
0  2014       1  2014q1
1  2015       3  2015q3

连接多列只是传递一系列列表或包含除第一列之外的所有列的数据框作为要str.cat()在第一列（系列）上调用的参数的问题：

>>> df = pd.DataFrame(
...     [['USA', 'Nevada', 'Las Vegas'],
...      ['Brazil', 'Pernambuco', 'Recife']],
...     columns=['Country', 'State', 'City'],
... )
>>> df['AllTogether'] = df['Country'].str.cat(df[['State', 'City']], sep=' - ')
>>> print(df)
  Country       State       City                   AllTogether
0     USA      Nevada  Las Vegas      USA - Nevada - Las Vegas
1  Brazil  Pernambuco     Recife  Brazil - Pernambuco - Recife

请注意，如果您的pandas数据框/系列具有空值，则需要包括参数na_rep以用字符串替换NaN值，否则合并的列将默认为NaN。

The method cat() of the .str accessor works really well for this:

>>> import pandas as pd
>>> df = pd.DataFrame([["2014", "q1"], 
...                    ["2015", "q3"]],
...                   columns=('Year', 'Quarter'))
>>> print(df)
   Year Quarter
0  2014      q1
1  2015      q3
>>> df['Period'] = df.Year.str.cat(df.Quarter)
>>> print(df)
   Year Quarter  Period
0  2014      q1  2014q1
1  2015      q3  2015q3

cat() even allows you to add a separator so, for example, suppose you only have integers for year and period, you can do this:

>>> import pandas as pd
>>> df = pd.DataFrame([[2014, 1],
...                    [2015, 3]],
...                   columns=('Year', 'Quarter'))
>>> print(df)
   Year Quarter
0  2014       1
1  2015       3
>>> df['Period'] = df.Year.astype(str).str.cat(df.Quarter.astype(str), sep='q')
>>> print(df)
   Year Quarter  Period
0  2014       1  2014q1
1  2015       3  2015q3

Joining multiple columns is just a matter of passing either a list of series or a dataframe containing all but the first column as a parameter to str.cat() invoked on the first column (Series):

>>> df = pd.DataFrame(
...     [['USA', 'Nevada', 'Las Vegas'],
...      ['Brazil', 'Pernambuco', 'Recife']],
...     columns=['Country', 'State', 'City'],
... )
>>> df['AllTogether'] = df['Country'].str.cat(df[['State', 'City']], sep=' - ')
>>> print(df)
  Country       State       City                   AllTogether
0     USA      Nevada  Las Vegas      USA - Nevada - Las Vegas
1  Brazil  Pernambuco     Recife  Brazil - Pernambuco - Recife

Do note that if your pandas dataframe/series has null values, you need to include the parameter na_rep to replace the NaN values with a string, otherwise the combined column will default to NaN.

回答 4

这次通过string.format（）使用lamba函数。

import pandas as pd
df = pd.DataFrame({'Year': ['2014', '2015'], 'Quarter': ['q1', 'q2']})
print df
df['YearQuarter'] = df[['Year','Quarter']].apply(lambda x : '{}{}'.format(x[0],x[1]), axis=1)
print df

  Quarter  Year
0      q1  2014
1      q2  2015
  Quarter  Year YearQuarter
0      q1  2014      2014q1
1      q2  2015      2015q2

这使您可以根据需要使用非字符串并重新格式化值。

import pandas as pd
df = pd.DataFrame({'Year': ['2014', '2015'], 'Quarter': [1, 2]})
print df.dtypes
print df

df['YearQuarter'] = df[['Year','Quarter']].apply(lambda x : '{}q{}'.format(x[0],x[1]), axis=1)
print df

Quarter     int64
Year       object
dtype: object
   Quarter  Year
0        1  2014
1        2  2015
   Quarter  Year YearQuarter
0        1  2014      2014q1
1        2  2015      2015q2

Use of a lamba function this time with string.format().

import pandas as pd
df = pd.DataFrame({'Year': ['2014', '2015'], 'Quarter': ['q1', 'q2']})
print df
df['YearQuarter'] = df[['Year','Quarter']].apply(lambda x : '{}{}'.format(x[0],x[1]), axis=1)
print df

  Quarter  Year
0      q1  2014
1      q2  2015
  Quarter  Year YearQuarter
0      q1  2014      2014q1
1      q2  2015      2015q2

This allows you to work with non-strings and reformat values as needed.

import pandas as pd
df = pd.DataFrame({'Year': ['2014', '2015'], 'Quarter': [1, 2]})
print df.dtypes
print df

df['YearQuarter'] = df[['Year','Quarter']].apply(lambda x : '{}q{}'.format(x[0],x[1]), axis=1)
print df

Quarter     int64
Year       object
dtype: object
   Quarter  Year
0        1  2014
1        2  2015
   Quarter  Year YearQuarter
0        1  2014      2014q1
1        2  2015      2015q2

回答 5

您问题的简单答案。

    year    quarter
0   2000    q1
1   2000    q2

> df['year_quarter'] = df['year'] + '' + df['quarter']

> print(df['year_quarter'])
  2000q1
  2000q2

Simple answer for your question.

    year    quarter
0   2000    q1
1   2000    q2

> df['year_quarter'] = df['year'] + '' + df['quarter']

> print(df['year_quarter'])
  2000q1
  2000q2

回答 6

虽然@silvado的答案很好，但如果更改df.map(str)为df.astype(str)它会更快：

import pandas as pd
df = pd.DataFrame({'Year': ['2014', '2015'], 'quarter': ['q1', 'q2']})

In [131]: %timeit df["Year"].map(str)
10000 loops, best of 3: 132 us per loop

In [132]: %timeit df["Year"].astype(str)
10000 loops, best of 3: 82.2 us per loop

Although the @silvado answer is good if you change df.map(str) to df.astype(str) it will be faster:

import pandas as pd
df = pd.DataFrame({'Year': ['2014', '2015'], 'quarter': ['q1', 'q2']})

In [131]: %timeit df["Year"].map(str)
10000 loops, best of 3: 132 us per loop

In [132]: %timeit df["Year"].astype(str)
10000 loops, best of 3: 82.2 us per loop

回答 7

让我们假设您 dataframe的df列Year和为Quarter。

import pandas as pd
df = pd.DataFrame({'Quarter':'q1 q2 q3 q4'.split(), 'Year':'2000'})

假设我们要看数据框；

df
>>>  Quarter    Year
   0    q1      2000
   1    q2      2000
   2    q3      2000
   3    q4      2000

最后，将Year和连接Quarter如下。

df['Period'] = df['Year'] + ' ' + df['Quarter']

现在print df ，您可以查看生成的数据框。

df
>>>  Quarter    Year    Period
    0   q1      2000    2000 q1
    1   q2      2000    2000 q2
    2   q3      2000    2000 q3
    3   q4      2000    2000 q4

如果您不想在年份和季度之间留出空间，只需将其删除即可；

df['Period'] = df['Year'] + df['Quarter']

Let us suppose your dataframe is df with columns Year and Quarter.

import pandas as pd
df = pd.DataFrame({'Quarter':'q1 q2 q3 q4'.split(), 'Year':'2000'})

Suppose we want to see the dataframe;

df
>>>  Quarter    Year
   0    q1      2000
   1    q2      2000
   2    q3      2000
   3    q4      2000

Finally, concatenate the Year and the Quarter as follows.

df['Period'] = df['Year'] + ' ' + df['Quarter']

You can now print df to see the resulting dataframe.

df
>>>  Quarter    Year    Period
    0   q1      2000    2000 q1
    1   q2      2000    2000 q2
    2   q3      2000    2000 q3
    3   q4      2000    2000 q4

If you do not want the space between the year and quarter, simply remove it by doing;

df['Period'] = df['Year'] + df['Quarter']

回答 8

这是我发现非常通用的实现：

In [1]: import pandas as pd 

In [2]: df = pd.DataFrame([[0, 'the', 'quick', 'brown'],
   ...:                    [1, 'fox', 'jumps', 'over'], 
   ...:                    [2, 'the', 'lazy', 'dog']],
   ...:                   columns=['c0', 'c1', 'c2', 'c3'])

In [3]: def str_join(df, sep, *cols):
   ...:     from functools import reduce
   ...:     return reduce(lambda x, y: x.astype(str).str.cat(y.astype(str), sep=sep), 
   ...:                   [df[col] for col in cols])
   ...: 

In [4]: df['cat'] = str_join(df, '-', 'c0', 'c1', 'c2', 'c3')

In [5]: df
Out[5]: 
   c0   c1     c2     c3                cat
0   0  the  quick  brown  0-the-quick-brown
1   1  fox  jumps   over   1-fox-jumps-over
2   2  the   lazy    dog     2-the-lazy-dog

Here is an implementation that I find very versatile:

In [1]: import pandas as pd 

In [2]: df = pd.DataFrame([[0, 'the', 'quick', 'brown'],
   ...:                    [1, 'fox', 'jumps', 'over'], 
   ...:                    [2, 'the', 'lazy', 'dog']],
   ...:                   columns=['c0', 'c1', 'c2', 'c3'])

In [3]: def str_join(df, sep, *cols):
   ...:     from functools import reduce
   ...:     return reduce(lambda x, y: x.astype(str).str.cat(y.astype(str), sep=sep), 
   ...:                   [df[col] for col in cols])
   ...: 

In [4]: df['cat'] = str_join(df, '-', 'c0', 'c1', 'c2', 'c3')

In [5]: df
Out[5]: 
   c0   c1     c2     c3                cat
0   0  the  quick  brown  0-the-quick-brown
1   1  fox  jumps   over   1-fox-jumps-over
2   2  the   lazy    dog     2-the-lazy-dog

回答 9

将数据插入数据框时，此命令应该可以解决您的问题：

df['period'] = df[['Year', 'quarter']].apply(lambda x: ' '.join(x.astype(str)), axis=1)

As your data are inserted to a dataframe, this command should solve your problem:

df['period'] = df[['Year', 'quarter']].apply(lambda x: ' '.join(x.astype(str)), axis=1)

回答 10

更有效的是

def concat_df_str1(df):
    """ run time: 1.3416s """
    return pd.Series([''.join(row.astype(str)) for row in df.values], index=df.index)

这是一个时间测试：

import numpy as np
import pandas as pd

from time import time


def concat_df_str1(df):
    """ run time: 1.3416s """
    return pd.Series([''.join(row.astype(str)) for row in df.values], index=df.index)


def concat_df_str2(df):
    """ run time: 5.2758s """
    return df.astype(str).sum(axis=1)


def concat_df_str3(df):
    """ run time: 5.0076s """
    df = df.astype(str)
    return df[0] + df[1] + df[2] + df[3] + df[4] + \
           df[5] + df[6] + df[7] + df[8] + df[9]


def concat_df_str4(df):
    """ run time: 7.8624s """
    return df.astype(str).apply(lambda x: ''.join(x), axis=1)


def main():
    df = pd.DataFrame(np.zeros(1000000).reshape(100000, 10))
    df = df.astype(int)

    time1 = time()
    df_en = concat_df_str4(df)
    print('run time: %.4fs' % (time() - time1))
    print(df_en.head(10))


if __name__ == '__main__':
    main()

最后，当使用sum（concat_df_str2）时，结果不是简单的concat，它将转换为整数。

more efficient is

def concat_df_str1(df):
    """ run time: 1.3416s """
    return pd.Series([''.join(row.astype(str)) for row in df.values], index=df.index)

and here is a time test:

import numpy as np
import pandas as pd

from time import time


def concat_df_str1(df):
    """ run time: 1.3416s """
    return pd.Series([''.join(row.astype(str)) for row in df.values], index=df.index)


def concat_df_str2(df):
    """ run time: 5.2758s """
    return df.astype(str).sum(axis=1)


def concat_df_str3(df):
    """ run time: 5.0076s """
    df = df.astype(str)
    return df[0] + df[1] + df[2] + df[3] + df[4] + \
           df[5] + df[6] + df[7] + df[8] + df[9]


def concat_df_str4(df):
    """ run time: 7.8624s """
    return df.astype(str).apply(lambda x: ''.join(x), axis=1)


def main():
    df = pd.DataFrame(np.zeros(1000000).reshape(100000, 10))
    df = df.astype(int)

    time1 = time()
    df_en = concat_df_str4(df)
    print('run time: %.4fs' % (time() - time1))
    print(df_en.head(10))


if __name__ == '__main__':
    main()

final, when sum(concat_df_str2) is used, the result is not simply concat, it will trans to integer.

回答 11

归纳为多列，为什么不这样做：

columns = ['whatever', 'columns', 'you', 'choose']
df['period'] = df[columns].astype(str).sum(axis=1)

generalising to multiple columns, why not:

columns = ['whatever', 'columns', 'you', 'choose']
df['period'] = df[columns].astype(str).sum(axis=1)

回答 12

使用zip甚至可以更快：

df["period"] = [''.join(i) for i in zip(df["Year"].map(str),df["quarter"])]

图形：

import pandas as pd
import numpy as np
import timeit
import matplotlib.pyplot as plt
from collections import defaultdict

df = pd.DataFrame({'Year': ['2014', '2015'], 'quarter': ['q1', 'q2']})

myfuncs = {
"df['Year'].astype(str) + df['quarter']":
    lambda: df['Year'].astype(str) + df['quarter'],
"df['Year'].map(str) + df['quarter']":
    lambda: df['Year'].map(str) + df['quarter'],
"df.Year.str.cat(df.quarter)":
    lambda: df.Year.str.cat(df.quarter),
"df.loc[:, ['Year','quarter']].astype(str).sum(axis=1)":
    lambda: df.loc[:, ['Year','quarter']].astype(str).sum(axis=1),
"df[['Year','quarter']].astype(str).sum(axis=1)":
    lambda: df[['Year','quarter']].astype(str).sum(axis=1),
    "df[['Year','quarter']].apply(lambda x : '{}{}'.format(x[0],x[1]), axis=1)":
    lambda: df[['Year','quarter']].apply(lambda x : '{}{}'.format(x[0],x[1]), axis=1),
    "[''.join(i) for i in zip(dataframe['Year'].map(str),dataframe['quarter'])]":
    lambda: [''.join(i) for i in zip(df["Year"].map(str),df["quarter"])]
}

d = defaultdict(dict)
step = 10
cont = True
while cont:
    lendf = len(df); print(lendf)
    for k,v in myfuncs.items():
        iters = 1
        t = 0
        while t < 0.2:
            ts = timeit.repeat(v, number=iters, repeat=3)
            t = min(ts)
            iters *= 10
        d[k][lendf] = t/iters
        if t > 2: cont = False
    df = pd.concat([df]*step)

pd.DataFrame(d).plot().legend(loc='upper center', bbox_to_anchor=(0.5, -0.15))
plt.yscale('log'); plt.xscale('log'); plt.ylabel('seconds'); plt.xlabel('df rows')
plt.show()

Using zip could be even quicker:

df["period"] = [''.join(i) for i in zip(df["Year"].map(str),df["quarter"])]

Graph:

import pandas as pd
import numpy as np
import timeit
import matplotlib.pyplot as plt
from collections import defaultdict

df = pd.DataFrame({'Year': ['2014', '2015'], 'quarter': ['q1', 'q2']})

myfuncs = {
"df['Year'].astype(str) + df['quarter']":
    lambda: df['Year'].astype(str) + df['quarter'],
"df['Year'].map(str) + df['quarter']":
    lambda: df['Year'].map(str) + df['quarter'],
"df.Year.str.cat(df.quarter)":
    lambda: df.Year.str.cat(df.quarter),
"df.loc[:, ['Year','quarter']].astype(str).sum(axis=1)":
    lambda: df.loc[:, ['Year','quarter']].astype(str).sum(axis=1),
"df[['Year','quarter']].astype(str).sum(axis=1)":
    lambda: df[['Year','quarter']].astype(str).sum(axis=1),
    "df[['Year','quarter']].apply(lambda x : '{}{}'.format(x[0],x[1]), axis=1)":
    lambda: df[['Year','quarter']].apply(lambda x : '{}{}'.format(x[0],x[1]), axis=1),
    "[''.join(i) for i in zip(dataframe['Year'].map(str),dataframe['quarter'])]":
    lambda: [''.join(i) for i in zip(df["Year"].map(str),df["quarter"])]
}

d = defaultdict(dict)
step = 10
cont = True
while cont:
    lendf = len(df); print(lendf)
    for k,v in myfuncs.items():
        iters = 1
        t = 0
        while t < 0.2:
            ts = timeit.repeat(v, number=iters, repeat=3)
            t = min(ts)
            iters *= 10
        d[k][lendf] = t/iters
        if t > 2: cont = False
    df = pd.concat([df]*step)

pd.DataFrame(d).plot().legend(loc='upper center', bbox_to_anchor=(0.5, -0.15))
plt.yscale('log'); plt.xscale('log'); plt.ylabel('seconds'); plt.xlabel('df rows')
plt.show()

回答 13

最简单的解决方案：

通用解决方案

df['combined_col'] = df[['col1', 'col2']].astype(str).apply('-'.join, axis=1)

特定问题的解决方案

df['quarter_year'] = df[['quarter', 'year']].astype(str).apply(''.join, axis=1)

在.join之前的引号内指定首选的分隔符

Simplest Solution:

Generic Solution

df['combined_col'] = df[['col1', 'col2']].astype(str).apply('-'.join, axis=1)

Question specific solution

df['quarter_year'] = df[['quarter', 'year']].astype(str).apply(''.join, axis=1)

Specify the preferred delimiter inside the quotes before .join

回答 14

此解决方案使用中间步骤将DataFrame的两列压缩为包含值列表的单列。这不仅适用于字符串，而且适用于所有类型的column-dtypes

import pandas as pd
df = pd.DataFrame({'Year': ['2014', '2015'], 'quarter': ['q1', 'q2']})
df['list']=df[['Year','quarter']].values.tolist()
df['period']=df['list'].apply(''.join)
print(df)

结果：

   Year quarter        list  period
0  2014      q1  [2014, q1]  2014q1
1  2015      q2  [2015, q2]  2015q2

This solution uses an intermediate step compressing two columns of the DataFrame to a single column containing a list of the values. This works not only for strings but for all kind of column-dtypes

import pandas as pd
df = pd.DataFrame({'Year': ['2014', '2015'], 'quarter': ['q1', 'q2']})
df['list']=df[['Year','quarter']].values.tolist()
df['period']=df['list'].apply(''.join)
print(df)

Result:

   Year quarter        list  period
0  2014      q1  [2014, q1]  2014q1
1  2015      q2  [2015, q2]  2015q2

回答 15

如前所述，您必须将每一列转换为字符串，然后使用加号运算符组合两个字符串列。使用NumPy可以大大提高性能。

%timeit df['Year'].values.astype(str) + df.quarter
71.1 ms ± 3.76 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

%timeit df['Year'].astype(str) + df['quarter']
565 ms ± 22.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

As many have mentioned previously, you must convert each column to string and then use the plus operator to combine two string columns. You can get a large performance improvement by using NumPy.

%timeit df['Year'].values.astype(str) + df.quarter
71.1 ms ± 3.76 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

%timeit df['Year'].astype(str) + df['quarter']
565 ms ± 22.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

回答 16

我认为在pandas中组合列的最好方法是将两个列都转换为整数，然后转换为str。

df[['Year', 'quarter']] = df[['Year', 'quarter']].astype(int).astype(str)
df['Period']= df['Year'] + 'q' + df['quarter']

I think the best way to combine the columns in pandas is by converting both the columns to integer and then to str.

df[['Year', 'quarter']] = df[['Year', 'quarter']].astype(int).astype(str)
df['Period']= df['Year'] + 'q' + df['quarter']

回答 17

这是我上面的解决方案的摘要，该方法使用列值之间的分隔符将具有int和str值的两列连接/合并为新列。为此有三种解决方案。

# be cautious about the separator, some symbols may cause "SyntaxError: EOL while scanning string literal".
# e.g. ";;" as separator would raise the SyntaxError

separator = "&&" 

# pd.Series.str.cat() method does not work to concatenate / combine two columns with int value and str value. This would raise "AttributeError: Can only use .cat accessor with a 'category' dtype"

df["period"] = df["Year"].map(str) + separator + df["quarter"]
df["period"] = df[['Year','quarter']].apply(lambda x : '{} && {}'.format(x[0],x[1]), axis=1)
df["period"] = df.apply(lambda x: f'{x["Year"]} && {x["quarter"]}', axis=1)

Here is my summary of the above solutions to concatenate / combine two columns with int and str value into a new column, using a separator between the values of columns. Three solutions work for this purpose.

# be cautious about the separator, some symbols may cause "SyntaxError: EOL while scanning string literal".
# e.g. ";;" as separator would raise the SyntaxError

separator = "&&" 

# pd.Series.str.cat() method does not work to concatenate / combine two columns with int value and str value. This would raise "AttributeError: Can only use .cat accessor with a 'category' dtype"

df["period"] = df["Year"].map(str) + separator + df["quarter"]
df["period"] = df[['Year','quarter']].apply(lambda x : '{} && {}'.format(x[0],x[1]), axis=1)
df["period"] = df.apply(lambda x: f'{x["Year"]} && {x["quarter"]}', axis=1)

回答 18

使用.combine_first。

df['Period'] = df['Year'].combine_first(df['Quarter'])

Use .combine_first.

df['Period'] = df['Year'].combine_first(df['Quarter'])

回答 19

def madd(x):
    """Performs element-wise string concatenation with multiple input arrays.

    Args:
        x: iterable of np.array.

    Returns: np.array.
    """
    for i, arr in enumerate(x):
        if type(arr.item(0)) is not str:
            x[i] = x[i].astype(str)
    return reduce(np.core.defchararray.add, x)

例如：

data = list(zip([2000]*4, ['q1', 'q2', 'q3', 'q4']))
df = pd.DataFrame(data=data, columns=['Year', 'quarter'])
df['period'] = madd([df[col].values for col in ['Year', 'quarter']])

df

    Year    quarter period
0   2000    q1  2000q1
1   2000    q2  2000q2
2   2000    q3  2000q3
3   2000    q4  2000q4

def madd(x):
    """Performs element-wise string concatenation with multiple input arrays.

    Args:
        x: iterable of np.array.

    Returns: np.array.
    """
    for i, arr in enumerate(x):
        if type(arr.item(0)) is not str:
            x[i] = x[i].astype(str)
    return reduce(np.core.defchararray.add, x)

For example:

data = list(zip([2000]*4, ['q1', 'q2', 'q3', 'q4']))
df = pd.DataFrame(data=data, columns=['Year', 'quarter'])
df['period'] = madd([df[col].values for col in ['Year', 'quarter']])

df

    Year    quarter period
0   2000    q1  2000q1
1   2000    q2  2000q2
2   2000    q3  2000q3
3   2000    q4  2000q4

回答 20

一个可以使用DataFrame的分配方法：

df= (pd.DataFrame({'Year': ['2014', '2015'], 'quarter': ['q1', 'q2']}).
  assign(period=lambda x: x.Year+x.quarter ))

One can use assign method of DataFrame:

df= (pd.DataFrame({'Year': ['2014', '2015'], 'quarter': ['q1', 'q2']}).
  assign(period=lambda x: x.Year+x.quarter ))

回答 21

dataframe["period"] = dataframe["Year"].astype(str).add(dataframe["quarter"])

或者如果值类似于[2000] [4]并想要设为[2000q4]

dataframe["period"] = dataframe["Year"].astype(str).add('q').add(dataframe["quarter"]).astype(str)

也.astype(str)用.map(str)作品代替。

dataframe["period"] = dataframe["Year"].astype(str).add(dataframe["quarter"])

or if values are like [2000] [4] and want to make [2000q4]

dataframe["period"] = dataframe["Year"].astype(str).add('q').add(dataframe["quarter"]).astype(str)

substituting .astype(str) with .map(str) works too.

知识问答

如何获取NumPy数组中N个最大值的索引？

2021年7月25日 Python实用宝典

问题：如何获取NumPy数组中N个最大值的索引？

NumPy提出了一种通过来获取数组最大值的索引的方法np.argmax。

我想要类似的事情，但是返回N最大值的索引。

例如，如果我有一个数组，[1, 3, 2, 4, 5]，function(array, n=3)将返回的索引[4, 3, 1]相对应的元素[5, 4, 3]。

NumPy proposes a way to get the index of the maximum value of an array via np.argmax.

I would like a similar thing, but returning the indexes of the N maximum values.

For instance, if I have an array, [1, 3, 2, 4, 5], function(array, n=3) would return the indices [4, 3, 1] which correspond to the elements [5, 4, 3].

回答 0

我想出的最简单的方法是：

In [1]: import numpy as np

In [2]: arr = np.array([1, 3, 2, 4, 5])

In [3]: arr.argsort()[-3:][::-1]
Out[3]: array([4, 3, 1])

这涉及数组的完整排序。我想知道是否numpy提供了一种进行部分排序的内置方法。到目前为止，我还没有找到一个。

如果这种解决方案太慢（尤其是对于小型解决方案n），则可能值得在Cython中编写代码。

The simplest I’ve been able to come up with is:

In [1]: import numpy as np

In [2]: arr = np.array([1, 3, 2, 4, 5])

In [3]: arr.argsort()[-3:][::-1]
Out[3]: array([4, 3, 1])

This involves a complete sort of the array. I wonder if numpy provides a built-in way to do a partial sort; so far I haven’t been able to find one.

If this solution turns out to be too slow (especially for small n), it may be worth looking at coding something up in Cython.

回答 1

较新的NumPy版本（1.8及更高版本）具有argpartition为此要求的功能。要获取四个最大元素的索引，请执行

>>> a = np.array([9, 4, 4, 3, 3, 9, 0, 4, 6, 0])
>>> a
array([9, 4, 4, 3, 3, 9, 0, 4, 6, 0])
>>> ind = np.argpartition(a, -4)[-4:]
>>> ind
array([1, 5, 8, 0])
>>> a[ind]
array([4, 9, 6, 9])

与之不同的是argsort，此函数在最坏的情况下以线性时间运行，但是返回的索引未排序，从评估结果可以看出a[ind]。如果您也需要它，请对它们进行排序：

>>> ind[np.argsort(a[ind])]
array([1, 8, 5, 0])

要以这种方式获得排序前k个元素，需要O（n + k log k）时间。

Newer NumPy versions (1.8 and up) have a function called argpartition for this. To get the indices of the four largest elements, do

>>> a = np.array([9, 4, 4, 3, 3, 9, 0, 4, 6, 0])
>>> a
array([9, 4, 4, 3, 3, 9, 0, 4, 6, 0])
>>> ind = np.argpartition(a, -4)[-4:]
>>> ind
array([1, 5, 8, 0])
>>> a[ind]
array([4, 9, 6, 9])

Unlike argsort, this function runs in linear time in the worst case, but the returned indices are not sorted, as can be seen from the result of evaluating a[ind]. If you need that too, sort them afterwards:

>>> ind[np.argsort(a[ind])]
array([1, 8, 5, 0])

To get the top-k elements in sorted order in this way takes O(n + k log k) time.

回答 2

更简单了：

idx = (-arr).argsort()[:n]

其中，n是最大值的数量。

Simpler yet:

idx = (-arr).argsort()[:n]

where n is the number of maximum values.

回答 3

采用：

>>> import heapq
>>> import numpy
>>> a = numpy.array([1, 3, 2, 4, 5])
>>> heapq.nlargest(3, range(len(a)), a.take)
[4, 3, 1]

对于常规的Python列表：

>>> a = [1, 3, 2, 4, 5]
>>> heapq.nlargest(3, range(len(a)), a.__getitem__)
[4, 3, 1]

如果您使用Python 2，请使用xrange代替range。

来源：heapq —堆队列算法

Use:

>>> import heapq
>>> import numpy
>>> a = numpy.array([1, 3, 2, 4, 5])
>>> heapq.nlargest(3, range(len(a)), a.take)
[4, 3, 1]

For regular Python lists:

>>> a = [1, 3, 2, 4, 5]
>>> heapq.nlargest(3, range(len(a)), a.__getitem__)
[4, 3, 1]

If you use Python 2, use xrange instead of range.

Source: heapq — Heap queue algorithm

回答 4

如果碰巧正在使用多维数组，则需要展平和分解索引：

def largest_indices(ary, n):
    """Returns the n largest indices from a numpy array."""
    flat = ary.flatten()
    indices = np.argpartition(flat, -n)[-n:]
    indices = indices[np.argsort(-flat[indices])]
    return np.unravel_index(indices, ary.shape)

例如：

>>> xs = np.sin(np.arange(9)).reshape((3, 3))
>>> xs
array([[ 0.        ,  0.84147098,  0.90929743],
       [ 0.14112001, -0.7568025 , -0.95892427],
       [-0.2794155 ,  0.6569866 ,  0.98935825]])
>>> largest_indices(xs, 3)
(array([2, 0, 0]), array([2, 2, 1]))
>>> xs[largest_indices(xs, 3)]
array([ 0.98935825,  0.90929743,  0.84147098])

If you happen to be working with a multidimensional array then you’ll need to flatten and unravel the indices:

def largest_indices(ary, n):
    """Returns the n largest indices from a numpy array."""
    flat = ary.flatten()
    indices = np.argpartition(flat, -n)[-n:]
    indices = indices[np.argsort(-flat[indices])]
    return np.unravel_index(indices, ary.shape)

For example:

>>> xs = np.sin(np.arange(9)).reshape((3, 3))
>>> xs
array([[ 0.        ,  0.84147098,  0.90929743],
       [ 0.14112001, -0.7568025 , -0.95892427],
       [-0.2794155 ,  0.6569866 ,  0.98935825]])
>>> largest_indices(xs, 3)
(array([2, 0, 0]), array([2, 2, 1]))
>>> xs[largest_indices(xs, 3)]
array([ 0.98935825,  0.90929743,  0.84147098])

回答 5

如果您不在乎可以使用的第K个最大元素的顺序，则argpartition它们的性能应比完整排序要好argsort。

K = 4 # We want the indices of the four largest values
a = np.array([0, 8, 0, 4, 5, 8, 8, 0, 4, 2])
np.argpartition(a,-K)[-K:]
array([4, 1, 5, 6])

学分到这个问题。

我进行了一些测试，随着数组的大小和K值的增加，它的argpartition表现似乎都胜过argsort了。

If you don’t care about the order of the K-th largest elements you can use argpartition, which should perform better than a full sort through argsort.

K = 4 # We want the indices of the four largest values
a = np.array([0, 8, 0, 4, 5, 8, 8, 0, 4, 2])
np.argpartition(a,-K)[-K:]
array([4, 1, 5, 6])

Credits go to this question.

I ran a few tests and it looks like argpartition outperforms argsort as the size of the array and the value of K increase.

回答 6

对于多维数组，可以使用axis关键字以沿期望的轴应用分区。

# For a 2D array
indices = np.argpartition(arr, -N, axis=1)[:, -N:]

对于抓取物品：

x = arr.shape[0]
arr[np.repeat(np.arange(x), N), indices.ravel()].reshape(x, N)

但是请注意，这不会返回排序结果。在这种情况下，您可以np.argsort()沿预期的轴使用：

indices = np.argsort(arr, axis=1)[:, -N:]

# Result
x = arr.shape[0]
arr[np.repeat(np.arange(x), N), indices.ravel()].reshape(x, N)

这是一个例子：

In [42]: a = np.random.randint(0, 20, (10, 10))

In [44]: a
Out[44]:
array([[ 7, 11, 12,  0,  2,  3,  4, 10,  6, 10],
       [16, 16,  4,  3, 18,  5, 10,  4, 14,  9],
       [ 2,  9, 15, 12, 18,  3, 13, 11,  5, 10],
       [14,  0,  9, 11,  1,  4,  9, 19, 18, 12],
       [ 0, 10,  5, 15,  9, 18,  5,  2, 16, 19],
       [14, 19,  3, 11, 13, 11, 13, 11,  1, 14],
       [ 7, 15, 18,  6,  5, 13,  1,  7,  9, 19],
       [11, 17, 11, 16, 14,  3, 16,  1, 12, 19],
       [ 2,  4, 14,  8,  6,  9, 14,  9,  1,  5],
       [ 1, 10, 15,  0,  1,  9, 18,  2,  2, 12]])

In [45]: np.argpartition(a, np.argmin(a, axis=0))[:, 1:] # 1 is because the first item is the minimum one.
Out[45]:
array([[4, 5, 6, 8, 0, 7, 9, 1, 2],
       [2, 7, 5, 9, 6, 8, 1, 0, 4],
       [5, 8, 1, 9, 7, 3, 6, 2, 4],
       [4, 5, 2, 6, 3, 9, 0, 8, 7],
       [7, 2, 6, 4, 1, 3, 8, 5, 9],
       [2, 3, 5, 7, 6, 4, 0, 9, 1],
       [4, 3, 0, 7, 8, 5, 1, 2, 9],
       [5, 2, 0, 8, 4, 6, 3, 1, 9],
       [0, 1, 9, 4, 3, 7, 5, 2, 6],
       [0, 4, 7, 8, 5, 1, 9, 2, 6]])

In [46]: np.argpartition(a, np.argmin(a, axis=0))[:, -3:]
Out[46]:
array([[9, 1, 2],
       [1, 0, 4],
       [6, 2, 4],
       [0, 8, 7],
       [8, 5, 9],
       [0, 9, 1],
       [1, 2, 9],
       [3, 1, 9],
       [5, 2, 6],
       [9, 2, 6]])

In [89]: a[np.repeat(np.arange(x), 3), ind.ravel()].reshape(x, 3)
Out[89]:
array([[10, 11, 12],
       [16, 16, 18],
       [13, 15, 18],
       [14, 18, 19],
       [16, 18, 19],
       [14, 14, 19],
       [15, 18, 19],
       [16, 17, 19],
       [ 9, 14, 14],
       [12, 15, 18]])

For multidimensional arrays you can use the axis keyword in order to apply the partitioning along the expected axis.

# For a 2D array
indices = np.argpartition(arr, -N, axis=1)[:, -N:]

And for grabbing the items:

x = arr.shape[0]
arr[np.repeat(np.arange(x), N), indices.ravel()].reshape(x, N)

But note that this won’t return a sorted result. In that case you can use np.argsort() along the intended axis:

indices = np.argsort(arr, axis=1)[:, -N:]

# Result
x = arr.shape[0]
arr[np.repeat(np.arange(x), N), indices.ravel()].reshape(x, N)

Here is an example:

In [42]: a = np.random.randint(0, 20, (10, 10))

In [44]: a
Out[44]:
array([[ 7, 11, 12,  0,  2,  3,  4, 10,  6, 10],
       [16, 16,  4,  3, 18,  5, 10,  4, 14,  9],
       [ 2,  9, 15, 12, 18,  3, 13, 11,  5, 10],
       [14,  0,  9, 11,  1,  4,  9, 19, 18, 12],
       [ 0, 10,  5, 15,  9, 18,  5,  2, 16, 19],
       [14, 19,  3, 11, 13, 11, 13, 11,  1, 14],
       [ 7, 15, 18,  6,  5, 13,  1,  7,  9, 19],
       [11, 17, 11, 16, 14,  3, 16,  1, 12, 19],
       [ 2,  4, 14,  8,  6,  9, 14,  9,  1,  5],
       [ 1, 10, 15,  0,  1,  9, 18,  2,  2, 12]])

In [45]: np.argpartition(a, np.argmin(a, axis=0))[:, 1:] # 1 is because the first item is the minimum one.
Out[45]:
array([[4, 5, 6, 8, 0, 7, 9, 1, 2],
       [2, 7, 5, 9, 6, 8, 1, 0, 4],
       [5, 8, 1, 9, 7, 3, 6, 2, 4],
       [4, 5, 2, 6, 3, 9, 0, 8, 7],
       [7, 2, 6, 4, 1, 3, 8, 5, 9],
       [2, 3, 5, 7, 6, 4, 0, 9, 1],
       [4, 3, 0, 7, 8, 5, 1, 2, 9],
       [5, 2, 0, 8, 4, 6, 3, 1, 9],
       [0, 1, 9, 4, 3, 7, 5, 2, 6],
       [0, 4, 7, 8, 5, 1, 9, 2, 6]])

In [46]: np.argpartition(a, np.argmin(a, axis=0))[:, -3:]
Out[46]:
array([[9, 1, 2],
       [1, 0, 4],
       [6, 2, 4],
       [0, 8, 7],
       [8, 5, 9],
       [0, 9, 1],
       [1, 2, 9],
       [3, 1, 9],
       [5, 2, 6],
       [9, 2, 6]])

In [89]: a[np.repeat(np.arange(x), 3), ind.ravel()].reshape(x, 3)
Out[89]:
array([[10, 11, 12],
       [16, 16, 18],
       [13, 15, 18],
       [14, 18, 19],
       [16, 18, 19],
       [14, 14, 19],
       [15, 18, 19],
       [16, 17, 19],
       [ 9, 14, 14],
       [12, 15, 18]])

回答 7

这将比完整排序要快，具体取决于原始数组的大小和所选内容的大小：

>>> A = np.random.randint(0,10,10)
>>> A
array([5, 1, 5, 5, 2, 3, 2, 4, 1, 0])
>>> B = np.zeros(3, int)
>>> for i in xrange(3):
...     idx = np.argmax(A)
...     B[i]=idx; A[idx]=0 #something smaller than A.min()
...     
>>> B
array([0, 2, 3])

当然，它涉及篡改原始阵列。您可以通过复制或替换原始值来解决（如果需要）的问题。…以您的使用案例中较便宜的价格为准

This will be faster than a full sort depending on the size of your original array and the size of your selection:

>>> A = np.random.randint(0,10,10)
>>> A
array([5, 1, 5, 5, 2, 3, 2, 4, 1, 0])
>>> B = np.zeros(3, int)
>>> for i in xrange(3):
...     idx = np.argmax(A)
...     B[i]=idx; A[idx]=0 #something smaller than A.min()
...     
>>> B
array([0, 2, 3])

It, of course, involves tampering with your original array. Which you could fix (if needed) by making a copy or replacing back the original values. …whichever is cheaper for your use case.

回答 8

方法np.argpartition仅返回k个最大的索引，执行局部排序，并且比np.argsort数组很大时要快（执行完整排序）。但是返回的索引不是按升序/降序排列的。让我们举一个例子：

我们可以看到，如果您要对前k个索引使用严格的升序，np.argpartition则不会返回您想要的结果。

除了在np.argpartition之后手动进行排序之外，我的解决方案是使用PyTorch（torch.topk一种用于神经网络构建的工具），为类似NumPy的API提供CPU和GPU支持。它与带有MKL的NumPy一样快，并且如果需要大型矩阵/矢量计算，则可以提供GPU增强。

严格的上升/下降前k个索引代码将是：

请注意，它torch.topk接受火炬张量，并返回type中的前k个值和前k个索引torch.Tensor。与np相似，torch.topk也接受轴参数，以便您可以处理多维数组/张量。

Method np.argpartition only returns the k largest indices, performs a local sort, and is faster than np.argsort(performing a full sort) when array is quite large. But the returned indices are NOT in ascending/descending order. Let’s say with an example:

We can see that if you want a strict ascending order top k indices, np.argpartition won’t return what you want.

Apart from doing a sort manually after np.argpartition, my solution is to use PyTorch, torch.topk, a tool for neural network construction, providing NumPy-like APIs with both CPU and GPU support. It’s as fast as NumPy with MKL, and offers a GPU boost if you need large matrix/vector calculations.

Strict ascend/descend top k indices code will be:

Note that torch.topk accepts a torch tensor, and returns both top k values and top k indices in type torch.Tensor. Similar with np, torch.topk also accepts an axis argument so that you can handle multi-dimensional arrays/tensors.

回答 9

采用：

from operator import itemgetter
from heapq import nlargest
result = nlargest(N, enumerate(your_list), itemgetter(1))

现在，result列表将包含N个元组（index，value），其中value已最大化。

Use:

from operator import itemgetter
from heapq import nlargest
result = nlargest(N, enumerate(your_list), itemgetter(1))

Now the result list would contain N tuples (index, value) where value is maximized.

回答 10

采用：

def max_indices(arr, k):
    '''
    Returns the indices of the k first largest elements of arr
    (in descending order in values)
    '''
    assert k <= arr.size, 'k should be smaller or equal to the array size'
    arr_ = arr.astype(float)  # make a copy of arr
    max_idxs = []
    for _ in range(k):
        max_element = np.max(arr_)
        if np.isinf(max_element):
            break
        else:
            idx = np.where(arr_ == max_element)
        max_idxs.append(idx)
        arr_[idx] = -np.inf
    return max_idxs

它也适用于2D阵列。例如，

In [0]: A = np.array([[ 0.51845014,  0.72528114],
                     [ 0.88421561,  0.18798661],
                     [ 0.89832036,  0.19448609],
                     [ 0.89832036,  0.19448609]])
In [1]: max_indices(A, 8)
Out[1]:
    [(array([2, 3], dtype=int64), array([0, 0], dtype=int64)),
     (array([1], dtype=int64), array([0], dtype=int64)),
     (array([0], dtype=int64), array([1], dtype=int64)),
     (array([0], dtype=int64), array([0], dtype=int64)),
     (array([2, 3], dtype=int64), array([1, 1], dtype=int64)),
     (array([1], dtype=int64), array([1], dtype=int64))]

In [2]: A[max_indices(A, 8)[0]][0]
Out[2]: array([ 0.89832036])

Use:

def max_indices(arr, k):
    '''
    Returns the indices of the k first largest elements of arr
    (in descending order in values)
    '''
    assert k <= arr.size, 'k should be smaller or equal to the array size'
    arr_ = arr.astype(float)  # make a copy of arr
    max_idxs = []
    for _ in range(k):
        max_element = np.max(arr_)
        if np.isinf(max_element):
            break
        else:
            idx = np.where(arr_ == max_element)
        max_idxs.append(idx)
        arr_[idx] = -np.inf
    return max_idxs

It also works with 2D arrays. For example,

In [0]: A = np.array([[ 0.51845014,  0.72528114],
                     [ 0.88421561,  0.18798661],
                     [ 0.89832036,  0.19448609],
                     [ 0.89832036,  0.19448609]])
In [1]: max_indices(A, 8)
Out[1]:
    [(array([2, 3], dtype=int64), array([0, 0], dtype=int64)),
     (array([1], dtype=int64), array([0], dtype=int64)),
     (array([0], dtype=int64), array([1], dtype=int64)),
     (array([0], dtype=int64), array([0], dtype=int64)),
     (array([2, 3], dtype=int64), array([1, 1], dtype=int64)),
     (array([1], dtype=int64), array([1], dtype=int64))]

In [2]: A[max_indices(A, 8)[0]][0]
Out[2]: array([ 0.89832036])

回答 11

bottleneck 如果仅为了获得N个最大值而对整个数组进行排序的开销太大，则具有部分排序函数。

_{我对这个模块一无所知。我只是谷歌搜索numpy partial sort。}

bottleneck has a partial sort function, if the expense of sorting the entire array just to get the N largest values is too great.

_{I know nothing about this module; I just googled numpy partial sort.}

回答 12

以下是查看最大元素及其位置的非常简单的方法。这axis是域；axis= 0表示按列最大数量，而axis1表示2D情况下按行最大数量。对于更大的尺寸，则取决于您。

M = np.random.random((3, 4))
print(M)
print(M.max(axis=1), M.argmax(axis=1))

The following is a very easy way to see the maximum elements and its positions. Here axis is the domain; axis = 0 means column wise maximum number and axis = 1 means row wise max number for the 2D case. And for higher dimensions it depends upon you.

M = np.random.random((3, 4))
print(M)
print(M.max(axis=1), M.argmax(axis=1))

回答 13

我发现使用起来最直观np.unique。

这个想法是，唯一方法返回输入值的索引。然后，根据最大唯一值和指标，可以重新创建原始值的位置。

multi_max = [1,1,2,2,4,0,0,4]
uniques, idx = np.unique(multi_max, return_inverse=True)
print np.squeeze(np.argwhere(idx == np.argmax(uniques)))
>> [4 7]

I found it most intuitive to use np.unique.

The idea is, that the unique method returns the indices of the input values. Then from the max unique value and the indicies, the position of the original values can be recreated.

multi_max = [1,1,2,2,4,0,0,4]
uniques, idx = np.unique(multi_max, return_inverse=True)
print np.squeeze(np.argwhere(idx == np.argmax(uniques)))
>> [4 7]

回答 14

我认为，最省时的方法是手动遍历数组，并保持k大小的最小堆大小，正如其他人提到的那样。

我还提出了一种蛮力方法：

top_k_index_list = [ ]
for i in range(k):
    top_k_index_list.append(np.argmax(my_array))
    my_array[top_k_index_list[-1]] = -float('inf')

在使用argmax获取其索引之后，将最大元素设置为较大的负值。然后下一次调用argmax将返回第二大元素。您可以记录这些元素的原始值，并根据需要恢复它们。

I think the most time efficiency way is manually iterate through the array and keep a k-size min-heap, as other people have mentioned.

And I also come up with a brute force approach:

top_k_index_list = [ ]
for i in range(k):
    top_k_index_list.append(np.argmax(my_array))
    my_array[top_k_index_list[-1]] = -float('inf')

Set the largest element to a large negative value after you use argmax to get its index. And then the next call of argmax will return the second largest element. And you can log the original value of these elements and recover them if you want.

回答 15

这段代码适用于numpy矩阵数组：

mat = np.array([[1, 3], [2, 5]]) # numpy matrix

n = 2  # n
n_largest_mat = np.sort(mat, axis=None)[-n:] # n_largest 
tf_n_largest = np.zeros((2,2), dtype=bool) # all false matrix
for x in n_largest_mat: 
  tf_n_largest = (tf_n_largest) | (mat == x) # true-false  

n_largest_elems = mat[tf_n_largest] # true-false indexing

这会产生一个真假n_largest矩阵索引，该索引也可以从矩阵数组中提取n_largest个元素

This code works for a numpy matrix array:

mat = np.array([[1, 3], [2, 5]]) # numpy matrix

n = 2  # n
n_largest_mat = np.sort(mat, axis=None)[-n:] # n_largest 
tf_n_largest = np.zeros((2,2), dtype=bool) # all false matrix
for x in n_largest_mat: 
  tf_n_largest = (tf_n_largest) | (mat == x) # true-false  

n_largest_elems = mat[tf_n_largest] # true-false indexing

This produces a true-false n_largest matrix indexing that also works to extract n_largest elements from a matrix array

知识问答

与常规Python列表相比，NumPy有什么优势？

2021年7月25日 Python实用宝典

问题：与常规Python列表相比，NumPy有什么优势？

与常规Python列表相比，NumPy有什么优势？

我大约有100个金融市场系列，我将创建一个100x100x100 = 1百万个单元的多维数据集数组。我将每个x与y和z回归（3变量），以用标准误差填充数组。

我听说对于“大型矩阵”，出于性能和可伸缩性的原因，我应该使用NumPy而不是Python列表。事实是，我知道Python列表，它们似乎对我有用。

如果我转到NumPy，会有什么好处？

如果我有1000个序列（即立方体中有10亿个浮点单元）怎么办？

What are the advantages of NumPy over regular Python lists?

I have approximately 100 financial markets series, and I am going to create a cube array of 100x100x100 = 1 million cells. I will be regressing (3-variable) each x with each y and z, to fill the array with standard errors.

I have heard that for “large matrices” I should use NumPy as opposed to Python lists, for performance and scalability reasons. Thing is, I know Python lists and they seem to work for me.

What will the benefits be if I move to NumPy?

What if I had 1000 series (that is, 1 billion floating point cells in the cube)?

回答 0

NumPy的数组比Python列表更紧凑-您在Python中描述的列表列表至少需要20 MB左右，而单元格中具有单精度浮点数的NumPy 3D数组则需要4 MB。使用NumPy可以更快地读取和写入项目。

也许您只关心一百万个单元就不会那么在意，但是您肯定会关心十亿个单元-两种方法都不适合32位体系结构，但是使用64位版本，NumPy可以节省约4 GB ，仅Python一项就至少需要12 GB（很多指针的大小加倍），这是一个昂贵得多的硬件！

差异主要是由于“间接性”造成的-Python列表是指向Python对象的指针的数组，每个指针至少4个字节，对于最小的Python对象也至少包含16个字节（类型指针为4，引用计数为4，类型为4值-内存分配器舍入为16）。NumPy数组是统一值的数组-单精度数字每个占用4个字节，双精度数字每个占用8个字节。灵活性较差，但您要为标准Python列表的灵活性付出高昂的代价！

NumPy’s arrays are more compact than Python lists — a list of lists as you describe, in Python, would take at least 20 MB or so, while a NumPy 3D array with single-precision floats in the cells would fit in 4 MB. Access in reading and writing items is also faster with NumPy.

Maybe you don’t care that much for just a million cells, but you definitely would for a billion cells — neither approach would fit in a 32-bit architecture, but with 64-bit builds NumPy would get away with 4 GB or so, Python alone would need at least about 12 GB (lots of pointers which double in size) — a much costlier piece of hardware!

The difference is mostly due to “indirectness” — a Python list is an array of pointers to Python objects, at least 4 bytes per pointer plus 16 bytes for even the smallest Python object (4 for type pointer, 4 for reference count, 4 for value — and the memory allocators rounds up to 16). A NumPy array is an array of uniform values — single-precision numbers takes 4 bytes each, double-precision ones, 8 bytes. Less flexible, but you pay substantially for the flexibility of standard Python lists!

回答 1

NumPy不仅效率更高；这也更加方便。您可以免费获得许多矢量和矩阵运算，有时这可以避免不必要的工作。而且它们也得到有效实施。

例如，您可以将多维数据集直接从文件读取到数组中：

x = numpy.fromfile(file=open("data"), dtype=float).reshape((100, 100, 100))

沿第二维求和：

s = x.sum(axis=1)

查找哪些单元格高于阈值：

(x > 0.5).nonzero()

删除沿第三维的每个偶数索引切片：

x[:, :, ::2]

同样，许多有用的库都可以与NumPy数组一起使用。例如，统计分析和可视化库。

即使您没有性能问题，学习NumPy也是值得的。

NumPy is not just more efficient; it is also more convenient. You get a lot of vector and matrix operations for free, which sometimes allow one to avoid unnecessary work. And they are also efficiently implemented.

For example, you could read your cube directly from a file into an array:

x = numpy.fromfile(file=open("data"), dtype=float).reshape((100, 100, 100))

Sum along the second dimension:

s = x.sum(axis=1)

Find which cells are above a threshold:

(x > 0.5).nonzero()

Remove every even-indexed slice along the third dimension:

x[:, :, ::2]

Also, many useful libraries work with NumPy arrays. For example, statistical analysis and visualization libraries.

Even if you don’t have performance problems, learning NumPy is worth the effort.

回答 2

Alex提到了内存效率，Roberto提到了便利性，这些都是不错的地方。对于其他一些想法，我将提到速度和功能。

功能性：NumPy，FFT，卷积，快速搜索，基本统计信息，线性代数，直方图等都内置了很多功能。实际上，没有FFT谁能活下去？

速度：这是一项对列表和NumPy数组求和的测试，表明NumPy数组的求和速度快10倍（在此测试中，里程可能会有所不同）。

from numpy import arange
from timeit import Timer

Nelements = 10000
Ntimeits = 10000

x = arange(Nelements)
y = range(Nelements)

t_numpy = Timer("x.sum()", "from __main__ import x")
t_list = Timer("sum(y)", "from __main__ import y")
print("numpy: %.3e" % (t_numpy.timeit(Ntimeits)/Ntimeits,))
print("list:  %.3e" % (t_list.timeit(Ntimeits)/Ntimeits,))

在我的系统上（运行备份时），它会给出：

numpy: 3.004e-05
list:  5.363e-04

Alex mentioned memory efficiency, and Roberto mentions convenience, and these are both good points. For a few more ideas, I’ll mention speed and functionality.

Functionality: You get a lot built in with NumPy, FFTs, convolutions, fast searching, basic statistics, linear algebra, histograms, etc. And really, who can live without FFTs?

Speed: Here’s a test on doing a sum over a list and a NumPy array, showing that the sum on the NumPy array is 10x faster (in this test — mileage may vary).

from numpy import arange
from timeit import Timer

Nelements = 10000
Ntimeits = 10000

x = arange(Nelements)
y = range(Nelements)

t_numpy = Timer("x.sum()", "from __main__ import x")
t_list = Timer("sum(y)", "from __main__ import y")
print("numpy: %.3e" % (t_numpy.timeit(Ntimeits)/Ntimeits,))
print("list:  %.3e" % (t_list.timeit(Ntimeits)/Ntimeits,))

which on my systems (while I’m running a backup) gives:

numpy: 3.004e-05
list:  5.363e-04

回答 3

这是scipy.org网站上的常见问题解答中的一个很好的答案：

与（嵌套）Python列表相比，NumPy数组有什么优势？

Python的列表是有效的通用容器。它们支持（相当）高效的插入，删除，附加和连接，并且Python的列表理解使它们易于构造和操作。但是，它们有一定的局限性：它们不支持“向量化”操作，例如逐元素加法和乘法，并且它们可以包含不同类型的对象这一事实意味着Python必须为每个元素存储类型信息，并且必须执行类型分派代码在每个元素上操作时。这也意味着有效的C循环几乎无法执行列表操作-每次迭代都需要类型检查和其他Python API簿记。

Here’s a nice answer from the FAQ on the scipy.org website:

What advantages do NumPy arrays offer over (nested) Python lists?

Python’s lists are efficient general-purpose containers. They support (fairly) efficient insertion, deletion, appending, and concatenation, and Python’s list comprehensions make them easy to construct and manipulate. However, they have certain limitations: they don’t support “vectorized” operations like elementwise addition and multiplication, and the fact that they can contain objects of differing types mean that Python must store type information for every element, and must execute type dispatching code when operating on each element. This also means that very few list operations can be carried out by efficient C loops – each iteration would require type checks and other Python API bookkeeping.

回答 4

所有人都强调了numpy数组和python列表之间的几乎所有主要区别，在这里我将向大家简单介绍一下：

Numpy数组在创建时具有固定的大小，这与python列表（可以动态增长）不同。更改ndarray的大小将创建一个新数组并删除原始数组。
Numpy数组中的所有元素都必须具有相同的数据类型（我们也可以具有异构类型，但这将不允许您进行数学运算），因此在内存中的大小将相同
Numpy数组有助于对大量数据进行数学运算和其他类型的运算。通常，与使用python顺序构建相比，此类操作执行效率更高且代码更少

All have highlighted almost all major differences between numpy array and python list, I will just brief them out here:

Numpy arrays have a fixed size at creation, unlike python lists (which can grow dynamically). Changing the size of ndarray will create a new array and delete the original.
The elements in a Numpy array are all required to be of the same data type (we can have the heterogeneous type as well but that will not gonna permit you mathematical operations) and thus will be the same size in memory
Numpy arrays are facilitated advances mathematical and other types of operations on large numbers of data. Typically such operations are executed more efficiently and with less code than is possible using pythons build in sequences

知识问答

将pandas数据框转换为NumPy数组

2021年7月25日 Python实用宝典

问题：将pandas数据框转换为NumPy数组

我对知道如何将熊猫数据框转换为NumPy数组感兴趣。

数据框：

import numpy as np
import pandas as pd

index = [1, 2, 3, 4, 5, 6, 7]
a = [np.nan, np.nan, np.nan, 0.1, 0.1, 0.1, 0.1]
b = [0.2, np.nan, 0.2, 0.2, 0.2, np.nan, np.nan]
c = [np.nan, 0.5, 0.5, np.nan, 0.5, 0.5, np.nan]
df = pd.DataFrame({'A': a, 'B': b, 'C': c}, index=index)
df = df.rename_axis('ID')

给

label   A    B    C
ID                                 
1   NaN  0.2  NaN
2   NaN  NaN  0.5
3   NaN  0.2  0.5
4   0.1  0.2  NaN
5   0.1  0.2  0.5
6   0.1  NaN  0.5
7   0.1  NaN  NaN

我想将其转换为NumPy数组，如下所示：

array([[ nan,  0.2,  nan],
       [ nan,  nan,  0.5],
       [ nan,  0.2,  0.5],
       [ 0.1,  0.2,  nan],
       [ 0.1,  0.2,  0.5],
       [ 0.1,  nan,  0.5],
       [ 0.1,  nan,  nan]])

我怎样才能做到这一点？

另外，是否可以像这样保留dtype？

array([[ 1, nan,  0.2,  nan],
       [ 2, nan,  nan,  0.5],
       [ 3, nan,  0.2,  0.5],
       [ 4, 0.1,  0.2,  nan],
       [ 5, 0.1,  0.2,  0.5],
       [ 6, 0.1,  nan,  0.5],
       [ 7, 0.1,  nan,  nan]],
     dtype=[('ID', '<i4'), ('A', '<f8'), ('B', '<f8'), ('B', '<f8')])

或类似的？

I am interested in knowing how to convert a pandas dataframe into a NumPy array.

dataframe:

import numpy as np
import pandas as pd

index = [1, 2, 3, 4, 5, 6, 7]
a = [np.nan, np.nan, np.nan, 0.1, 0.1, 0.1, 0.1]
b = [0.2, np.nan, 0.2, 0.2, 0.2, np.nan, np.nan]
c = [np.nan, 0.5, 0.5, np.nan, 0.5, 0.5, np.nan]
df = pd.DataFrame({'A': a, 'B': b, 'C': c}, index=index)
df = df.rename_axis('ID')

gives

label   A    B    C
ID                                 
1   NaN  0.2  NaN
2   NaN  NaN  0.5
3   NaN  0.2  0.5
4   0.1  0.2  NaN
5   0.1  0.2  0.5
6   0.1  NaN  0.5
7   0.1  NaN  NaN

I would like to convert this to a NumPy array, as so:

array([[ nan,  0.2,  nan],
       [ nan,  nan,  0.5],
       [ nan,  0.2,  0.5],
       [ 0.1,  0.2,  nan],
       [ 0.1,  0.2,  0.5],
       [ 0.1,  nan,  0.5],
       [ 0.1,  nan,  nan]])

How can I do this?

As a bonus, is it possible to preserve the dtypes, like this?

array([[ 1, nan,  0.2,  nan],
       [ 2, nan,  nan,  0.5],
       [ 3, nan,  0.2,  0.5],
       [ 4, 0.1,  0.2,  nan],
       [ 5, 0.1,  0.2,  0.5],
       [ 6, 0.1,  nan,  0.5],
       [ 7, 0.1,  nan,  nan]],
     dtype=[('ID', '<i4'), ('A', '<f8'), ('B', '<f8'), ('B', '<f8')])

or similar?

回答 0

要将熊猫数据框（df）转换为numpy ndarray，请使用以下代码：

df.values

array([[nan, 0.2, nan],
       [nan, nan, 0.5],
       [nan, 0.2, 0.5],
       [0.1, 0.2, nan],
       [0.1, 0.2, 0.5],
       [0.1, nan, 0.5],
       [0.1, nan, nan]])

To convert a pandas dataframe (df) to a numpy ndarray, use this code:

df.values

array([[nan, 0.2, nan],
       [nan, nan, 0.5],
       [nan, 0.2, 0.5],
       [0.1, 0.2, nan],
       [0.1, 0.2, 0.5],
       [0.1, nan, 0.5],
       [0.1, nan, nan]])

回答 1

弃用的用法`values`和`as_matrix()`！

pandas v0.24.0引入了两种从pandas对象获取NumPy数组的新方法：

to_numpy()，其定义上Index，Series,和DataFrame对象，并
array，仅在Index和Series对象上定义。

如果您访问的v0.24文档.values，则会看到一个红色的大警告：

警告：我们建议DataFrame.to_numpy()改为使用。

请参阅v0.24.0发行说明的本部分以及此答案以获取更多信息。

追求更好的一致性： `to_numpy()`

本着整个API更好的一致性的精神，to_numpy引入了一种新方法来从DataFrames中提取底层的NumPy数组。

# Setup.
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]}, index=['a', 'b', 'c'])

df.to_numpy()
array([[1, 4],
       [2, 5],
       [3, 6]])

如上所述，该方法也在Index和Series对象上定义（请参见此处）。

df.index.to_numpy()
# array(['a', 'b', 'c'], dtype=object)

df['A'].to_numpy()
#  array([1, 2, 3])

默认情况下，将返回视图，因此所做的任何修改都会影响原始视图。

v = df.to_numpy()
v[0, 0] = -1

df
   A  B
a -1  4
b  2  5
c  3  6

如果您需要副本，请使用to_numpy(copy=True。

Pandas> = 1.0更新为ExtensionTypes

如果您使用的是熊猫1.x，那么您可能会更多地处理扩展类型。您必须多加注意，这些扩展名类型已正确转换。

a = pd.array([1, 2, None], dtype="Int64")                                  
a                                                                          

<IntegerArray>
[1, 2, <NA>]
Length: 3, dtype: Int64 

# Wrong
a.to_numpy()                                                               
# array([1, 2, <NA>], dtype=object)  # yuck, objects

# Right
a.to_numpy(dtype='float', na_value=np.nan)                                 
# array([ 1.,  2., nan])

在文档中对此进行了标注。

如果您需要`dtypes`…

如另一个答案所示，DataFrame.to_records是执行此操作的好方法。

df.to_records()
# rec.array([('a', -1, 4), ('b',  2, 5), ('c',  3, 6)],
#           dtype=[('index', 'O'), ('A', '<i8'), ('B', '<i8')])

to_numpy不幸的是，这不能做到。但是，您也可以使用np.rec.fromrecords：

v = df.reset_index()
np.rec.fromrecords(v, names=v.columns.tolist())
# rec.array([('a', -1, 4), ('b',  2, 5), ('c',  3, 6)],
#          dtype=[('index', '<U1'), ('A', '<i8'), ('B', '<i8')])

在性能方面，它几乎是相同的（实际上，使用rec.fromrecords速度要快一些）。

df2 = pd.concat([df] * 10000)

%timeit df2.to_records()
%%timeit
v = df2.reset_index()
np.rec.fromrecords(v, names=v.columns.tolist())

11.1 ms ± 557 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
9.67 ms ± 126 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

添加新方法的理由

to_numpy()（除了array）作为两个GitHub问题GH19954和GH23623下讨论的结果而添加。

具体来说，文档提到了基本原理：

[…] .values尚不清楚返回的值是实际数组，它的某种转换还是熊猫自定义数组之一（如Categorical）。例如，使用PeriodIndex，每次都会.values 生成一个新ndarray的周期对象。[…]

to_numpy旨在提高API的一致性，这是朝正确方向迈出的重要一步。.values不会在当前版本中被弃用，但我希望这种情况可能会在将来的某个时刻发生，因此，我敦促用户尽快向较新的API迁移。

批判其他解决方案

DataFrame.values 如前所述，具有不一致的行为。

DataFrame.get_values()只是一个包装器DataFrame.values，因此上述所有内容均适用。

DataFrame.as_matrix()现在已弃用，请勿使用！

Deprecate your usage of `values` and `as_matrix()`!

pandas v0.24.0 introduced two new methods for obtaining NumPy arrays from pandas objects:

to_numpy(), which is defined on Index, Series, and DataFrame objects, and
array, which is defined on Index and Series objects only.

If you visit the v0.24 docs for .values, you will see a big red warning that says:

Warning: We recommend using DataFrame.to_numpy() instead.

See this section of the v0.24.0 release notes, and this answer for more information.

Towards Better Consistency: `to_numpy()`

In the spirit of better consistency throughout the API, a new method to_numpy has been introduced to extract the underlying NumPy array from DataFrames.

# Setup.
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]}, index=['a', 'b', 'c'])

df.to_numpy()
array([[1, 4],
       [2, 5],
       [3, 6]])

As mentioned above, this method is also defined on Index and Series objects (see here).

df.index.to_numpy()
# array(['a', 'b', 'c'], dtype=object)

df['A'].to_numpy()
#  array([1, 2, 3])

By default, a view is returned, so any modifications made will affect the original.

v = df.to_numpy()
v[0, 0] = -1

df
   A  B
a -1  4
b  2  5
c  3  6

If you need a copy instead, use to_numpy(copy=True).

pandas >= 1.0 update for ExtensionTypes

If you’re using pandas 1.x, chances are you’ll be dealing with extension types a lot more. You’ll have to be a little more careful that these extension types are correctly converted.

a = pd.array([1, 2, None], dtype="Int64")                                  
a                                                                          

<IntegerArray>
[1, 2, <NA>]
Length: 3, dtype: Int64 

# Wrong
a.to_numpy()                                                               
# array([1, 2, <NA>], dtype=object)  # yuck, objects

# Right
a.to_numpy(dtype='float', na_value=np.nan)                                 
# array([ 1.,  2., nan])

This is called out in the docs.

If you need the `dtypes`…

As shown in another answer, DataFrame.to_records is a good way to do this.

df.to_records()
# rec.array([('a', -1, 4), ('b',  2, 5), ('c',  3, 6)],
#           dtype=[('index', 'O'), ('A', '<i8'), ('B', '<i8')])

This cannot be done with to_numpy, unfortunately. However, as an alternative, you can use np.rec.fromrecords:

v = df.reset_index()
np.rec.fromrecords(v, names=v.columns.tolist())
# rec.array([('a', -1, 4), ('b',  2, 5), ('c',  3, 6)],
#          dtype=[('index', '<U1'), ('A', '<i8'), ('B', '<i8')])

Performance wise, it’s nearly the same (actually, using rec.fromrecords is a bit faster).

df2 = pd.concat([df] * 10000)

%timeit df2.to_records()
%%timeit
v = df2.reset_index()
np.rec.fromrecords(v, names=v.columns.tolist())

11.1 ms ± 557 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
9.67 ms ± 126 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Rationale for Adding a New Method

to_numpy() (in addition to array) was added as a result of discussions under two GitHub issues GH19954 and GH23623.

Specifically, the docs mention the rationale:

[…] with .values it was unclear whether the returned value would be the actual array, some transformation of it, or one of pandas custom arrays (like Categorical). For example, with PeriodIndex, .values generates a new ndarray of period objects each time. […]

to_numpy aim to improve the consistency of the API, which is a major step in the right direction. .values will not be deprecated in the current version, but I expect this may happen at some point in the future, so I would urge users to migrate towards the newer API, as soon as you can.

Critique of Other Solutions

DataFrame.values has inconsistent behaviour, as already noted.

DataFrame.get_values() is simply a wrapper around DataFrame.values, so everything said above applies.

DataFrame.as_matrix() is deprecated now, do NOT use!

回答 2

注意：.as_matrix()不建议使用此答案中的方法。熊猫0.23.4警告：

方法.as_matrix将在以后的版本中删除。请改用.values。

熊猫内置一些东西…

numpy_matrix = df.as_matrix()

给

array([[nan, 0.2, nan],
       [nan, nan, 0.5],
       [nan, 0.2, 0.5],
       [0.1, 0.2, nan],
       [0.1, 0.2, 0.5],
       [0.1, nan, 0.5],
       [0.1, nan, nan]])

Note: The .as_matrix() method used in this answer is deprecated. Pandas 0.23.4 warns:

Method .as_matrix will be removed in a future version. Use .values instead.

Pandas has something built in…

numpy_matrix = df.as_matrix()

gives

array([[nan, 0.2, nan],
       [nan, nan, 0.5],
       [nan, 0.2, 0.5],
       [0.1, 0.2, nan],
       [0.1, 0.2, 0.5],
       [0.1, nan, 0.5],
       [0.1, nan, nan]])

回答 3

我只需要链接DataFrame.reset_index（）和DataFrame.values函数来获得数据帧的Numpy表示，包括索引：

In [8]: df
Out[8]: 
          A         B         C
0 -0.982726  0.150726  0.691625
1  0.617297 -0.471879  0.505547
2  0.417123 -1.356803 -1.013499
3 -0.166363 -0.957758  1.178659
4 -0.164103  0.074516 -0.674325
5 -0.340169 -0.293698  1.231791
6 -1.062825  0.556273  1.508058
7  0.959610  0.247539  0.091333

[8 rows x 3 columns]

In [9]: df.reset_index().values
Out[9]:
array([[ 0.        , -0.98272574,  0.150726  ,  0.69162512],
       [ 1.        ,  0.61729734, -0.47187926,  0.50554728],
       [ 2.        ,  0.4171228 , -1.35680324, -1.01349922],
       [ 3.        , -0.16636303, -0.95775849,  1.17865945],
       [ 4.        , -0.16410334,  0.0745164 , -0.67432474],
       [ 5.        , -0.34016865, -0.29369841,  1.23179064],
       [ 6.        , -1.06282542,  0.55627285,  1.50805754],
       [ 7.        ,  0.95961001,  0.24753911,  0.09133339]])

为了获得dtypes，我们需要使用view将此ndarray转换为结构化数组：

In [10]: df.reset_index().values.ravel().view(dtype=[('index', int), ('A', float), ('B', float), ('C', float)])
Out[10]:
array([( 0, -0.98272574,  0.150726  ,  0.69162512),
       ( 1,  0.61729734, -0.47187926,  0.50554728),
       ( 2,  0.4171228 , -1.35680324, -1.01349922),
       ( 3, -0.16636303, -0.95775849,  1.17865945),
       ( 4, -0.16410334,  0.0745164 , -0.67432474),
       ( 5, -0.34016865, -0.29369841,  1.23179064),
       ( 6, -1.06282542,  0.55627285,  1.50805754),
       ( 7,  0.95961001,  0.24753911,  0.09133339),
       dtype=[('index', '<i8'), ('A', '<f8'), ('B', '<f8'), ('C', '<f8')])

I would just chain the DataFrame.reset_index() and DataFrame.values functions to get the Numpy representation of the dataframe, including the index:

In [8]: df
Out[8]: 
          A         B         C
0 -0.982726  0.150726  0.691625
1  0.617297 -0.471879  0.505547
2  0.417123 -1.356803 -1.013499
3 -0.166363 -0.957758  1.178659
4 -0.164103  0.074516 -0.674325
5 -0.340169 -0.293698  1.231791
6 -1.062825  0.556273  1.508058
7  0.959610  0.247539  0.091333

[8 rows x 3 columns]

In [9]: df.reset_index().values
Out[9]:
array([[ 0.        , -0.98272574,  0.150726  ,  0.69162512],
       [ 1.        ,  0.61729734, -0.47187926,  0.50554728],
       [ 2.        ,  0.4171228 , -1.35680324, -1.01349922],
       [ 3.        , -0.16636303, -0.95775849,  1.17865945],
       [ 4.        , -0.16410334,  0.0745164 , -0.67432474],
       [ 5.        , -0.34016865, -0.29369841,  1.23179064],
       [ 6.        , -1.06282542,  0.55627285,  1.50805754],
       [ 7.        ,  0.95961001,  0.24753911,  0.09133339]])

To get the dtypes we’d need to transform this ndarray into a structured array using view:

In [10]: df.reset_index().values.ravel().view(dtype=[('index', int), ('A', float), ('B', float), ('C', float)])
Out[10]:
array([( 0, -0.98272574,  0.150726  ,  0.69162512),
       ( 1,  0.61729734, -0.47187926,  0.50554728),
       ( 2,  0.4171228 , -1.35680324, -1.01349922),
       ( 3, -0.16636303, -0.95775849,  1.17865945),
       ( 4, -0.16410334,  0.0745164 , -0.67432474),
       ( 5, -0.34016865, -0.29369841,  1.23179064),
       ( 6, -1.06282542,  0.55627285,  1.50805754),
       ( 7,  0.95961001,  0.24753911,  0.09133339),
       dtype=[('index', '<i8'), ('A', '<f8'), ('B', '<f8'), ('C', '<f8')])

回答 4

您可以使用该to_records方法，但是如果dtypes不是您一开始就想要的，就必须多花点时间。就我而言，从字符串复制了DF，索引类型为字符串（以objectdtypes在pandas中表示）：

In [102]: df
Out[102]: 
label    A    B    C
ID                  
1      NaN  0.2  NaN
2      NaN  NaN  0.5
3      NaN  0.2  0.5
4      0.1  0.2  NaN
5      0.1  0.2  0.5
6      0.1  NaN  0.5
7      0.1  NaN  NaN

In [103]: df.index.dtype
Out[103]: dtype('object')
In [104]: df.to_records()
Out[104]: 
rec.array([(1, nan, 0.2, nan), (2, nan, nan, 0.5), (3, nan, 0.2, 0.5),
       (4, 0.1, 0.2, nan), (5, 0.1, 0.2, 0.5), (6, 0.1, nan, 0.5),
       (7, 0.1, nan, nan)], 
      dtype=[('index', '|O8'), ('A', '<f8'), ('B', '<f8'), ('C', '<f8')])
In [106]: df.to_records().dtype
Out[106]: dtype([('index', '|O8'), ('A', '<f8'), ('B', '<f8'), ('C', '<f8')])

转换recarray dtype对我不起作用，但是已经可以在Pandas中做到这一点：

In [109]: df.index = df.index.astype('i8')
In [111]: df.to_records().view([('ID', '<i8'), ('A', '<f8'), ('B', '<f8'), ('C', '<f8')])
Out[111]:
rec.array([(1, nan, 0.2, nan), (2, nan, nan, 0.5), (3, nan, 0.2, 0.5),
       (4, 0.1, 0.2, nan), (5, 0.1, 0.2, 0.5), (6, 0.1, nan, 0.5),
       (7, 0.1, nan, nan)], 
      dtype=[('ID', '<i8'), ('A', '<f8'), ('B', '<f8'), ('C', '<f8')])

请注意，Pandas ID在导出的记录数组中没有正确地将索引名设置为（错误？），因此我们可以从类型转换中受益，也可以对此进行更正。

目前，Pandas只有8个字节的整数i8，并且浮点数f8（请参阅本期）。

You can use the to_records method, but have to play around a bit with the dtypes if they are not what you want from the get go. In my case, having copied your DF from a string, the index type is string (represented by an object dtype in pandas):

In [102]: df
Out[102]: 
label    A    B    C
ID                  
1      NaN  0.2  NaN
2      NaN  NaN  0.5
3      NaN  0.2  0.5
4      0.1  0.2  NaN
5      0.1  0.2  0.5
6      0.1  NaN  0.5
7      0.1  NaN  NaN

In [103]: df.index.dtype
Out[103]: dtype('object')
In [104]: df.to_records()
Out[104]: 
rec.array([(1, nan, 0.2, nan), (2, nan, nan, 0.5), (3, nan, 0.2, 0.5),
       (4, 0.1, 0.2, nan), (5, 0.1, 0.2, 0.5), (6, 0.1, nan, 0.5),
       (7, 0.1, nan, nan)], 
      dtype=[('index', '|O8'), ('A', '<f8'), ('B', '<f8'), ('C', '<f8')])
In [106]: df.to_records().dtype
Out[106]: dtype([('index', '|O8'), ('A', '<f8'), ('B', '<f8'), ('C', '<f8')])

Converting the recarray dtype does not work for me, but one can do this in Pandas already:

In [109]: df.index = df.index.astype('i8')
In [111]: df.to_records().view([('ID', '<i8'), ('A', '<f8'), ('B', '<f8'), ('C', '<f8')])
Out[111]:
rec.array([(1, nan, 0.2, nan), (2, nan, nan, 0.5), (3, nan, 0.2, 0.5),
       (4, 0.1, 0.2, nan), (5, 0.1, 0.2, 0.5), (6, 0.1, nan, 0.5),
       (7, 0.1, nan, nan)], 
      dtype=[('ID', '<i8'), ('A', '<f8'), ('B', '<f8'), ('C', '<f8')])

Note that Pandas does not set the name of the index properly (to ID) in the exported record array (a bug?), so we profit from the type conversion to also correct for that.

At the moment Pandas has only 8-byte integers, i8, and floats, f8 (see this issue).

回答 5

似乎df.to_records()会为您工作。您正在寻找的确切功能已被要求并to_records指出作为替代。

我使用您的示例在本地进行了尝试，该调用产生的结果与您正在寻找的输出非常相似：

rec.array([(1, nan, 0.2, nan), (2, nan, nan, 0.5), (3, nan, 0.2, 0.5),
       (4, 0.1, 0.2, nan), (5, 0.1, 0.2, 0.5), (6, 0.1, nan, 0.5),
       (7, 0.1, nan, nan)],
      dtype=[(u'ID', '<i8'), (u'A', '<f8'), (u'B', '<f8'), (u'C', '<f8')])

请注意，这是一个recarray而不是array。您可以通过将其构造函数调用为，将结果移入常规numpy数组np.array(df.to_records())。

It seems like df.to_records() will work for you. The exact feature you’re looking for was requested and to_records pointed to as an alternative.

I tried this out locally using your example, and that call yields something very similar to the output you were looking for:

rec.array([(1, nan, 0.2, nan), (2, nan, nan, 0.5), (3, nan, 0.2, 0.5),
       (4, 0.1, 0.2, nan), (5, 0.1, 0.2, 0.5), (6, 0.1, nan, 0.5),
       (7, 0.1, nan, nan)],
      dtype=[(u'ID', '<i8'), (u'A', '<f8'), (u'B', '<f8'), (u'C', '<f8')])

Note that this is a recarray rather than an array. You could move the result in to regular numpy array by calling its constructor as np.array(df.to_records()).

回答 6

尝试这个：

a = numpy.asarray(df)

Try this:

a = numpy.asarray(df)

回答 7

这是我从pandas DataFrame制作结构数组的方法。

创建数据框

import pandas as pd
import numpy as np
import six

NaN = float('nan')
ID = [1, 2, 3, 4, 5, 6, 7]
A = [NaN, NaN, NaN, 0.1, 0.1, 0.1, 0.1]
B = [0.2, NaN, 0.2, 0.2, 0.2, NaN, NaN]
C = [NaN, 0.5, 0.5, NaN, 0.5, 0.5, NaN]
columns = {'A':A, 'B':B, 'C':C}
df = pd.DataFrame(columns, index=ID)
df.index.name = 'ID'
print(df)

      A    B    C
ID               
1   NaN  0.2  NaN
2   NaN  NaN  0.5
3   NaN  0.2  0.5
4   0.1  0.2  NaN
5   0.1  0.2  0.5
6   0.1  NaN  0.5
7   0.1  NaN  NaN

定义函数以从pandas DataFrame中创建一个numpy结构数组（而不是记录数组）。

def df_to_sarray(df):
    """
    Convert a pandas DataFrame object to a numpy structured array.
    This is functionally equivalent to but more efficient than
    np.array(df.to_array())

    :param df: the data frame to convert
    :return: a numpy structured array representation of df
    """

    v = df.values
    cols = df.columns

    if six.PY2:  # python 2 needs .encode() but 3 does not
        types = [(cols[i].encode(), df[k].dtype.type) for (i, k) in enumerate(cols)]
    else:
        types = [(cols[i], df[k].dtype.type) for (i, k) in enumerate(cols)]
    dtype = np.dtype(types)
    z = np.zeros(v.shape[0], dtype)
    for (i, k) in enumerate(z.dtype.names):
        z[k] = v[:, i]
    return z

使用reset_index使包括索引作为其数据的一部分，新的数据帧。将该数据帧转换为结构数组。

sa = df_to_sarray(df.reset_index())
sa

array([(1L, nan, 0.2, nan), (2L, nan, nan, 0.5), (3L, nan, 0.2, 0.5),
       (4L, 0.1, 0.2, nan), (5L, 0.1, 0.2, 0.5), (6L, 0.1, nan, 0.5),
       (7L, 0.1, nan, nan)], 
      dtype=[('ID', '<i8'), ('A', '<f8'), ('B', '<f8'), ('C', '<f8')])

编辑：更新df_to_sarray以避免错误调用.encode（）与Python 3.感谢约瑟夫·加尔文和宁静为他们的意见和解决方案。

Here is my approach to making a structure array from a pandas DataFrame.

Create the data frame

import pandas as pd
import numpy as np
import six

NaN = float('nan')
ID = [1, 2, 3, 4, 5, 6, 7]
A = [NaN, NaN, NaN, 0.1, 0.1, 0.1, 0.1]
B = [0.2, NaN, 0.2, 0.2, 0.2, NaN, NaN]
C = [NaN, 0.5, 0.5, NaN, 0.5, 0.5, NaN]
columns = {'A':A, 'B':B, 'C':C}
df = pd.DataFrame(columns, index=ID)
df.index.name = 'ID'
print(df)

      A    B    C
ID               
1   NaN  0.2  NaN
2   NaN  NaN  0.5
3   NaN  0.2  0.5
4   0.1  0.2  NaN
5   0.1  0.2  0.5
6   0.1  NaN  0.5
7   0.1  NaN  NaN

Define function to make a numpy structure array (not a record array) from a pandas DataFrame.

def df_to_sarray(df):
    """
    Convert a pandas DataFrame object to a numpy structured array.
    This is functionally equivalent to but more efficient than
    np.array(df.to_array())

    :param df: the data frame to convert
    :return: a numpy structured array representation of df
    """

    v = df.values
    cols = df.columns

    if six.PY2:  # python 2 needs .encode() but 3 does not
        types = [(cols[i].encode(), df[k].dtype.type) for (i, k) in enumerate(cols)]
    else:
        types = [(cols[i], df[k].dtype.type) for (i, k) in enumerate(cols)]
    dtype = np.dtype(types)
    z = np.zeros(v.shape[0], dtype)
    for (i, k) in enumerate(z.dtype.names):
        z[k] = v[:, i]
    return z

Use reset_index to make a new data frame that includes the index as part of its data. Convert that data frame to a structure array.

sa = df_to_sarray(df.reset_index())
sa

array([(1L, nan, 0.2, nan), (2L, nan, nan, 0.5), (3L, nan, 0.2, 0.5),
       (4L, 0.1, 0.2, nan), (5L, 0.1, 0.2, 0.5), (6L, 0.1, nan, 0.5),
       (7L, 0.1, nan, nan)], 
      dtype=[('ID', '<i8'), ('A', '<f8'), ('B', '<f8'), ('C', '<f8')])

EDIT: Updated df_to_sarray to avoid error calling .encode() with python 3. Thanks to Joseph Garvin and halcyon for their comment and solution.

回答 8

将数据帧转换为其Numpy数组表示形式的两种方法。

mah_np_array = df.as_matrix(columns=None)
mah_np_array = df.values

Doc：https : //pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.as_matrix.html

Two ways to convert the data-frame to its Numpy-array representation.

mah_np_array = df.as_matrix(columns=None)
mah_np_array = df.values

Doc: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.as_matrix.html

回答 9

示例DataFrame的更简单方法：

df

         gbm       nnet        reg
0  12.097439  12.047437  12.100953
1  12.109811  12.070209  12.095288
2  11.720734  11.622139  11.740523
3  11.824557  11.926414  11.926527
4  11.800868  11.727730  11.729737
5  12.490984  12.502440  12.530894

采用：

np.array(df.to_records().view(type=np.matrix))

得到：

array([[(0, 12.097439  , 12.047437, 12.10095324),
        (1, 12.10981081, 12.070209, 12.09528824),
        (2, 11.72073428, 11.622139, 11.74052253),
        (3, 11.82455653, 11.926414, 11.92652727),
        (4, 11.80086775, 11.72773 , 11.72973699),
        (5, 12.49098389, 12.50244 , 12.53089367)]],
dtype=(numpy.record, [('index', '<i8'), ('gbm', '<f8'), ('nnet', '<f4'),
       ('reg', '<f8')]))

A Simpler Way for Example DataFrame:

df

         gbm       nnet        reg
0  12.097439  12.047437  12.100953
1  12.109811  12.070209  12.095288
2  11.720734  11.622139  11.740523
3  11.824557  11.926414  11.926527
4  11.800868  11.727730  11.729737
5  12.490984  12.502440  12.530894

USE:

np.array(df.to_records().view(type=np.matrix))

GET:

array([[(0, 12.097439  , 12.047437, 12.10095324),
        (1, 12.10981081, 12.070209, 12.09528824),
        (2, 11.72073428, 11.622139, 11.74052253),
        (3, 11.82455653, 11.926414, 11.92652727),
        (4, 11.80086775, 11.72773 , 11.72973699),
        (5, 12.49098389, 12.50244 , 12.53089367)]],
dtype=(numpy.record, [('index', '<i8'), ('gbm', '<f8'), ('nnet', '<f4'),
       ('reg', '<f8')]))

回答 10

从数据框导出到arcgis表时遇到了类似的问题，偶然发现了来自usgs的解决方案（https://my.usgs.gov/confluence/display/cdi/pandas.DataFrame+to+ArcGIS+Table）。简而言之，您的问题具有类似的解决方案：

df

      A    B    C
ID               
1   NaN  0.2  NaN
2   NaN  NaN  0.5
3   NaN  0.2  0.5
4   0.1  0.2  NaN
5   0.1  0.2  0.5
6   0.1  NaN  0.5
7   0.1  NaN  NaN

np_data = np.array(np.rec.fromrecords(df.values))
np_names = df.dtypes.index.tolist()
np_data.dtype.names = tuple([name.encode('UTF8') for name in np_names])

np_data

array([( nan,  0.2,  nan), ( nan,  nan,  0.5), ( nan,  0.2,  0.5),
       ( 0.1,  0.2,  nan), ( 0.1,  0.2,  0.5), ( 0.1,  nan,  0.5),
       ( 0.1,  nan,  nan)], 
      dtype=(numpy.record, [('A', '<f8'), ('B', '<f8'), ('C', '<f8')]))

Just had a similar problem when exporting from dataframe to arcgis table and stumbled on a solution from usgs (https://my.usgs.gov/confluence/display/cdi/pandas.DataFrame+to+ArcGIS+Table). In short your problem has a similar solution:

df

      A    B    C
ID               
1   NaN  0.2  NaN
2   NaN  NaN  0.5
3   NaN  0.2  0.5
4   0.1  0.2  NaN
5   0.1  0.2  0.5
6   0.1  NaN  0.5
7   0.1  NaN  NaN

np_data = np.array(np.rec.fromrecords(df.values))
np_names = df.dtypes.index.tolist()
np_data.dtype.names = tuple([name.encode('UTF8') for name in np_names])

np_data

array([( nan,  0.2,  nan), ( nan,  nan,  0.5), ( nan,  0.2,  0.5),
       ( 0.1,  0.2,  nan), ( 0.1,  0.2,  0.5), ( 0.1,  nan,  0.5),
       ( 0.1,  nan,  nan)], 
      dtype=(numpy.record, [('A', '<f8'), ('B', '<f8'), ('C', '<f8')]))

回答 11

我经历了以上答案。“ as_matrix（） ”方法可以使用，但是现在已经过时了。对我来说，有效的是“ .to_numpy（） ”。

这将返回一个多维数组。如果您要从excel工作表中读取数据，并且需要从任何索引访问数据，则我会更喜欢使用此方法。希望这可以帮助：）

I went through the answers above. The “as_matrix()” method works but its obsolete now. For me, What worked was “.to_numpy()“.

This returns a multidimensional array. I’ll prefer using this method if you’re reading data from excel sheet and you need to access data from any index. Hope this helps :)

回答 12

除了陨石的答案，我找到了代码

df.index = df.index.astype('i8')

对我不起作用。因此，我将代码放在这里，以方便陷入此问题的其他人。

city_cluster_df = pd.read_csv(text_filepath, encoding='utf-8')
# the field 'city_en' is a string, when converted to Numpy array, it will be an object
city_cluster_arr = city_cluster_df[['city_en','lat','lon','cluster','cluster_filtered']].to_records()
descr=city_cluster_arr.dtype.descr
# change the field 'city_en' to string type (the index for 'city_en' here is 1 because before the field is the row index of dataframe)
descr[1]=(descr[1][0], "S20")
newArr=city_cluster_arr.astype(np.dtype(descr))

Further to meteore’s answer, I found the code

df.index = df.index.astype('i8')

doesn’t work for me. So I put my code here for the convenience of others stuck with this issue.

city_cluster_df = pd.read_csv(text_filepath, encoding='utf-8')
# the field 'city_en' is a string, when converted to Numpy array, it will be an object
city_cluster_arr = city_cluster_df[['city_en','lat','lon','cluster','cluster_filtered']].to_records()
descr=city_cluster_arr.dtype.descr
# change the field 'city_en' to string type (the index for 'city_en' here is 1 because before the field is the row index of dataframe)
descr[1]=(descr[1][0], "S20")
newArr=city_cluster_arr.astype(np.dtype(descr))

回答 13

将数据帧转换为numpy数组的简单方法：

import pandas as pd
df = pd.DataFrame({"A": [1, 2], "B": [3, 4]})
df_to_array = df.to_numpy()
array([[1, 3],
   [2, 4]])

鼓励使用to_numpy来保持一致性。

参考：https : //pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_numpy.html

A simple way to convert dataframe to numpy array:

import pandas as pd
df = pd.DataFrame({"A": [1, 2], "B": [3, 4]})
df_to_array = df.to_numpy()
array([[1, 3],
   [2, 4]])

Use of to_numpy is encouraged to preserve consistency.

Reference: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_numpy.html

回答 14

尝试这个：

np.array(df) 

array([['ID', nan, nan, nan],
   ['1', nan, 0.2, nan],
   ['2', nan, nan, 0.5],
   ['3', nan, 0.2, 0.5],
   ['4', 0.1, 0.2, nan],
   ['5', 0.1, 0.2, 0.5],
   ['6', 0.1, nan, 0.5],
   ['7', 0.1, nan, nan]], dtype=object)

有关更多信息，请访问：[ https://docs.scipy.org/doc/numpy/reference/generated/numpy.array.html] 对numpy 1.16.5和pandas 0.25.2有效。

Try this:

np.array(df) 

array([['ID', nan, nan, nan],
   ['1', nan, 0.2, nan],
   ['2', nan, nan, 0.5],
   ['3', nan, 0.2, 0.5],
   ['4', 0.1, 0.2, nan],
   ['5', 0.1, 0.2, 0.5],
   ['6', 0.1, nan, 0.5],
   ['7', 0.1, nan, nan]], dtype=object)

Some more information at: [https://docs.scipy.org/doc/numpy/reference/generated/numpy.array.html] Valid for numpy 1.16.5 and pandas 0.25.2.

知识问答

如何访问NumPy多维数组的第i列？

2021年7月25日 Python实用宝典

问题：如何访问NumPy多维数组的第i列？

假设我有：

test = numpy.array([[1, 2], [3, 4], [5, 6]])

test[i]使我得到数组的第i行（例如[1, 2]）。如何访问第ith列？（例如[1, 3, 5]）。另外，这将是一项昂贵的操作吗？

Suppose I have:

test = numpy.array([[1, 2], [3, 4], [5, 6]])

test[i] gets me ith line of the array (eg [1, 2]). How can I access the ith column? (eg [1, 3, 5]). Also, would this be an expensive operation?

回答 0

>>> test[:,0]
array([1, 3, 5])

同样，

>>> test[1,:]
array([3, 4])

使您可以访问行。NumPy参考资料的第1.4节（索引）对此进行了介绍。这很快，至少以我的经验而言。它肯定比循环访问每个元素要快得多。

>>> test[:,0]
array([1, 3, 5])

Similarly,

>>> test[1,:]
array([3, 4])

lets you access rows. This is covered in Section 1.4 (Indexing) of the NumPy reference. This is quick, at least in my experience. It’s certainly much quicker than accessing each element in a loop.

回答 1

如果您想一次访问多个列，则可以执行以下操作：

>>> test = np.arange(9).reshape((3,3))
>>> test
array([[0, 1, 2],
       [3, 4, 5],
       [6, 7, 8]])
>>> test[:,[0,2]]
array([[0, 2],
       [3, 5],
       [6, 8]])

And if you want to access more than one column at a time you could do:

>>> test = np.arange(9).reshape((3,3))
>>> test
array([[0, 1, 2],
       [3, 4, 5],
       [6, 7, 8]])
>>> test[:,[0,2]]
array([[0, 2],
       [3, 5],
       [6, 8]])

回答 2

>>> test[:,0]
array([1, 3, 5])

该命令为您提供了行向量，如果您只想在其上循环，就可以了，但是如果您要与其他尺寸为3xN的数组进行堆叠，则可以

ValueError: all the input arrays must have same number of dimensions

而

>>> test[:,[0]]
array([[1],
       [3],
       [5]])

为您提供列向量，以便您可以进行串联或hstack操作。

例如

>>> np.hstack((test, test[:,[0]]))
array([[1, 2, 1],
       [3, 4, 3],
       [5, 6, 5]])

>>> test[:,0]
array([1, 3, 5])

this command gives you a row vector, if you just want to loop over it, it’s fine, but if you want to hstack with some other array with dimension 3xN, you will have

ValueError: all the input arrays must have same number of dimensions

while

>>> test[:,[0]]
array([[1],
       [3],
       [5]])

gives you a column vector, so that you can do concatenate or hstack operation.

e.g.

>>> np.hstack((test, test[:,[0]]))
array([[1, 2, 1],
       [3, 4, 3],
       [5, 6, 5]])

回答 3

您还可以转置并返回一行：

In [4]: test.T[0]
Out[4]: array([1, 3, 5])

You could also transpose and return a row:

In [4]: test.T[0]
Out[4]: array([1, 3, 5])

回答 4

要获得几个独立的列，只需：

> test[:,[0,2]]

你会得到第0列和第2列

To get several and indepent columns, just:

> test[:,[0,2]]

you will get colums 0 and 2

回答 5

尽管问题已得到回答，但让我提及一些细微差别。

假设您对数组的第一列感兴趣

arr = numpy.array([[1, 2],
                   [3, 4],
                   [5, 6]])

从其他答案中已经知道，要以“行向量”（shape的数组(3,)）的形式获取它，可以使用切片：

arr_c1_ref = arr[:, 1]  # creates a reference to the 1st column of the arr
arr_c1_copy = arr[:, 1].copy()  # creates a copy of the 1st column of the arr

要检查一个数组是视图还是另一个数组的副本，可以执行以下操作：

arr_c1_ref.base is arr  # True
arr_c1_copy.base is arr  # False

参见ndarray.base。

除了两者之间的明显区别（修改arr_c1_ref将影响arr），遍历它们中每一个的字节步数也不同：

arr_c1_ref.strides[0]  # 8 bytes
arr_c1_copy.strides[0]  # 4 bytes

看大步。为什么这很重要？假设您有一个很大的数组，A而不是arr：

A = np.random.randint(2, size=(10000,10000), dtype='int32')
A_c1_ref = A[:, 1] 
A_c1_copy = A[:, 1].copy()

并且您要计算第一列的所有元素的总和，即A_c1_ref.sum()或A_c1_copy.sum()。使用复制的版本要快得多：

%timeit A_c1_ref.sum()  # ~248 µs
%timeit A_c1_copy.sum()  # ~12.8 µs

这是由于前面提到的跨步数不同：

A_c1_ref.strides[0]  # 40000 bytes
A_c1_copy.strides[0]  # 4 bytes

尽管使用列副本似乎更好，但由于创建副本需要时间并使用更多的内存（在这种情况下，我花了大约200 µs的时间来创建副本）并不总是正确的。 A_c1_copy）。但是，如果我们首先需要复制，或者需要在数组的特定列上执行许多不同的操作，并且可以牺牲内存以提高速度，那么复制就可以了。

如果我们有兴趣主要使用列，最好以列大（’F’）顺序而不是行大（’C’）顺序创建数组（这是默认设置）），然后像以前一样进行切片以获取一列而不复制它：

A = np.asfortranarray(A)  # or np.array(A, order='F')
A_c1_ref = A[:, 1]
A_c1_ref.strides[0]  # 4 bytes
%timeit A_c1_ref.sum()  # ~12.6 µs vs ~248 µs

现在，在列视图上执行求和运算（或其他任何运算）要快得多。

最后，让我注意到，转置数组并使用行切片与在原始数组上使用列切片相同，因为转置是通过交换原始数组的形状和步幅来完成的。

A.T[1,:].strides[0]  # 40000

Although the question has been answered, let me mention some nuances.

Let’s say you are interested in the first column of the array

arr = numpy.array([[1, 2],
                   [3, 4],
                   [5, 6]])

As you already know from other answers, to get it in the form of “row vector” (array of shape (3,)), you use slicing:

arr_c1_ref = arr[:, 1]  # creates a reference to the 1st column of the arr
arr_c1_copy = arr[:, 1].copy()  # creates a copy of the 1st column of the arr

To check if an array is a view or a copy of another array you can do the following:

arr_c1_ref.base is arr  # True
arr_c1_copy.base is arr  # False

see ndarray.base.

Besides the obvious difference between the two (modifying arr_c1_ref will affect arr), the number of byte-steps for traversing each of them is different:

arr_c1_ref.strides[0]  # 8 bytes
arr_c1_copy.strides[0]  # 4 bytes

see strides. Why is this important? Imagine that you have a very big array A instead of the arr:

A = np.random.randint(2, size=(10000,10000), dtype='int32')
A_c1_ref = A[:, 1] 
A_c1_copy = A[:, 1].copy()

and you want to compute the sum of all the elements of the first column, i.e. A_c1_ref.sum() or A_c1_copy.sum(). Using the copied version is much faster:

%timeit A_c1_ref.sum()  # ~248 µs
%timeit A_c1_copy.sum()  # ~12.8 µs

This is due to the different number of strides mentioned before:

A_c1_ref.strides[0]  # 40000 bytes
A_c1_copy.strides[0]  # 4 bytes

Although it might seem that using column copies is better, it is not always true for the reason that making a copy takes time and uses more memory (in this case it took me approx. 200 µs to create the A_c1_copy). However if we need the copy in the first place, or we need to do many different operations on a specific column of the array and we are ok with sacrificing memory for speed, then making a copy is the way to go.

In the case that we are interested in working mostly with columns, it could be a good idea to create our array in column-major (‘F’) order instead of the row-major (‘C’) order (which is the default), and then do the slicing as before to get a column without copying it:

A = np.asfortranarray(A)  # or np.array(A, order='F')
A_c1_ref = A[:, 1]
A_c1_ref.strides[0]  # 4 bytes
%timeit A_c1_ref.sum()  # ~12.6 µs vs ~248 µs

Now, performing the sum operation (or any other) on a column-view is much faster.

Finally let me note that transposing an array and using row-slicing is the same as using the column-slicing on the original array, because transposing is done by just swapping the shape and the strides of the original array.

A.T[1,:].strides[0]  # 40000

回答 6

>>> test
array([[0, 1, 2, 3, 4],
       [5, 6, 7, 8, 9]])

>>> ncol = test.shape[1]
>>> ncol
5L

然后，您可以通过以下方式选择第二至第四列：

>>> test[0:, 1:(ncol - 1)]
array([[1, 2, 3],
       [6, 7, 8]])

>>> test
array([[0, 1, 2, 3, 4],
       [5, 6, 7, 8, 9]])

>>> ncol = test.shape[1]
>>> ncol
5L

Then you can select the 2nd – 4th column this way:

>>> test[0:, 1:(ncol - 1)]
array([[1, 2, 3],
       [6, 7, 8]])