熊猫:在数据框中创建两个新列,并使用从现有列中计算出的值

问题:熊猫:在数据框中创建两个新列,并使用从现有列中计算出的值

我正在使用pandas库,我想将两个新列添加到df具有n列(n> 0)的数据框中。
这些新列是由于将函数应用于数据框中的某一列而产生的。

要应用的功能如下:

def calculate(x):
    ...operate...
    return z, y

为仅返回值的函数创建新列的一种方法是:

df['new_col']) = df['column_A'].map(a_function)

因此,我想要的但尝试失败的(*)是这样的:

(df['new_col_zetas'], df['new_col_ys']) = df['column_A'].map(calculate)

实现此目的的最佳方法是什么?我毫无头绪地扫描了文档

** df['column_A'].map(calculate)返回一个熊猫系列,每个项目都由一个元组z,y组成。尝试将其分配给两个数据框列会产生ValueError。*

I am working with the pandas library and I want to add two new columns to a dataframe df with n columns (n > 0).
These new columns result from the application of a function to one of the columns in the dataframe.

The function to apply is like:

def calculate(x):
    ...operate...
    return z, y

One method for creating a new column for a function returning only a value is:

df['new_col']) = df['column_A'].map(a_function)

So, what I want, and tried unsuccesfully (*), is something like:

(df['new_col_zetas'], df['new_col_ys']) = df['column_A'].map(calculate)

What the best way to accomplish this could be ? I scanned the documentation with no clue.

**df['column_A'].map(calculate) returns a pandas Series each item consisting of a tuple z, y. And trying to assign this to two dataframe columns produces a ValueError.*


回答 0

我只用zip

In [1]: from pandas import *

In [2]: def calculate(x):
   ...:     return x*2, x*3
   ...: 

In [3]: df = DataFrame({'a': [1,2,3], 'b': [2,3,4]})

In [4]: df
Out[4]: 
   a  b
0  1  2
1  2  3
2  3  4

In [5]: df["A1"], df["A2"] = zip(*df["a"].map(calculate))

In [6]: df
Out[6]: 
   a  b  A1  A2
0  1  2   2   3
1  2  3   4   6
2  3  4   6   9

I’d just use zip:

In [1]: from pandas import *

In [2]: def calculate(x):
   ...:     return x*2, x*3
   ...: 

In [3]: df = DataFrame({'a': [1,2,3], 'b': [2,3,4]})

In [4]: df
Out[4]: 
   a  b
0  1  2
1  2  3
2  3  4

In [5]: df["A1"], df["A2"] = zip(*df["a"].map(calculate))

In [6]: df
Out[6]: 
   a  b  A1  A2
0  1  2   2   3
1  2  3   4   6
2  3  4   6   9

回答 1

我认为最佳答案是有缺陷的。希望没有人使用来将所有大熊猫大量导入其命名空间from pandas import *。同样,在将map方法传递给字典或系列时,应保留该方法的使用时间。它可以带一个函数,但这就是apply它的用途。

所以,如果您必须使用上述方法,我会这样写

df["A1"], df["A2"] = zip(*df["a"].apply(calculate))

实际上,这里没有理由使用zip。您可以简单地做到这一点:

df["A1"], df["A2"] = calculate(df['a'])

在较大的DataFrame上,第二种方法也快得多

df = pd.DataFrame({'a': [1,2,3] * 100000, 'b': [2,3,4] * 100000})

创建了300,000行的DataFrame

%timeit df["A1"], df["A2"] = calculate(df['a'])
2.65 ms ± 92.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%timeit df["A1"], df["A2"] = zip(*df["a"].apply(calculate))
159 ms ± 5.24 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

比拉链快60倍


通常,避免使用Apply

Apply通常不会比遍历Python列表快多少。让我们测试一个for循环的性能,以执行与上述相同的操作

%%timeit
A1, A2 = [], []
for val in df['a']:
    A1.append(val**2)
    A2.append(val**3)

df['A1'] = A1
df['A2'] = A2

298 ms ± 7.14 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

因此,这是缓慢的两倍,这并不是可怕的性能下降,但是如果我们对上述内容进行cythonize,我们将获得更好的性能。假设您正在使用ipython:

%load_ext cython

%%cython
cpdef power(vals):
    A1, A2 = [], []
    cdef double val
    for val in vals:
        A1.append(val**2)
        A2.append(val**3)

    return A1, A2

%timeit df['A1'], df['A2'] = power(df['a'])
72.7 ms ± 2.16 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

直接分配而不适用

如果使用直接矢量化操作,则可以进一步提高速度。

%timeit df['A1'], df['A2'] = df['a'] ** 2, df['a'] ** 3
5.13 ms ± 320 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

这利用了NumPy极其快速的矢量化操作的优势,而不是我们的循环。现在,我们的速度比原始速度提高了30倍。


最简单的速度测试 apply

上面的示例应该清楚地显示出速度有多慢apply,但是正是如此,让我们来看一个最基本的示例。让我们平方一千万个带和不带数字的序列

s = pd.Series(np.random.rand(10000000))

%timeit s.apply(calc)
3.3 s ± 57.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

如果不套用,则速度提高了50倍

%timeit s ** 2
66 ms ± 2 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

The top answer is flawed in my opinion. Hopefully, no one is mass importing all of pandas into their namespace with from pandas import *. Also, the map method should be reserved for those times when passing it a dictionary or Series. It can take a function but this is what apply is used for.

So, if you must use the above approach, I would write it like this

df["A1"], df["A2"] = zip(*df["a"].apply(calculate))

There’s actually no reason to use zip here. You can simply do this:

df["A1"], df["A2"] = calculate(df['a'])

This second method is also much faster on larger DataFrames

df = pd.DataFrame({'a': [1,2,3] * 100000, 'b': [2,3,4] * 100000})

DataFrame created with 300,000 rows

%timeit df["A1"], df["A2"] = calculate(df['a'])
2.65 ms ± 92.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%timeit df["A1"], df["A2"] = zip(*df["a"].apply(calculate))
159 ms ± 5.24 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

60x faster than zip


In general, avoid using apply

Apply is generally not much faster than iterating over a Python list. Let’s test the performance of a for-loop to do the same thing as above

%%timeit
A1, A2 = [], []
for val in df['a']:
    A1.append(val**2)
    A2.append(val**3)

df['A1'] = A1
df['A2'] = A2

298 ms ± 7.14 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

So this is twice as slow which isn’t a terrible performance regression, but if we cythonize the above, we get much better performance. Assuming, you are using ipython:

%load_ext cython

%%cython
cpdef power(vals):
    A1, A2 = [], []
    cdef double val
    for val in vals:
        A1.append(val**2)
        A2.append(val**3)

    return A1, A2

%timeit df['A1'], df['A2'] = power(df['a'])
72.7 ms ± 2.16 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

Directly assigning without apply

You can get even greater speed improvements if you use the direct vectorized operations.

%timeit df['A1'], df['A2'] = df['a'] ** 2, df['a'] ** 3
5.13 ms ± 320 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

This takes advantage of NumPy’s extremely fast vectorized operations instead of our loops. We now have a 30x speedup over the original.


The simplest speed test with apply

The above example should clearly show how slow apply can be, but just so its extra clear let’s look at the most basic example. Let’s square a Series of 10 million numbers with and without apply

s = pd.Series(np.random.rand(10000000))

%timeit s.apply(calc)
3.3 s ± 57.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Without apply is 50x faster

%timeit s ** 2
66 ms ± 2 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)