熊猫:索引数据框时出现多种情况-意外行为

问题:熊猫:索引数据框时出现多种情况-意外行为

我正在按两列中的值过滤数据框中的行。

出于某种原因,OR运算符的行为类似于我期望AND运算符的行为,反之亦然。

我的测试代码:

import pandas as pd

df = pd.DataFrame({'a': range(5), 'b': range(5) })

# let's insert some -1 values
df['a'][1] = -1
df['b'][1] = -1
df['a'][3] = -1
df['b'][4] = -1

df1 = df[(df.a != -1) & (df.b != -1)]
df2 = df[(df.a != -1) | (df.b != -1)]

print pd.concat([df, df1, df2], axis=1,
                keys = [ 'original df', 'using AND (&)', 'using OR (|)',])

结果:

      original df      using AND (&)      using OR (|)    
             a  b              a   b             a   b
0            0  0              0   0             0   0
1           -1 -1            NaN NaN           NaN NaN
2            2  2              2   2             2   2
3           -1  3            NaN NaN            -1   3
4            4 -1            NaN NaN             4  -1

[5 rows x 6 columns]

如您所见,AND运算符将删除其中至少一个等于的每一行-1。另一方面,OR运算符要求两个值相等-1才能删除它们。我期望结果恰好相反。任何人都可以解释这种行为吗?

我正在使用熊猫0.13.1。

I am filtering rows in a dataframe by values in two columns.

For some reason the OR operator behaves like I would expect AND operator to behave and vice versa.

My test code:

import pandas as pd

df = pd.DataFrame({'a': range(5), 'b': range(5) })

# let's insert some -1 values
df['a'][1] = -1
df['b'][1] = -1
df['a'][3] = -1
df['b'][4] = -1

df1 = df[(df.a != -1) & (df.b != -1)]
df2 = df[(df.a != -1) | (df.b != -1)]

print pd.concat([df, df1, df2], axis=1,
                keys = [ 'original df', 'using AND (&)', 'using OR (|)',])

And the result:

      original df      using AND (&)      using OR (|)    
             a  b              a   b             a   b
0            0  0              0   0             0   0
1           -1 -1            NaN NaN           NaN NaN
2            2  2              2   2             2   2
3           -1  3            NaN NaN            -1   3
4            4 -1            NaN NaN             4  -1

[5 rows x 6 columns]

As you can see, the AND operator drops every row in which at least one value equals -1. On the other hand, the OR operator requires both values to be equal to -1 to drop them. I would expect exactly the opposite result. Could anyone explain this behavior, please?

I am using pandas 0.13.1.


回答 0

如您所见,AND运算符会删除每一行中至少有一个等于-1的值。另一方面,OR运算符要求两个值都等于-1才能删除它们。

那就对了。请记住,您是根据要保留的内容而不是要丢弃的内容来写条件。对于df1

df1 = df[(df.a != -1) & (df.b != -1)]

您说的是“保留其中df.a不是-1且df.b不是-1的行”,这与删除其中至少一个值为-1的每一行相同。

对于df2

df2 = df[(df.a != -1) | (df.b != -1)]

您说的是“保留其中任一行df.adf.b都不为-1的行”,这与删除两个值均为-1的行相同。

PS:连锁访问会给df['a'][1] = -1您带来麻烦。最好养成使用.loc和的习惯.iloc

As you can see, the AND operator drops every row in which at least one value equals -1. On the other hand, the OR operator requires both values to be equal to -1 to drop them.

That’s right. Remember that you’re writing the condition in terms of what you want to keep, not in terms of what you want to drop. For df1:

df1 = df[(df.a != -1) & (df.b != -1)]

You’re saying “keep the rows in which df.a isn’t -1 and df.b isn’t -1″, which is the same as dropping every row in which at least one value is -1.

For df2:

df2 = df[(df.a != -1) | (df.b != -1)]

You’re saying “keep the rows in which either df.a or df.b is not -1″, which is the same as dropping rows where both values are -1.

PS: chained access like df['a'][1] = -1 can get you into trouble. It’s better to get into the habit of using .loc and .iloc.


回答 1

您可以使用query(),即:

df_filtered = df.query('a == 4 & b != 2')

You can use query(), i.e.:

df_filtered = df.query('a == 4 & b != 2')

回答 2

这里有一些数学逻辑理论

“ NOT a AND NOT b”“ NOT(a OR b)”相同,因此:

“ a NOT -1 AND b NOT -1” 等同于 “ NOT(a为-1 OR b为-1)”,与“(a is -1 OR b为-1)”的(补数)相反。

因此,如果您想要完全相反的结果,则df1和df2应如下所示:

df1 = df[(df.a != -1) & (df.b != -1)]
df2 = df[(df.a == -1) | (df.b == -1)]

A little mathematical logic theory here:

“NOT a AND NOT b” is the same as “NOT (a OR b)”, so:

“a NOT -1 AND b NOT -1” is equivalent of “NOT (a is -1 OR b is -1)”, which is opposite (Complement) of “(a is -1 OR b is -1)”.

So if you want exact opposite result, df1 and df2 should be as below:

df1 = df[(df.a != -1) & (df.b != -1)]
df2 = df[(df.a == -1) | (df.b == -1)]