问题：在Python Pandas中删除所有重复的行
该pandas drop_duplicates功能非常适合“统一”数据帧。但是，要传递的关键字参数之一是take_last=True或take_last=False，而我想删除所有在列的子集中重复的行。这可能吗？
A B C
0 foo 0 A
1 foo 1 A
2 foo 1 B
3 bar 1 A
作为一个例子，我想下降匹配列的行A和C所以这应该丢弃的行0和1。
The pandas drop_duplicates function is great for “uniquifying” a dataframe. However, one of the keyword arguments to pass is take_last=True or take_last=False, while I would like to drop all rows which are duplicates across a subset of columns. Is this possible?
A B C
0 foo 0 A
1 foo 1 A
2 foo 1 B
3 bar 1 A
As an example, I would like to drop rows which match on columns A and C so this should drop rows 0 and 1.
回答 0
现在，通过drop_duplicates和keep参数，这在熊猫中要容易得多。
import pandas as pd
df = pd.DataFrame({"A":["foo", "foo", "foo", "bar"], "B":[0,1,1,1], "C":["A","A","B","A"]})
df.drop_duplicates(subset=['A', 'C'], keep=False)
This is much easier in pandas now with drop_duplicates and the keep parameter.
import pandas as pd
df = pd.DataFrame({"A":["foo", "foo", "foo", "bar"], "B":[0,1,1,1], "C":["A","A","B","A"]})
df.drop_duplicates(subset=['A', 'C'], keep=False)
回答 1
只想添加到本对drop_duplicates的答案中：
keep ：{‘first’，’last’，False}，默认为’first’
first：删除第一次出现的重复项。
last：除去最后一次出现的重复项。
False：删除所有重复项。
因此，将其设置keep为False将为您提供所需的答案。
DataFrame.drop_duplicates（* args，** kwargs）返回删除了重复行的DataFrame，可以选择仅考虑某些列
参数：subset：列标签或标签序列，可选的仅考虑某些列来标识重复项，默认情况下使用所有列keep：{‘first’，’last’，False}，默认为’first’first：删除重复项，除了第一次出现。last：除去最后一次出现的重复项。False：删除所有重复项。take_last：已弃用，就位：布尔值，默认为False是否将副本放置在适当位置或返回副本cols：仅kwargs子集的参数[不建议使用]返回：重复数据删除：DataFrame
Just want to add to Ben’s answer on drop_duplicates:
keep : {‘first’, ‘last’, False}, default ‘first’
first : Drop duplicates except for the first occurrence.
last : Drop duplicates except for the last occurrence.
False : Drop all duplicates.
So setting keep to False will give you desired answer.
DataFrame.drop_duplicates(*args, **kwargs) Return DataFrame with
duplicate rows removed, optionally only considering certain columns
Parameters: subset : column label or sequence of labels, optional
Only consider certain columns for identifying duplicates, by default
use all of the columns keep : {‘first’, ‘last’, False}, default
‘first’ first : Drop duplicates except for the first occurrence. last
: Drop duplicates except for the last occurrence. False : Drop all
duplicates. take_last : deprecated inplace : boolean, default False
Whether to drop duplicates in place or to return a copy cols : kwargs
only argument of subset [deprecated] Returns: deduplicated :
DataFrame
回答 2
如果要将结果存储在另一个数据集中：
df.drop_duplicates(keep=False)
要么
df.drop_duplicates(keep=False, inplace=False)
如果需要更新相同的数据集：
df.drop_duplicates(keep=False, inplace=True)
上面的示例将删除所有重复项并保留一个，类似于DISTINCT *SQL
If you want result to be stored in another dataset:
df.drop_duplicates(keep=False)
or
df.drop_duplicates(keep=False, inplace=False)
If same dataset needs to be updated:
df.drop_duplicates(keep=False, inplace=True)
Above examples will remove all duplicates and keep one, similar to DISTINCT * in SQL
回答 3
使用groupby和filter
import pandas as pd
df = pd.DataFrame({"A":["foo", "foo", "foo", "bar"], "B":[0,1,1,1], "C":["A","A","B","A"]})
df.groupby(["A", "C"]).filter(lambda df:df.shape[0] == 1)
use groupby and filter
import pandas as pd
df = pd.DataFrame({"A":["foo", "foo", "foo", "bar"], "B":[0,1,1,1], "C":["A","A","B","A"]})
df.groupby(["A", "C"]).filter(lambda df:df.shape[0] == 1)
回答 4
实际上，仅删除第0行和第1行（保留包含匹配的A和C的所有观察值）：
In [335]:
df['AC']=df.A+df.C
In [336]:
print df.drop_duplicates('C', take_last=True) #this dataset is a special case, in general, one may need to first drop_duplicates by 'c' and then by 'a'.
A B C AC
2 foo 1 B fooB
3 bar 1 A barA
[2 rows x 4 columns]
但是我怀疑您真正想要的是什么（保留包含匹配的A和C的观察值）：
In [337]:
print df.drop_duplicates('AC')
A B C AC
0 foo 0 A fooA
2 foo 1 B fooB
3 bar 1 A barA
[3 rows x 4 columns]
编辑：
因此，现在更加清楚了：
In [352]:
DG=df.groupby(['A', 'C'])
print pd.concat([DG.get_group(item) for item, value in DG.groups.items() if len(value)==1])
A B C
2 foo 1 B
3 bar 1 A
[2 rows x 3 columns]
Actually, drop rows 0 and 1 only requires (any observations containing matched A and C is kept.):
In [335]:
df['AC']=df.A+df.C
In [336]:
print df.drop_duplicates('C', take_last=True) #this dataset is a special case, in general, one may need to first drop_duplicates by 'c' and then by 'a'.
A B C AC
2 foo 1 B fooB
3 bar 1 A barA
[2 rows x 4 columns]
But I suspect what you really want is this (one observation containing matched A and C is kept.):
In [337]:
print df.drop_duplicates('AC')
A B C AC
0 foo 0 A fooA
2 foo 1 B fooB
3 bar 1 A barA
[3 rows x 4 columns]
Edit:
Now it is much clearer, therefore:
In [352]:
DG=df.groupby(['A', 'C'])
print pd.concat([DG.get_group(item) for item, value in DG.groups.items() if len(value)==1])
A B C
2 foo 1 B
3 bar 1 A
[2 rows x 3 columns]
回答 5
试试这些各种各样的东西
df = pd.DataFrame({"A":["foo", "foo", "foo", "bar","foo"], "B":[0,1,1,1,1], "C":["A","A","B","A","A"]})
>>>df.drop_duplicates( "A" , keep='first')
要么
>>>df.drop_duplicates( keep='first')
要么
>>>df.drop_duplicates( keep='last')
Try these various things
df = pd.DataFrame({"A":["foo", "foo", "foo", "bar","foo"], "B":[0,1,1,1,1], "C":["A","A","B","A","A"]})
>>>df.drop_duplicates( "A" , keep='first')
or
>>>df.drop_duplicates( keep='first')
or
>>>df.drop_duplicates( keep='last')
