问题：Python分组依据
假设我有一组数据对，其中索引0是值，索引1是类型：
input = [
('11013331', 'KAT'),
('9085267', 'NOT'),
('5238761', 'ETH'),
('5349618', 'ETH'),
('11788544', 'NOT'),
('962142', 'ETH'),
('7795297', 'ETH'),
('7341464', 'ETH'),
('9843236', 'KAT'),
('5594916', 'ETH'),
('1550003', 'ETH')
]
我想按它们的类型（按第一个索引字符串）将它们分组，如下所示：
result = [
{
type:'KAT',
items: ['11013331', '9843236']
},
{
type:'NOT',
items: ['9085267', '11788544']
},
{
type:'ETH',
items: ['5238761', '962142', '7795297', '7341464', '5594916', '1550003']
}
]
如何有效地做到这一点？
Assume that I have a set of data pair where index 0 is the value and index 1 is the type:
input = [
('11013331', 'KAT'),
('9085267', 'NOT'),
('5238761', 'ETH'),
('5349618', 'ETH'),
('11788544', 'NOT'),
('962142', 'ETH'),
('7795297', 'ETH'),
('7341464', 'ETH'),
('9843236', 'KAT'),
('5594916', 'ETH'),
('1550003', 'ETH')
]
I want to group them by their type (by the 1st indexed string) as such:
result = [
{
type:'KAT',
items: ['11013331', '9843236']
},
{
type:'NOT',
items: ['9085267', '11788544']
},
{
type:'ETH',
items: ['5238761', '962142', '7795297', '7341464', '5594916', '1550003']
}
]
How can I achieve this in an efficient way?
回答 0
分两步完成。首先，创建字典。
>>> input = [('11013331', 'KAT'), ('9085267', 'NOT'), ('5238761', 'ETH'), ('5349618', 'ETH'), ('11788544', 'NOT'), ('962142', 'ETH'), ('7795297', 'ETH'), ('7341464', 'ETH'), ('9843236', 'KAT'), ('5594916', 'ETH'), ('1550003', 'ETH')]
>>> from collections import defaultdict
>>> res = defaultdict(list)
>>> for v, k in input: res[k].append(v)
...
然后，将该字典转换为预期的格式。
>>> [{'type':k, 'items':v} for k,v in res.items()]
[{'items': ['9085267', '11788544'], 'type': 'NOT'}, {'items': ['5238761', '5349618', '962142', '7795297', '7341464', '5594916', '1550003'], 'type': 'ETH'}, {'items': ['11013331', '9843236'], 'type': 'KAT'}]
使用itertools.groupby也可以，但是它要求输入首先被排序。
>>> sorted_input = sorted(input, key=itemgetter(1))
>>> groups = groupby(sorted_input, key=itemgetter(1))
>>> [{'type':k, 'items':[x[0] for x in v]} for k, v in groups]
[{'items': ['5238761', '5349618', '962142', '7795297', '7341464', '5594916', '1550003'], 'type': 'ETH'}, {'items': ['11013331', '9843236'], 'type': 'KAT'}, {'items': ['9085267', '11788544'], 'type': 'NOT'}]
请注意，这两个都不遵守键的原始顺序。如果需要保留订单，则需要一个OrderedDict。
>>> from collections import OrderedDict
>>> res = OrderedDict()
>>> for v, k in input:
... if k in res: res[k].append(v)
... else: res[k] = [v]
...
>>> [{'type':k, 'items':v} for k,v in res.items()]
[{'items': ['11013331', '9843236'], 'type': 'KAT'}, {'items': ['9085267', '11788544'], 'type': 'NOT'}, {'items': ['5238761', '5349618', '962142', '7795297', '7341464', '5594916', '1550003'], 'type': 'ETH'}]
Do it in 2 steps. First, create a dictionary.
>>> input = [('11013331', 'KAT'), ('9085267', 'NOT'), ('5238761', 'ETH'), ('5349618', 'ETH'), ('11788544', 'NOT'), ('962142', 'ETH'), ('7795297', 'ETH'), ('7341464', 'ETH'), ('9843236', 'KAT'), ('5594916', 'ETH'), ('1550003', 'ETH')]
>>> from collections import defaultdict
>>> res = defaultdict(list)
>>> for v, k in input: res[k].append(v)
...
Then, convert that dictionary into the expected format.
>>> [{'type':k, 'items':v} for k,v in res.items()]
[{'items': ['9085267', '11788544'], 'type': 'NOT'}, {'items': ['5238761', '5349618', '962142', '7795297', '7341464', '5594916', '1550003'], 'type': 'ETH'}, {'items': ['11013331', '9843236'], 'type': 'KAT'}]
It is also possible with itertools.groupby but it requires the input to be sorted first.
>>> sorted_input = sorted(input, key=itemgetter(1))
>>> groups = groupby(sorted_input, key=itemgetter(1))
>>> [{'type':k, 'items':[x[0] for x in v]} for k, v in groups]
[{'items': ['5238761', '5349618', '962142', '7795297', '7341464', '5594916', '1550003'], 'type': 'ETH'}, {'items': ['11013331', '9843236'], 'type': 'KAT'}, {'items': ['9085267', '11788544'], 'type': 'NOT'}]
Note both of these do not respect the original order of the keys. You need an OrderedDict if you need to keep the order.
>>> from collections import OrderedDict
>>> res = OrderedDict()
>>> for v, k in input:
... if k in res: res[k].append(v)
... else: res[k] = [v]
...
>>> [{'type':k, 'items':v} for k,v in res.items()]
[{'items': ['11013331', '9843236'], 'type': 'KAT'}, {'items': ['9085267', '11788544'], 'type': 'NOT'}, {'items': ['5238761', '5349618', '962142', '7795297', '7341464', '5594916', '1550003'], 'type': 'ETH'}]
回答 1
Python的内置itertools模块实际上具有一个groupbyfunction，但是为此，必须首先对要分组的元素进行排序，以使要分组的元素在列表中是连续的：
from operator import itemgetter
sortkeyfn = itemgetter(1)
input = [('11013331', 'KAT'), ('9085267', 'NOT'), ('5238761', 'ETH'),
('5349618', 'ETH'), ('11788544', 'NOT'), ('962142', 'ETH'), ('7795297', 'ETH'),
('7341464', 'ETH'), ('9843236', 'KAT'), ('5594916', 'ETH'), ('1550003', 'ETH')]
input.sort(key=sortkeyfn)
现在输入看起来像：
[('5238761', 'ETH'), ('5349618', 'ETH'), ('962142', 'ETH'), ('7795297', 'ETH'),
('7341464', 'ETH'), ('5594916', 'ETH'), ('1550003', 'ETH'), ('11013331', 'KAT'),
('9843236', 'KAT'), ('9085267', 'NOT'), ('11788544', 'NOT')]
groupby返回格式为的2元组序列(key, values_iterator)。我们想要的是将其转换为字典列表，其中“类型”是键，而“项目”是values_iterator返回的元组的第0个元素的列表。像这样：
from itertools import groupby
result = []
for key,valuesiter in groupby(input, key=sortkeyfn):
result.append(dict(type=key, items=list(v[0] for v in valuesiter)))
现在result包含您想要的字典，如您的问题所述。
但是，您可能会考虑仅对此做出一个单独的dict，按类型键入，每个值都包含值列表。在当前形式中，要查找特定类型的值，必须遍历列表以查找包含匹配的“ type”键的字典，然后从中获取“ items”元素。如果您使用单个词典而不是一个1项词典的列表，则可以通过在主词典中进行单键查找来查找特定类型的项目。使用groupby，这看起来像：
result = {}
for key,valuesiter in groupby(input, key=sortkeyfn):
result[key] = list(v[0] for v in valuesiter)
result现在包含此字典（这类似于res@KennyTM答案中的中间defaultdict）：
{'NOT': ['9085267', '11788544'],
'ETH': ['5238761', '5349618', '962142', '7795297', '7341464', '5594916', '1550003'],
'KAT': ['11013331', '9843236']}
（如果您希望将其减少为单层，则可以：
result = dict((key,list(v[0] for v in valuesiter)
for key,valuesiter in groupby(input, key=sortkeyfn))
或使用新奇的dict-comprehension形式：
result = {key:list(v[0] for v in valuesiter)
for key,valuesiter in groupby(input, key=sortkeyfn)}
Python’s built-in itertools module actually has a groupby function , but for that the elements to be grouped must first be sorted such that the elements to be grouped are contiguous in the list:
from operator import itemgetter
sortkeyfn = itemgetter(1)
input = [('11013331', 'KAT'), ('9085267', 'NOT'), ('5238761', 'ETH'),
('5349618', 'ETH'), ('11788544', 'NOT'), ('962142', 'ETH'), ('7795297', 'ETH'),
('7341464', 'ETH'), ('9843236', 'KAT'), ('5594916', 'ETH'), ('1550003', 'ETH')]
input.sort(key=sortkeyfn)
Now input looks like:
[('5238761', 'ETH'), ('5349618', 'ETH'), ('962142', 'ETH'), ('7795297', 'ETH'),
('7341464', 'ETH'), ('5594916', 'ETH'), ('1550003', 'ETH'), ('11013331', 'KAT'),
('9843236', 'KAT'), ('9085267', 'NOT'), ('11788544', 'NOT')]
groupby returns a sequence of 2-tuples, of the form (key, values_iterator). What we want is to turn this into a list of dicts where the ‘type’ is the key, and ‘items’ is a list of the 0’th elements of the tuples returned by the values_iterator. Like this:
from itertools import groupby
result = []
for key,valuesiter in groupby(input, key=sortkeyfn):
result.append(dict(type=key, items=list(v[0] for v in valuesiter)))
Now result contains your desired dict, as stated in your question.
You might consider, though, just making a single dict out of this, keyed by type, and each value containing the list of values. In your current form, to find the values for a particular type, you’ll have to iterate over the list to find the dict containing the matching ‘type’ key, and then get the ‘items’ element from it. If you use a single dict instead of a list of 1-item dicts, you can find the items for a particular type with a single keyed lookup into the master dict. Using groupby, this would look like:
result = {}
for key,valuesiter in groupby(input, key=sortkeyfn):
result[key] = list(v[0] for v in valuesiter)
result now contains this dict (this is similar to the intermediate res defaultdict in @KennyTM’s answer):
{'NOT': ['9085267', '11788544'],
'ETH': ['5238761', '5349618', '962142', '7795297', '7341464', '5594916', '1550003'],
'KAT': ['11013331', '9843236']}
(If you want to reduce this to a one-liner, you can:
result = dict((key,list(v[0] for v in valuesiter)
for key,valuesiter in groupby(input, key=sortkeyfn))
or using the newfangled dict-comprehension form:
result = {key:list(v[0] for v in valuesiter)
for key,valuesiter in groupby(input, key=sortkeyfn)}
回答 2
我也喜欢熊猫简单的分组。它功能强大，简单，最适合大型数据集
result = pandas.DataFrame(input).groupby(1).groups
I also liked pandas simple grouping. it’s powerful, simple and most adequate for large data set
result = pandas.DataFrame(input).groupby(1).groups
回答 3
此答案类似于@PaulMcG的答案，但不需要对输入进行排序。
对于那些进行函数式编程的人，groupBy可以将其写在一行中（不包括导入！），itertools.groupby与之不同的是，它不需要对输入进行排序：
from functools import reduce # import needed for python3; builtin in python2
from collections import defaultdict
def groupBy(key, seq):
return reduce(lambda grp, val: grp[key(val)].append(val) or grp, seq, defaultdict(list))
（之所以这样做... or grp，lambda是因为要reduce()使其正常工作，lambda需要返回其第一个参数；因为list.append()总是返回，None所以or意志总是返回grp。也就是说，它是一个黑客绕过Python的限制，即在拉姆达只能计算一个表达式。）
这将返回一个字典，该字典的键是通过评估给定的函数找到的，其值是按原始顺序列出的原始项目的列表。对于OP的示例，将其称为as groupBy(lambda pair: pair[1], input)将返回此字典：
{'KAT': [('11013331', 'KAT'), ('9843236', 'KAT')],
'NOT': [('9085267', 'NOT'), ('11788544', 'NOT')],
'ETH': [('5238761', 'ETH'), ('5349618', 'ETH'), ('962142', 'ETH'), ('7795297', 'ETH'), ('7341464', 'ETH'), ('5594916', 'ETH'), ('1550003', 'ETH')]}
按照@PaulMcG的回答，可以通过将其包装在列表推导中找到OP要求的格式。这样就可以做到：
result = {key: [pair[0] for pair in values],
for key, values in groupBy(lambda pair: pair[1], input).items()}
This answer is similar to @PaulMcG’s answer but doesn’t require sorting the input.
For those into functional programming, groupBy can be written in one line (not including imports!), and unlike itertools.groupby it doesn’t require the input to be sorted:
from functools import reduce # import needed for python3; builtin in python2
from collections import defaultdict
def groupBy(key, seq):
return reduce(lambda grp, val: grp[key(val)].append(val) or grp, seq, defaultdict(list))
(The reason for ... or grp in the lambda is that for this reduce() to work, the lambda needs to return its first argument; because list.append() always returns None the or will always return grp. I.e. it’s a hack to get around python’s restriction that a lambda can only evaluate a single expression.)
This returns a dict whose keys are found by evaluating the given function and whose values are a list of the original items in the original order. For the OP’s example, calling this as groupBy(lambda pair: pair[1], input) will return this dict:
{'KAT': [('11013331', 'KAT'), ('9843236', 'KAT')],
'NOT': [('9085267', 'NOT'), ('11788544', 'NOT')],
'ETH': [('5238761', 'ETH'), ('5349618', 'ETH'), ('962142', 'ETH'), ('7795297', 'ETH'), ('7341464', 'ETH'), ('5594916', 'ETH'), ('1550003', 'ETH')]}
And as per @PaulMcG’s answer the OP’s requested format can be found by wrapping that in a list comprehension. So this will do it:
result = {key: [pair[0] for pair in values],
for key, values in groupBy(lambda pair: pair[1], input).items()}
回答 4
以下函数将通过具有任何索引的键快速（无需排序）对任意长度的元组进行分组：
# given a sequence of tuples like [(3,'c',6),(7,'a',2),(88,'c',4),(45,'a',0)],
# returns a dict grouping tuples by idx-th element - with idx=1 we have:
# if merge is True {'c':(3,6,88,4), 'a':(7,2,45,0)}
# if merge is False {'c':((3,6),(88,4)), 'a':((7,2),(45,0))}
def group_by(seqs,idx=0,merge=True):
d = dict()
for seq in seqs:
k = seq[idx]
v = d.get(k,tuple()) + (seq[:idx]+seq[idx+1:] if merge else (seq[:idx]+seq[idx+1:],))
d.update({k:v})
return d
对于您的问题，要分组的键的索引为1，因此：
group_by(input,1)
给
{'ETH': ('5238761','5349618','962142','7795297','7341464','5594916','1550003'),
'KAT': ('11013331', '9843236'),
'NOT': ('9085267', '11788544')}
这不完全是您要求的输出，但也可能满足您的需求。
The following function will quickly (no sorting required) group tuples of any length by a key having any index:
# given a sequence of tuples like [(3,'c',6),(7,'a',2),(88,'c',4),(45,'a',0)],
# returns a dict grouping tuples by idx-th element - with idx=1 we have:
# if merge is True {'c':(3,6,88,4), 'a':(7,2,45,0)}
# if merge is False {'c':((3,6),(88,4)), 'a':((7,2),(45,0))}
def group_by(seqs,idx=0,merge=True):
d = dict()
for seq in seqs:
k = seq[idx]
v = d.get(k,tuple()) + (seq[:idx]+seq[idx+1:] if merge else (seq[:idx]+seq[idx+1:],))
d.update({k:v})
return d
In the case of your question, the index of key you want to group by is 1, therefore:
group_by(input,1)
gives
{'ETH': ('5238761','5349618','962142','7795297','7341464','5594916','1550003'),
'KAT': ('11013331', '9843236'),
'NOT': ('9085267', '11788544')}
which is not exactly the output you asked for, but might as well suit your needs.
回答 5
result = []
# Make a set of your "types":
input_set = set([tpl[1] for tpl in input])
>>> set(['ETH', 'KAT', 'NOT'])
# Iterate over the input_set
for type_ in input_set:
# a dict to gather things:
D = {}
# filter all tuples from your input with the same type as type_
tuples = filter(lambda tpl: tpl[1] == type_, input)
# write them in the D:
D["type"] = type_
D["itmes"] = [tpl[0] for tpl in tuples]
# append D to results:
result.append(D)
result
>>> [{'itmes': ['9085267', '11788544'], 'type': 'NOT'}, {'itmes': ['5238761', '5349618', '962142', '7795297', '7341464', '5594916', '1550003'], 'type': 'ETH'}, {'itmes': ['11013331', '9843236'], 'type': 'KAT'}]
result = []
# Make a set of your "types":
input_set = set([tpl[1] for tpl in input])
>>> set(['ETH', 'KAT', 'NOT'])
# Iterate over the input_set
for type_ in input_set:
# a dict to gather things:
D = {}
# filter all tuples from your input with the same type as type_
tuples = filter(lambda tpl: tpl[1] == type_, input)
# write them in the D:
D["type"] = type_
D["itmes"] = [tpl[0] for tpl in tuples]
# append D to results:
result.append(D)
result
>>> [{'itmes': ['9085267', '11788544'], 'type': 'NOT'}, {'itmes': ['5238761', '5349618', '962142', '7795297', '7341464', '5594916', '1550003'], 'type': 'ETH'}, {'itmes': ['11013331', '9843236'], 'type': 'KAT'}]
