如何使用itertools.groupby()?

问题:如何使用itertools.groupby()?

我还没有找到关于如何实际使用Python itertools.groupby()函数的可理解的解释。我想做的是这样的:

  • 列出一个列表-在这种情况下,是对象化lxml元素的子元素
  • 根据一些标准将其分为几组
  • 然后,稍后分别遍历每个组。

我已经阅读了文档示例,但是尝试将它们应用于简单的数字列表之外时遇到了麻烦。

那么,我该如何使用itertools.groupby()?我应该使用另一种技术吗?指向良好“前提”阅读的指针也将受到赞赏。

I haven’t been able to find an understandable explanation of how to actually use Python’s itertools.groupby() function. What I’m trying to do is this:

  • Take a list – in this case, the children of an objectified lxml element
  • Divide it into groups based on some criteria
  • Then later iterate over each of these groups separately.

I’ve reviewed the documentation, and the examples, but I’ve had trouble trying to apply them beyond a simple list of numbers.

So, how do I use of itertools.groupby()? Is there another technique I should be using? Pointers to good “prerequisite” reading would also be appreciated.


回答 0

重要说明:您必须先对数据进行排序


我没有得到的部分是在示例构造中

groups = []
uniquekeys = []
for k, g in groupby(data, keyfunc):
   groups.append(list(g))    # Store group iterator as a list
   uniquekeys.append(k)

k是当前的分组密钥,并且g是一个迭代器,您可以用来迭代该分组密钥定义的组。换句话说,groupby迭代器本身返回迭代器。

这是使用更清晰的变量名的示例:

from itertools import groupby

things = [("animal", "bear"), ("animal", "duck"), ("plant", "cactus"), ("vehicle", "speed boat"), ("vehicle", "school bus")]

for key, group in groupby(things, lambda x: x[0]):
    for thing in group:
        print "A %s is a %s." % (thing[1], key)
    print " "

这将为您提供输出:

熊是动物。
鸭子是动物。

仙人掌是植物。

快艇是车辆。
校车是车辆。

在此示例中,things是一个元组列表,其中每个元组中的第一项是第二项所属的组。

groupby()函数有两个参数:(1)要分组的数据和(2)将数据分组的函数。

在这里,lambda x: x[0]告诉groupby()使用每个元组中的第一项作为分组键。

在上面的for语句中,groupby返回三个(键,组迭代器)对-每个唯一键一次。您可以使用返回的迭代器来迭代该组中的每个单个项目。

这是一个使用列表推导的具有相同数据的稍微不同的示例:

for key, group in groupby(things, lambda x: x[0]):
    listOfThings = " and ".join([thing[1] for thing in group])
    print key + "s:  " + listOfThings + "."

这将为您提供输出:

动物:熊和鸭。
植物:仙人掌。
车辆:快艇和校车。

IMPORTANT NOTE: You have to sort your data first.


The part I didn’t get is that in the example construction

groups = []
uniquekeys = []
for k, g in groupby(data, keyfunc):
   groups.append(list(g))    # Store group iterator as a list
   uniquekeys.append(k)

k is the current grouping key, and g is an iterator that you can use to iterate over the group defined by that grouping key. In other words, the groupby iterator itself returns iterators.

Here’s an example of that, using clearer variable names:

from itertools import groupby

things = [("animal", "bear"), ("animal", "duck"), ("plant", "cactus"), ("vehicle", "speed boat"), ("vehicle", "school bus")]

for key, group in groupby(things, lambda x: x[0]):
    for thing in group:
        print "A %s is a %s." % (thing[1], key)
    print " "

This will give you the output:

A bear is a animal.
A duck is a animal.

A cactus is a plant.

A speed boat is a vehicle.
A school bus is a vehicle.

In this example, things is a list of tuples where the first item in each tuple is the group the second item belongs to.

The groupby() function takes two arguments: (1) the data to group and (2) the function to group it with.

Here, lambda x: x[0] tells groupby() to use the first item in each tuple as the grouping key.

In the above for statement, groupby returns three (key, group iterator) pairs – once for each unique key. You can use the returned iterator to iterate over each individual item in that group.

Here’s a slightly different example with the same data, using a list comprehension:

for key, group in groupby(things, lambda x: x[0]):
    listOfThings = " and ".join([thing[1] for thing in group])
    print key + "s:  " + listOfThings + "."

This will give you the output:

animals: bear and duck.
plants: cactus.
vehicles: speed boat and school bus.


回答 1

Python文档上的示例非常简单:

groups = []
uniquekeys = []
for k, g in groupby(data, keyfunc):
    groups.append(list(g))      # Store group iterator as a list
    uniquekeys.append(k)

因此,在您的情况下,数据是节点列表,keyfunc准则功能的逻辑将在此进行然后groupby()对数据进行分组。

在调用之前,您必须小心按照条件对数据进行排序,groupby否则它将无法正常工作。groupby方法实际上只是遍历列表,并且每当键更改时,它都会创建一个新组。

The example on the Python docs is quite straightforward:

groups = []
uniquekeys = []
for k, g in groupby(data, keyfunc):
    groups.append(list(g))      # Store group iterator as a list
    uniquekeys.append(k)

So in your case, data is a list of nodes, keyfunc is where the logic of your criteria function goes and then groupby() groups the data.

You must be careful to sort the data by the criteria before you call groupby or it won’t work. groupby method actually just iterates through a list and whenever the key changes it creates a new group.


回答 2

itertools.groupby 是用于分组项目的工具。

文档中,我们进一步了解了它的作用:

# [k for k, g in groupby('AAAABBBCCDAABBB')] --> A B C D A B

# [list(g) for k, g in groupby('AAAABBBCCD')] --> AAAA BBB CC D

groupby 对象产生密钥组对,其中组是生成器。

特征

  • A.将连续的项目组合在一起
  • B.归因于可迭代项,将某项的所有出现分组
  • C.指定如何使用按键功能 对项目进行分组*

比较

# Define a printer for comparing outputs
>>> def print_groupby(iterable, keyfunc=None):
...    for k, g in it.groupby(iterable, keyfunc):
...        print("key: '{}'--> group: {}".format(k, list(g)))

# Feature A: group consecutive occurrences
>>> print_groupby("BCAACACAADBBB")
key: 'B'--> group: ['B']
key: 'C'--> group: ['C']
key: 'A'--> group: ['A', 'A']
key: 'C'--> group: ['C']
key: 'A'--> group: ['A']
key: 'C'--> group: ['C']
key: 'A'--> group: ['A', 'A']
key: 'D'--> group: ['D']
key: 'B'--> group: ['B', 'B', 'B']

# Feature B: group all occurrences
>>> print_groupby(sorted("BCAACACAADBBB"))
key: 'A'--> group: ['A', 'A', 'A', 'A', 'A']
key: 'B'--> group: ['B', 'B', 'B', 'B']
key: 'C'--> group: ['C', 'C', 'C']
key: 'D'--> group: ['D']

# Feature C: group by a key function
>>> # keyfunc = lambda s: s.islower()                      # equivalent
>>> def keyfunc(s):
...     """Return a True if a string is lowercase, else False."""   
...     return s.islower()
>>> print_groupby(sorted("bCAaCacAADBbB"), keyfunc)
key: 'False'--> group: ['A', 'A', 'A', 'B', 'B', 'C', 'C', 'D']
key: 'True'--> group: ['a', 'a', 'b', 'b', 'c']

用途

注意:后面的几个示例来自VíctorTerrón的PyCon (谈话) (西班牙语),“黎明时的功夫与Itertools”。另请参见用C编写的groupby源代码

*传递和比较所有项目,影响结果的功能。按键功能的对象包括 sorted()max()min()


响应

# OP: Yes, you can use `groupby`, e.g. 
[do_something(list(g)) for _, g in groupby(lxml_elements, criteria_func)]

itertools.groupby is a tool for grouping items.

From the docs, we glean further what it might do:

# [k for k, g in groupby('AAAABBBCCDAABBB')] --> A B C D A B

# [list(g) for k, g in groupby('AAAABBBCCD')] --> AAAA BBB CC D

groupby objects yield key-group pairs where the group is a generator.

Features

  • A. Group consecutive items together
  • B. Group all occurrences of an item, given a sorted iterable
  • C. Specify how to group items with a key function *

Comparisons

# Define a printer for comparing outputs
>>> def print_groupby(iterable, keyfunc=None):
...    for k, g in it.groupby(iterable, keyfunc):
...        print("key: '{}'--> group: {}".format(k, list(g)))

# Feature A: group consecutive occurrences
>>> print_groupby("BCAACACAADBBB")
key: 'B'--> group: ['B']
key: 'C'--> group: ['C']
key: 'A'--> group: ['A', 'A']
key: 'C'--> group: ['C']
key: 'A'--> group: ['A']
key: 'C'--> group: ['C']
key: 'A'--> group: ['A', 'A']
key: 'D'--> group: ['D']
key: 'B'--> group: ['B', 'B', 'B']

# Feature B: group all occurrences
>>> print_groupby(sorted("BCAACACAADBBB"))
key: 'A'--> group: ['A', 'A', 'A', 'A', 'A']
key: 'B'--> group: ['B', 'B', 'B', 'B']
key: 'C'--> group: ['C', 'C', 'C']
key: 'D'--> group: ['D']

# Feature C: group by a key function
>>> # keyfunc = lambda s: s.islower()                      # equivalent
>>> def keyfunc(s):
...     """Return a True if a string is lowercase, else False."""   
...     return s.islower()
>>> print_groupby(sorted("bCAaCacAADBbB"), keyfunc)
key: 'False'--> group: ['A', 'A', 'A', 'B', 'B', 'C', 'C', 'D']
key: 'True'--> group: ['a', 'a', 'b', 'b', 'c']

Uses

Note: Several of the latter examples derive from Víctor Terrón’s PyCon (talk) (Spanish), “Kung Fu at Dawn with Itertools”. See also the groupbysource code written in C.

* A function where all items are passed through and compared, influencing the result. Other objects with key functions include sorted(), max() and min().


Response

# OP: Yes, you can use `groupby`, e.g. 
[do_something(list(g)) for _, g in groupby(lxml_elements, criteria_func)]

回答 3

groupby的一个绝妙技巧是在一行中运行长度编码:

[(c,len(list(cgen))) for c,cgen in groupby(some_string)]

将为您提供2元组的列表,其中第一个元素是char,第二个是重复数。

编辑:请注意,这是itertools.groupby与SQL GROUP BY语义分离的内容:itertools不会(通常不能)事先对迭代器进行排序,因此不会合并具有相同“键”的组。

A neato trick with groupby is to run length encoding in one line:

[(c,len(list(cgen))) for c,cgen in groupby(some_string)]

will give you a list of 2-tuples where the first element is the char and the 2nd is the number of repetitions.

Edit: Note that this is what separates itertools.groupby from the SQL GROUP BY semantics: itertools doesn’t (and in general can’t) sort the iterator in advance, so groups with the same “key” aren’t merged.


回答 4

另一个例子:

for key, igroup in itertools.groupby(xrange(12), lambda x: x // 5):
    print key, list(igroup)

结果是

0 [0, 1, 2, 3, 4]
1 [5, 6, 7, 8, 9]
2 [10, 11]

请注意,igroup是一个迭代器(文档中称为子迭代器)。

这对于分块生成器很有用:

def chunker(items, chunk_size):
    '''Group items in chunks of chunk_size'''
    for _key, group in itertools.groupby(enumerate(items), lambda x: x[0] // chunk_size):
        yield (g[1] for g in group)

with open('file.txt') as fobj:
    for chunk in chunker(fobj):
        process(chunk)

groupby的另一个示例-不对键进行排序时。在以下示例中,xx中的项目按yy中的值分组。在这种情况下,首先输出一组零,然后输出一组零,再输出一组零。

xx = range(10)
yy = [0, 0, 0, 1, 1, 1, 0, 0, 0, 0]
for group in itertools.groupby(iter(xx), lambda x: yy[x]):
    print group[0], list(group[1])

生成:

0 [0, 1, 2]
1 [3, 4, 5]
0 [6, 7, 8, 9]

Another example:

for key, igroup in itertools.groupby(xrange(12), lambda x: x // 5):
    print key, list(igroup)

results in

0 [0, 1, 2, 3, 4]
1 [5, 6, 7, 8, 9]
2 [10, 11]

Note that igroup is an iterator (a sub-iterator as the documentation calls it).

This is useful for chunking a generator:

def chunker(items, chunk_size):
    '''Group items in chunks of chunk_size'''
    for _key, group in itertools.groupby(enumerate(items), lambda x: x[0] // chunk_size):
        yield (g[1] for g in group)

with open('file.txt') as fobj:
    for chunk in chunker(fobj):
        process(chunk)

Another example of groupby – when the keys are not sorted. In the following example, items in xx are grouped by values in yy. In this case, one set of zeros is output first, followed by a set of ones, followed again by a set of zeros.

xx = range(10)
yy = [0, 0, 0, 1, 1, 1, 0, 0, 0, 0]
for group in itertools.groupby(iter(xx), lambda x: yy[x]):
    print group[0], list(group[1])

Produces:

0 [0, 1, 2]
1 [3, 4, 5]
0 [6, 7, 8, 9]

回答 5

警告:

语法列表(groupby(…))不能按您期望的方式工作。似乎破坏了内部迭代器对象,因此使用

for x in list(groupby(range(10))):
    print(list(x[1]))

将生成:

[]
[]
[]
[]
[]
[]
[]
[]
[]
[9]

代替list(groupby(…)),尝试[[k,list(g))for groupby(…)中的k,g,或者如果您经常使用该语法,

def groupbylist(*args, **kwargs):
    return [(k, list(g)) for k, g in groupby(*args, **kwargs)]

并访问了groupby功能,同时避免了那些讨厌的(对于小数据)迭代器。

WARNING:

The syntax list(groupby(…)) won’t work the way that you intend. It seems to destroy the internal iterator objects, so using

for x in list(groupby(range(10))):
    print(list(x[1]))

will produce:

[]
[]
[]
[]
[]
[]
[]
[]
[]
[9]

Instead, of list(groupby(…)), try [(k, list(g)) for k,g in groupby(…)], or if you use that syntax often,

def groupbylist(*args, **kwargs):
    return [(k, list(g)) for k, g in groupby(*args, **kwargs)]

and get access to the groupby functionality while avoiding those pesky (for small data) iterators all together.


回答 6

我想再举一个没有排序的groupby无法正常工作的例子。改编自James Sulak的例子

from itertools import groupby

things = [("vehicle", "bear"), ("animal", "duck"), ("animal", "cactus"), ("vehicle", "speed boat"), ("vehicle", "school bus")]

for key, group in groupby(things, lambda x: x[0]):
    for thing in group:
        print "A %s is a %s." % (thing[1], key)
    print " "

输出是

A bear is a vehicle.

A duck is a animal.
A cactus is a animal.

A speed boat is a vehicle.
A school bus is a vehicle.

有两组带有车辆的车辆,而一个可以预期只有一组

I would like to give another example where groupby without sort is not working. Adapted from example by James Sulak

from itertools import groupby

things = [("vehicle", "bear"), ("animal", "duck"), ("animal", "cactus"), ("vehicle", "speed boat"), ("vehicle", "school bus")]

for key, group in groupby(things, lambda x: x[0]):
    for thing in group:
        print "A %s is a %s." % (thing[1], key)
    print " "

output is

A bear is a vehicle.

A duck is a animal.
A cactus is a animal.

A speed boat is a vehicle.
A school bus is a vehicle.

there are two groups with vehicule, whereas one could expect only one group


回答 7

@CaptSolo,我尝试了您的示例,但没有成功。

from itertools import groupby 
[(c,len(list(cs))) for c,cs in groupby('Pedro Manoel')]

输出:

[('P', 1), ('e', 1), ('d', 1), ('r', 1), ('o', 1), (' ', 1), ('M', 1), ('a', 1), ('n', 1), ('o', 1), ('e', 1), ('l', 1)]

如您所见,有两个o和两个e,但是它们分成不同的组。从那时起,我意识到您需要对传递给groupby函数的列表进行排序。因此,正确的用法是:

name = list('Pedro Manoel')
name.sort()
[(c,len(list(cs))) for c,cs in groupby(name)]

输出:

[(' ', 1), ('M', 1), ('P', 1), ('a', 1), ('d', 1), ('e', 2), ('l', 1), ('n', 1), ('o', 2), ('r', 1)]

请记住,如果列表未排序,groupby函数将无法工作

@CaptSolo, I tried your example, but it didn’t work.

from itertools import groupby 
[(c,len(list(cs))) for c,cs in groupby('Pedro Manoel')]

Output:

[('P', 1), ('e', 1), ('d', 1), ('r', 1), ('o', 1), (' ', 1), ('M', 1), ('a', 1), ('n', 1), ('o', 1), ('e', 1), ('l', 1)]

As you can see, there are two o’s and two e’s, but they got into separate groups. That’s when I realized you need to sort the list passed to the groupby function. So, the correct usage would be:

name = list('Pedro Manoel')
name.sort()
[(c,len(list(cs))) for c,cs in groupby(name)]

Output:

[(' ', 1), ('M', 1), ('P', 1), ('a', 1), ('d', 1), ('e', 2), ('l', 1), ('n', 1), ('o', 2), ('r', 1)]

Just remembering, if the list is not sorted, the groupby function will not work!


回答 8

排序和分组

from itertools import groupby

val = [{'name': 'satyajit', 'address': 'btm', 'pin': 560076}, 
       {'name': 'Mukul', 'address': 'Silk board', 'pin': 560078},
       {'name': 'Preetam', 'address': 'btm', 'pin': 560076}]


for pin, list_data in groupby(sorted(val, key=lambda k: k['pin']),lambda x: x['pin']):
...     print pin
...     for rec in list_data:
...             print rec
... 
o/p:

560076
{'name': 'satyajit', 'pin': 560076, 'address': 'btm'}
{'name': 'Preetam', 'pin': 560076, 'address': 'btm'}
560078
{'name': 'Mukul', 'pin': 560078, 'address': 'Silk board'}

Sorting and groupby

from itertools import groupby

val = [{'name': 'satyajit', 'address': 'btm', 'pin': 560076}, 
       {'name': 'Mukul', 'address': 'Silk board', 'pin': 560078},
       {'name': 'Preetam', 'address': 'btm', 'pin': 560076}]


for pin, list_data in groupby(sorted(val, key=lambda k: k['pin']),lambda x: x['pin']):
...     print pin
...     for rec in list_data:
...             print rec
... 
o/p:

560076
{'name': 'satyajit', 'pin': 560076, 'address': 'btm'}
{'name': 'Preetam', 'pin': 560076, 'address': 'btm'}
560078
{'name': 'Mukul', 'pin': 560078, 'address': 'Silk board'}

回答 9

如何使用Python的itertools.groupby()?

您可以使用groupby对事物进行分组以进行迭代。您为groupby提供了一个可迭代的对象,以及一个可选的函数/可调用对象,通过它可以检查从可迭代对象中出来的项,然后返回一个迭代器,该迭代器给出了可调用键和实际项的结果的二元组。另一个可迭代的。从帮助中:

groupby(iterable[, keyfunc]) -> create an iterator which returns
(key, sub-iterator) grouped by each value of key(value).

这是一个使用协程对计数进行分组的groupby示例,它使用可调用的键(在这种情况下为coroutine.send)来吐出计数,无论迭代多少次,以及元素的分组子迭代器:

import itertools


def grouper(iterable, n):
    def coroutine(n):
        yield # queue up coroutine
        for i in itertools.count():
            for j in range(n):
                yield i
    groups = coroutine(n)
    next(groups) # queue up coroutine

    for c, objs in itertools.groupby(iterable, groups.send):
        yield c, list(objs)
    # or instead of materializing a list of objs, just:
    # return itertools.groupby(iterable, groups.send)

list(grouper(range(10), 3))

版画

[(0, [0, 1, 2]), (1, [3, 4, 5]), (2, [6, 7, 8]), (3, [9])]

How do I use Python’s itertools.groupby()?

You can use groupby to group things to iterate over. You give groupby an iterable, and a optional key function/callable by which to check the items as they come out of the iterable, and it returns an iterator that gives a two-tuple of the result of the key callable and the actual items in another iterable. From the help:

groupby(iterable[, keyfunc]) -> create an iterator which returns
(key, sub-iterator) grouped by each value of key(value).

Here’s an example of groupby using a coroutine to group by a count, it uses a key callable (in this case, coroutine.send) to just spit out the count for however many iterations and a grouped sub-iterator of elements:

import itertools


def grouper(iterable, n):
    def coroutine(n):
        yield # queue up coroutine
        for i in itertools.count():
            for j in range(n):
                yield i
    groups = coroutine(n)
    next(groups) # queue up coroutine

    for c, objs in itertools.groupby(iterable, groups.send):
        yield c, list(objs)
    # or instead of materializing a list of objs, just:
    # return itertools.groupby(iterable, groups.send)

list(grouper(range(10), 3))

prints

[(0, [0, 1, 2]), (1, [3, 4, 5]), (2, [6, 7, 8]), (3, [9])]

回答 10

我遇到的一个有用的示例可能会有所帮助:

from itertools import groupby

#user input

myinput = input()

#creating empty list to store output

myoutput = []

for k,g in groupby(myinput):

    myoutput.append((len(list(g)),int(k)))

print(*myoutput)

输入样本:14445221

样本输出:(1,1)(3,4)(1,5)(2,2)(1,1)

One useful example that I came across may be helpful:

from itertools import groupby

#user input

myinput = input()

#creating empty list to store output

myoutput = []

for k,g in groupby(myinput):

    myoutput.append((len(list(g)),int(k)))

print(*myoutput)

Sample input: 14445221

Sample output: (1,1) (3,4) (1,5) (2,2) (1,1)


回答 11

这个基本的实现帮助我理解了此功能。希望它也能帮助其他人:

arr = [(1, "A"), (1, "B"), (1, "C"), (2, "D"), (2, "E"), (3, "F")]

for k,g in groupby(arr, lambda x: x[0]):
    print("--", k, "--")
    for tup in g:
        print(tup[1])  # tup[0] == k
-- 1 --
A
B
C
-- 2 --
D
E
-- 3 --
F

This basic implementation helped me understand this function. Hope it helps others as well:

arr = [(1, "A"), (1, "B"), (1, "C"), (2, "D"), (2, "E"), (3, "F")]

for k,g in groupby(arr, lambda x: x[0]):
    print("--", k, "--")
    for tup in g:
        print(tup[1])  # tup[0] == k
-- 1 --
A
B
C
-- 2 --
D
E
-- 3 --
F

回答 12

您可以编写自己的groupby函数:

           def groupby(data):
                kv = {}
                for k,v in data:
                    if k not in kv:
                         kv[k]=[v]
                    else:
                        kv[k].append(v)
           return kv

     Run on ipython:
       In [10]: data = [('a', 1), ('b',2),('a',2)]

        In [11]: groupby(data)
        Out[11]: {'a': [1, 2], 'b': [2]}

You can write own groupby function:

           def groupby(data):
                kv = {}
                for k,v in data:
                    if k not in kv:
                         kv[k]=[v]
                    else:
                        kv[k].append(v)
           return kv

     Run on ipython:
       In [10]: data = [('a', 1), ('b',2),('a',2)]

        In [11]: groupby(data)
        Out[11]: {'a': [1, 2], 'b': [2]}