问题:是否在Python 3.6+中订购了字典?

与以前的版本不同,字典在Python 3.6中排序(至少在CPython实现下)。这似乎是一个重大更改,但只是文档中的一小段。它被描述为CPython实现细节而不是语言功能,但这也意味着将来可能会成为标准。

在保留元素顺序的同时,新的字典实现如何比旧的实现更好?

以下是文档中的文字:

dict()现在使用PyPy率先提出的“紧凑”表示形式。与Python 3.5相比,新dict()的内存使用量减少了20%至25%。PEP 468(在函数中保留** kwarg的顺序。)由此实现。此新实现的顺序保留方面被认为是实现细节,因此不应依赖(将来可能会更改,但是希望在更改语言规范之前,先在几个发行版中使用该新dict实现该语言,为所有当前和将来的Python实现强制要求保留顺序的语义;这还有助于保留与仍旧有效的随机迭代顺序的旧版本语言(例如Python 3.5)的向后兼容性。(由INADA Naoki在发行27350最初由Raymond Hettinger提出的想法。)

2017年12月更新:Python 3.7 保证dict保留插入顺序

Dictionaries are ordered in Python 3.6 (under the CPython implementation at least) unlike in previous incarnations. This seems like a substantial change, but it’s only a short paragraph in the documentation. It is described as a CPython implementation detail rather than a language feature, but also implies this may become standard in the future.

How does the new dictionary implementation perform better than the older one while preserving element order?

Here is the text from the documentation:

dict() now uses a “compact” representation pioneered by PyPy. The memory usage of the new dict() is between 20% and 25% smaller compared to Python 3.5. PEP 468 (Preserving the order of **kwargs in a function.) is implemented by this. The order-preserving aspect of this new implementation is considered an implementation detail and should not be relied upon (this may change in the future, but it is desired to have this new dict implementation in the language for a few releases before changing the language spec to mandate order-preserving semantics for all current and future Python implementations; this also helps preserve backwards-compatibility with older versions of the language where random iteration order is still in effect, e.g. Python 3.5). (Contributed by INADA Naoki in issue 27350. Idea originally suggested by Raymond Hettinger.)

Update December 2017: dicts retaining insertion order is guaranteed for Python 3.7


回答 0

是否在Python 3.6+中订购了字典?

它们是插入顺序[1]。从Python 3.6开始,对于Python的CPython实现,字典会记住插入项目的顺序这在Python 3.6中被视为实现细节;你需要使用OrderedDict,如果你想多数民众赞成插入排序保证不同的Python的其它实现(与其他有序行为[1] )。

从Python 3.7开始,它不再是实现细节,而是成为一种语言功能。从GvR的py​​thon-dev消息中

做到这一点。裁定“裁定保留插入顺序”。谢谢!

这只是意味着您可以依靠它。如果其他Python实现希望成为Python 3.7的一致实现,则还必须提供插入顺序字典。


在保留元素顺序的同时,Python 3.6字典实现如何比旧的实现更好的性能[2]

本质上,通过保留两个数组

  • 第一个数组,按插入顺序dk_entries保存字典的条目(类型PyDictKeyEntry)。保留顺序是通过仅附加数组来实现的,在该数组中始终在末尾插入新项(插入顺序)。

  • 第二个保留dk_entries数组的索引(即,指示中相应条目位置的值dk_entries)。该数组充当哈希表。对键进行哈希处理时,它会导致存储在其中的索引之一,dk_indices并且通过indexing获取相应的条目dk_entries。由于只有索引被保留,此数组的类型取决于字典的整体大小(范围从类型int8_t1字节)到int32_t/ int64_t4/ 8字节)上32/ 64位构建)

在以前的实现中,必须分配类型PyDictKeyEntry和大小的稀疏数组dk_size。不幸的是,由于性能原因,该阵列不允许2/3 * dk_size满载,这也导致了很多空白。(并且空白区域具有大小!)。PyDictKeyEntry

现在不是这种情况,因为仅存储了必需的条目(已插入的条目),并且保留了一个稀疏类型的数组intX_tX取决于dict的大小)2/3 * dk_size。空格从类型更改PyDictKeyEntryintX_t

因此,显然,创建一个类型PyDictKeyEntry稀疏的数组比存储ints 的稀疏数组需要更多的内存。

如果有兴趣,可以在Python-Dev上查看有关此功能的完整对话,这是一本好书。


在Raymond Hettinger提出的原始建议中,可以看到使用的数据结构的可视化效果,该可视化体现了该思想的要旨。

例如,字典:

d = {'timmy': 'red', 'barry': 'green', 'guido': 'blue'}

当前存储为[keyhash,key,value]:

entries = [['--', '--', '--'],
           [-8522787127447073495, 'barry', 'green'],
           ['--', '--', '--'],
           ['--', '--', '--'],
           ['--', '--', '--'],
           [-9092791511155847987, 'timmy', 'red'],
           ['--', '--', '--'],
           [-6480567542315338377, 'guido', 'blue']]

相反,数据应按以下方式组织:

indices =  [None, 1, None, None, None, 0, None, 2]
entries =  [[-9092791511155847987, 'timmy', 'red'],
            [-8522787127447073495, 'barry', 'green'],
            [-6480567542315338377, 'guido', 'blue']]

正如您现在可以从视觉上看到的那样,在原始建议中,很多空间实际上是空的,以减少冲突并加快查找速度。使用新方法,可以通过将稀疏移动到真正需要的索引中来减少所需的内存。


[1]:我说“插入有序”而不是“有序”,因为在存在OrderedDict的情况下,“有序”暗示了dict对象不提供的其他行为。OrderedDicts是可逆的,提供顺序敏感的方法,并且主要是,提供一个订单sensive相等测试(==!=)。dict目前不提供任何这些行为/方法。


[2]:新的字典实现通过更紧凑的设计而在内存方面表现更好;这是这里的主要好处。在速度方面,差异并不那么明显,在某些地方,新的dict可能会引入轻微的回归(例如,关键查找),而在其他地方(会想到迭代和调整大小),应该会提高性能。

总体而言,由于引入的紧凑性,字典的性能(尤其是在现实生活中)得以提高。

Are dictionaries ordered in Python 3.6+?

They are insertion ordered[1]. As of Python 3.6, for the CPython implementation of Python, dictionaries remember the order of items inserted. This is considered an implementation detail in Python 3.6; you need to use OrderedDict if you want insertion ordering that’s guaranteed across other implementations of Python (and other ordered behavior[1]).

As of Python 3.7, this is no longer an implementation detail and instead becomes a language feature. From a python-dev message by GvR:

Make it so. “Dict keeps insertion order” is the ruling. Thanks!

This simply means that you can depend on it. Other implementations of Python must also offer an insertion ordered dictionary if they wish to be a conforming implementation of Python 3.7.


How does the Python 3.6 dictionary implementation perform better[2] than the older one while preserving element order?

Essentially, by keeping two arrays.

  • The first array, dk_entries, holds the entries (of type PyDictKeyEntry) for the dictionary in the order that they were inserted. Preserving order is achieved by this being an append only array where new items are always inserted at the end (insertion order).

  • The second, , holds the indices for the dk_entries array (that is, values that indicate the position of the corresponding entry in dk_entries). This array acts as the hash table. When a key is hashed it leads to one of the indices stored in dk_indices and the corresponding entry is fetched by indexing dk_entries. Since only indices are kept, the type of this array depends on the overall size of the dictionary (ranging from type int8_t(1 byte) to int32_t/int64_t (4/8 bytes) on 32/64 bit builds)

In the previous implementation, a sparse array of type PyDictKeyEntry and size dk_size had to be allocated; unfortunately, it also resulted in a lot of empty space since that array was not allowed to be more than 2/3 * dk_size full for performance reasons. (and the empty space still had PyDictKeyEntry size!).

This is not the case now since only the required entries are stored (those that have been inserted) and a sparse array of type intX_t (X depending on dict size) 2/3 * dk_sizes full is kept. The empty space changed from type PyDictKeyEntry to intX_t.

So, obviously, creating a sparse array of type PyDictKeyEntry is much more memory demanding than a sparse array for storing ints.

You can see the full conversation on Python-Dev regarding this feature if interested, it is a good read.


In the original proposal made by Raymond Hettinger, a visualization of the data structures used can be seen which captures the gist of the idea.

For example, the dictionary:

d = {'timmy': 'red', 'barry': 'green', 'guido': 'blue'}

is currently stored as [keyhash, key, value]:

entries = [['--', '--', '--'],
           [-8522787127447073495, 'barry', 'green'],
           ['--', '--', '--'],
           ['--', '--', '--'],
           ['--', '--', '--'],
           [-9092791511155847987, 'timmy', 'red'],
           ['--', '--', '--'],
           [-6480567542315338377, 'guido', 'blue']]

Instead, the data should be organized as follows:

indices =  [None, 1, None, None, None, 0, None, 2]
entries =  [[-9092791511155847987, 'timmy', 'red'],
            [-8522787127447073495, 'barry', 'green'],
            [-6480567542315338377, 'guido', 'blue']]

As you can visually now see, in the original proposal, a lot of space is essentially empty to reduce collisions and make look-ups faster. With the new approach, you reduce the memory required by moving the sparseness where it’s really required, in the indices.


[1]: I say “insertion ordered” and not “ordered” since, with the existence of OrderedDict, “ordered” suggests further behavior that the dict object doesn’t provide. OrderedDicts are reversible, provide order sensitive methods and, mainly, provide an order-sensive equality tests (==, !=). dicts currently don’t offer any of those behaviors/methods.


[2]: The new dictionary implementations performs better memory wise by being designed more compactly; that’s the main benefit here. Speed wise, the difference isn’t so drastic, there’s places where the new dict might introduce slight regressions (key-lookups, for example) while in others (iteration and resizing come to mind) a performance boost should be present.

Overall, the performance of the dictionary, especially in real-life situations, improves due to the compactness introduced.


回答 1

以下是回答最初的第一个问题:

我应该在Python 3.6中使用dict还是OrderedDict在Python 3.6中使用?

我认为文档中的这句话实际上足以回答您的问题

此新实现的顺序保留方面被视为实现细节,不应依赖于此

dict并不明确表示它是有序集合,因此,如果您要保持一致并且不依赖于新实现的副作用,则应坚持使用OrderedDict

使您的代码成为未来的证明:)

有关于辩论在这里

编辑:Python 3.7将保留此功能, 请参阅

Below is answering the original first question:

Should I use dict or OrderedDict in Python 3.6?

I think this sentence from the documentation is actually enough to answer your question

The order-preserving aspect of this new implementation is considered an implementation detail and should not be relied upon

dict is not explicitly meant to be an ordered collection, so if you want to stay consistent and not rely on a side effect of the new implementation you should stick with OrderedDict.

Make your code future proof :)

There’s a debate about that here.

EDIT: Python 3.7 will keep this as a feature see


回答 2

更新:Guido van Rossum 在邮件列表宣布,从 Python 3.7开始dict,所有Python实现中必须保留插入顺序。

Update: Guido van Rossum announced on the mailing list that as of Python 3.7 dicts in all Python implementations must preserve insertion order.


回答 3

我想添加到上面的讨论中,但没有评论的声誉。

Python 3.8尚未发布,但它甚至将包含reversed()字典上的函数(消除了的另一个区别OrderedDict。)。

现在可以使用reversed()以反向插入顺序迭代Dict和dictviews。(由RémiLapeyre在bpo-33462中贡献。) 查看python 3.8的新增功能

我没有提到相等运算符或的其他功能,OrderedDict因此它们仍然不完全相同。

I wanted to add to the discussion above but don’t have the reputation to comment.

Python 3.8 is not quite released yet, but it will even include the reversed() function on dictionaries (removing another difference from OrderedDict.

Dict and dictviews are now iterable in reversed insertion order using reversed(). (Contributed by Rémi Lapeyre in bpo-33462.) See what’s new in python 3.8

I don’t see any mention of the equality operator or other features of OrderedDict so they are still not entirely the same.


声明:本站所有文章,如无特殊说明或标注,均为本站原创发布。任何个人或组织,在未征得本站同意时,禁止复制、盗用、采集、发布本站内容到任何网站、书籍等各类媒体平台。如若本站内容侵犯了原著者的合法权益,可联系我们进行处理。