Python中泡菜的常见用例

问题:Python中泡菜的常见用例

我看过泡菜文档,但是我不知道泡菜在哪里有用。

泡菜有哪些常见用例?

I’ve looked at the pickle documentation, but I don’t understand where pickle is useful.

What are some common use-cases for pickle?


回答 0

我遇到的一些用途:

1)将程序的状态数据保存到磁盘,以便它可以在重新启动时从中断处继续执行(持久性)

2)在多核或分布式系统中通过TCP连接发送python数据(编组)

3)将python对象存储在数据库中

4)将任意python对象转换为字符串,以便可以将其用作字典键(例如,用于缓存和备忘录)。

最后一个存在一些问题-两个相同的对象可以被腌制并导致不同的字符串-甚至相同的对象两次被腌制也可以具有不同的表示形式。这是因为泡菜可以包括参考计数信息。

为了强调@lunaryorn的评论-切勿从不可靠的来源获取字符串,因为精心制作的pickle可以在系统上执行任意代码。例如,请参阅https://blog.nelhage.com/2011/03/exploiting-pickle/

Some uses that I have come across:

1) saving a program’s state data to disk so that it can carry on where it left off when restarted (persistence)

2) sending python data over a TCP connection in a multi-core or distributed system (marshalling)

3) storing python objects in a database

4) converting an arbitrary python object to a string so that it can be used as a dictionary key (e.g. for caching & memoization).

There are some issues with the last one – two identical objects can be pickled and result in different strings – or even the same object pickled twice can have different representations. This is because the pickle can include reference count information.

To emphasise @lunaryorn’s comment – you should never unpickle a string from an untrusted source, since a carefully crafted pickle could execute arbitrary code on your system. For example see https://blog.nelhage.com/2011/03/exploiting-pickle/


回答 1

最小往返次数示例

>>> import pickle
>>> a = Anon()
>>> a.foo = 'bar'
>>> pickled = pickle.dumps(a)
>>> unpickled = pickle.loads(pickled)
>>> unpickled.foo
'bar'

编辑:但作为酸洗的现实世界的例子的问题,也许最先进的使用酸洗的(你必须相当深挖掘到源)ZODB: http://svn.zope.org/

否则,PyPI会提到几个:http ://pypi.python.org/pypi?:action=search&term=pickle&submit=search

我个人已经看到了几个通过网络发送的腌制对象的示例,它们是一种易于使用的网络传输协议。

Minimal roundtrip example..

>>> import pickle
>>> a = Anon()
>>> a.foo = 'bar'
>>> pickled = pickle.dumps(a)
>>> unpickled = pickle.loads(pickled)
>>> unpickled.foo
'bar'

Edit: but as for the question of real-world examples of pickling, perhaps the most advanced use of pickling (you’d have to dig quite deep into the source) is ZODB: http://svn.zope.org/

Otherwise, PyPI mentions several: http://pypi.python.org/pypi?:action=search&term=pickle&submit=search

I have personally seen several examples of pickled objects being sent over the network as an easy to use network transfer protocol.


回答 2

酸洗对于分布式和并行计算绝对必要。

假设您要使用并行映射简化multiprocessing(或使用pyina跨群集节点),那么您需要确保要在并行资源上映射的函数可以腌制。如果没有腌制,则无法将其发送到其他进程,计算机等上的其他资源。另请参见此处的示例。

为此,我使用dill,它可以在python中序列化几乎所有内容。Dill还有一些很好的工具,可以帮助您了解在代码失败时导致酸洗失败的原因。

而且,是的,人们使用挑选来保存计算状态,您的ipython会话等。

Pickling is absolutely necessary for distributed and parallel computing.

Say you wanted to do a parallel map-reduce with multiprocessing (or across cluster nodes with pyina), then you need to make sure the function you want to have mapped across the parallel resources will pickle. If it doesn’t pickle, you can’t send it to the other resources on another process, computer, etc. Also see here for a good example.

To do this, I use dill, which can serialize almost anything in python. Dill also has some good tools for helping you understand what is causing your pickling to fail when your code fails.

And, yes, people use picking to save the state of a calculation, or your ipython session, or whatever.


回答 3

我已经在我的一个项目中使用了它。如果该应用在工作期间终止(它完成了冗长的任务并处理了许多数据),那么我需要保存整个数据结构,并在再次运行该应用后重新加载它。我之所以使用cPickle,是因为速度至关重要,并且数据量确实很大。

I have used it in one of my projects. If the app was terminated during it’s working (it did a lengthy task and processed lots of data), I needed to save the whole data structure and reload it after the app was run again. I used cPickle for this, as speed was a crucial thing and the size of data was really big.


回答 4

对于您的数据结构和类,Pickle类似于“另存为..”和“打开..”。假设我要保存数据结构,以便在程序运行之间保持持久性。

保存:

with open("save.p", "wb") as f:    
    pickle.dump(myStuff, f)        

正在加载:

try:
    with open("save.p", "rb") as f:
        myStuff = pickle.load(f)
except:
    myStuff = defaultdict(dict)

现在,我不必从头开始重新构建myStuff,而我可以从上次停止的地方继续学习。

Pickle is like “Save As..” and “Open..” for your data structures and classes. Let’s say I want to save my data structures so that it is persistent between program runs.

Saving:

with open("save.p", "wb") as f:    
    pickle.dump(myStuff, f)        

Loading:

try:
    with open("save.p", "rb") as f:
        myStuff = pickle.load(f)
except:
    myStuff = defaultdict(dict)

Now I don’t have to build myStuff from scratch all over again, and I can just pick(le) up from where I left off.


回答 5

对于初学者(就像我一样),很难理解为什么在阅读官方文档时首先使用泡菜。可能是因为文档暗示您已经知道序列化的全部目的。仅在阅读了序列化的一般说明之后,我才了解该模块的原因及其常见用例。不考虑特定编程语言的序列化的广泛解释也可能会有所帮助:https : //stackoverflow.com/a/14482962/4383472什么是序列化?https://stackoverflow.com/a/3984483/4383472

For the beginner (as is the case with me) it’s really hard to understand why use pickle in the first place when reading the official documentation. It’s maybe because the docs imply that you already know the whole purpose of serialization. Only after reading the general description of serialization have I understood the reason for this module and its common use cases. Also broad explanations of serialization disregarding a particular programming language may help: https://stackoverflow.com/a/14482962/4383472, What is serialization?, https://stackoverflow.com/a/3984483/4383472


回答 6

要添加一个真实的示例:用于Python 的Sphinx文档工具使用pickle来缓存已解析的文档和文档之间的交叉引用,以加快文档的后续构建。

To add a real-world example: The Sphinx documentation tool for Python uses pickle to cache parsed documents and cross-references between documents, to speed up subsequent builds of the documentation.


回答 7

我可以告诉你我使用它的用途,并且已经看到它的用途:

  • 游戏资料保存
  • 游戏数据可以像生命和健康一样保存
  • 以前输入程序的说号的记录

那些是我至少用过的

I can tell you the uses I use it for and have seen it used for:

  • Game profile saves
  • Game data saves like lives and health
  • Previous records of say numbers inputed to a program

Those are the ones I use it for at least


回答 8

当时,我在网站的一个网站上进行网页爬取时使用了腌制,因此我想存储超过8000k的URL,并希望尽快处理它们,所以我使用腌制是因为它的输出质量非常高。

您可以轻松地到达url,甚至在作业目录关键字停止的位置也可以非常快速地获取url详细信息以恢复该过程。

I use pickling during web scrapping one of website at that time I want to store more than 8000k urls and want to process them as fast as possible so I use pickling because its output quality is very high.

you can easily reach to url and where you stop even job directory key word also fetch url details very fast for resuming the process.