标签归档:output-buffering

Python的file.flush()到底在做什么?

问题:Python的file.flush()到底在做什么?

我在Python 文档的File Objects中找到了这个:

flush()不一定会将文件的数据写入磁盘。使用flush()和os.fsync()来确保此行为。

所以我的问题是:Python到底在flush做什么?我以为这会强制将数据写入磁盘,但现在我发现并没有。为什么?

I found this in the Python documentation for File Objects:

flush() does not necessarily write the file’s data to disk. Use flush() followed by os.fsync() to ensure this behavior.

So my question is: what exactly is Python’s flush doing? I thought that it forces to write data to the disk, but now I see that it doesn’t. Why?


回答 0

通常涉及两个级别的缓冲:

  1. 内部缓冲器
  2. 操作系统缓冲区

内部缓冲区是由您针对其进行编程的运行时/库/语言创建的缓冲区,其目的是通过避免每次写入都调用系统来加快处理速度。取而代之的是,当您写入文件对象时,您将写入其缓冲区,并且只要缓冲区被填满,就会使用系统调用将数据写入实际文件。

但是,由于操作系统缓冲区的原因,这可能并不意味着数据已写入disk。这可能仅意味着将数据从运行时维护的缓冲区复制到操作系统维护的缓冲区。

如果您写了一些东西,并且它最终在缓冲区中(仅),并且切断了计算机的电源,则当计算机关闭时,该数据将不在磁盘上。

因此,为了帮助您在各自的对象上使用flushfsync方法。

第一个flush会简单地将程序缓冲区中残留的所有数据写到实际文件中。通常,这意味着数据将从程序缓冲区复制到操作系统缓冲区。

具体来说,这意味着如果另一个进程打开了要读取的相同文件,它将能够访问刚刷新到该文件的数据。但是,这不一定意味着它已“永久”存储在磁盘上。

为此,您需要调用os.fsync确保所有操作系统缓冲区与它们所使用的存储设备同步的方法,换句话说,该方法会将数据从操作系统缓冲区复制到磁盘。

通常,您无需为这两种方法烦恼,但是,如果您对磁盘上实际存储的内容抱有偏执是好事,则应按照说明进行两次调用。


2018年补遗。

请注意,具有缓存机制的磁盘现在比2013年更加普遍,因此现在涉及的缓存和缓冲区级别更高。我认为这些缓冲区也将由sync / flush调用处理,但我真的不知道。

There’s typically two levels of buffering involved:

  1. Internal buffers
  2. Operating system buffers

The internal buffers are buffers created by the runtime/library/language that you’re programming against and is meant to speed things up by avoiding system calls for every write. Instead, when you write to a file object, you write into its buffer, and whenever the buffer fills up, the data is written to the actual file using system calls.

However, due to the operating system buffers, this might not mean that the data is written to disk. It may just mean that the data is copied from the buffers maintained by your runtime into the buffers maintained by the operating system.

If you write something, and it ends up in the buffer (only), and the power is cut to your machine, that data is not on disk when the machine turns off.

So, in order to help with that you have the flush and fsync methods, on their respective objects.

The first, flush, will simply write out any data that lingers in a program buffer to the actual file. Typically this means that the data will be copied from the program buffer to the operating system buffer.

Specifically what this means is that if another process has that same file open for reading, it will be able to access the data you just flushed to the file. However, it does not necessarily mean it has been “permanently” stored on disk.

To do that, you need to call the os.fsync method which ensures all operating system buffers are synchronized with the storage devices they’re for, in other words, that method will copy data from the operating system buffers to the disk.

Typically you don’t need to bother with either method, but if you’re in a scenario where paranoia about what actually ends up on disk is a good thing, you should make both calls as instructed.


Addendum in 2018.

Note that disks with cache mechanisms is now much more common than back in 2013, so now there are even more levels of caching and buffers involved. I assume these buffers will be handled by the sync/flush calls as well, but I don’t really know.


回答 1

因为操作系统可能不会这样做。刷新操作将文件数据强制进入RAM中的文件缓存,然后从那里开始,操作系统的工作就是将其实际发送到磁盘。

Because the operating system may not do so. The flush operation forces the file data into the file cache in RAM, and from there it’s the OS’s job to actually send it to the disk.


回答 2

它刷新内部缓冲区,这应该导致操作系统将缓冲区写出到文件中。[1] 除非您另行配置,否则Python使用操作系统的默认缓冲。

但是有时OS仍然选择不合作。尤其是在Windows / NTFS中具有诸如写入延迟之类的奇妙功能。基本上清除了内部缓冲区,但OS缓冲区仍保持不变。因此,os.fsync()在这种情况下,您必须告诉操作系统将其写入磁盘。

[1] http://docs.python.org/library/stdtypes.html

It flushes the internal buffer, which is supposed to cause the OS to write out the buffer to the file.[1] Python uses the OS’s default buffering unless you configure it do otherwise.

But sometimes the OS still chooses not to cooperate. Especially with wonderful things like write-delays in Windows/NTFS. Basically the internal buffer is flushed, but the OS buffer is still holding on to it. So you have to tell the OS to write it to disk with os.fsync() in those cases.

[1] http://docs.python.org/library/stdtypes.html


回答 3

基本上,flush()清除RAM缓冲区,其真正功能是让您随后继续写入它-但不应将其视为最佳/最安全的文件写入功能。这将冲刷您的RAM,以获取更多数据,仅此而已。如果要确保安全地将数据写入文件,请改用close()。

Basically, flush() cleans out your RAM buffer, its real power is that it lets you continue to write to it afterwards – but it shouldn’t be thought of as the best/safest write to file feature. It’s flushing your RAM for more data to come, that is all. If you want to ensure data gets written to file safely then use close() instead.