标签归档:file-locking

在Python中锁定文件

问题:在Python中锁定文件

我需要锁定一个文件才能用Python编写。将同时从多个Python进程访问它。我在网上找到了一些解决方案,但是大多数解决方案出于我的目的而失败,因为它们通常仅基于Unix或Windows。

I need to lock a file for writing in Python. It will be accessed from multiple Python processes at once. I have found some solutions online, but most fail for my purposes as they are often only Unix based or Windows based.


回答 0

好吧,所以我最后讲了我在这里编写的代码,在我的网站上的链接已失效,在archive.org查看也可以在GitHub上查看)。我可以按以下方式使用它:

from filelock import FileLock

with FileLock("myfile.txt.lock"):
    print("Lock acquired.")
    with open("myfile.txt"):
        # work with the file as it is now locked

Alright, so I ended up going with the code I wrote here, on my website link is dead, view on archive.org (also available on GitHub). I can use it in the following fashion:

from filelock import FileLock

with FileLock("myfile.txt.lock"):
    print("Lock acquired.")
    with open("myfile.txt"):
        # work with the file as it is now locked

回答 1

这里有一个跨平台的文件锁定模块:Portalocker

尽管正如Kevin所说,一次要避免从多个进程写入文件是要避免的。

如果您可以将问题塞入数据库,则可以使用SQLite。它支持并发访问并处理自己的锁定。

There is a cross-platform file locking module here: Portalocker

Although as Kevin says, writing to a file from multiple processes at once is something you want to avoid if at all possible.

If you can shoehorn your problem into a database, you could use SQLite. It supports concurrent access and handles its own locking.


回答 2

其他解决方案引用了大量外部代码库。如果您愿意自己做,这里是一些跨平台解决方案的代码,该解决方案使用Linux / DOS系统上的相应文件锁定工具。

try:
    # Posix based file locking (Linux, Ubuntu, MacOS, etc.)
    import fcntl, os
    def lock_file(f):
        fcntl.lockf(f, fcntl.LOCK_EX)
    def unlock_file(f):
        fcntl.lockf(f, fcntl.LOCK_UN)
except ModuleNotFoundError:
    # Windows file locking
    import msvcrt, os
    def file_size(f):
        return os.path.getsize( os.path.realpath(f.name) )
    def lock_file(f):
        msvcrt.locking(f.fileno(), msvcrt.LK_RLCK, file_size(f))
    def unlock_file(f):
        msvcrt.locking(f.fileno(), msvcrt.LK_UNLCK, file_size(f))


# Class for ensuring that all file operations are atomic, treat
# initialization like a standard call to 'open' that happens to be atomic.
# This file opener *must* be used in a "with" block.
class AtomicOpen:
    # Open the file with arguments provided by user. Then acquire
    # a lock on that file object (WARNING: Advisory locking).
    def __init__(self, path, *args, **kwargs):
        # Open the file and acquire a lock on the file before operating
        self.file = open(path,*args, **kwargs)
        # Lock the opened file
        lock_file(self.file)

    # Return the opened file object (knowing a lock has been obtained).
    def __enter__(self, *args, **kwargs): return self.file

    # Unlock the file and close the file object.
    def __exit__(self, exc_type=None, exc_value=None, traceback=None):        
        # Flush to make sure all buffered contents are written to file.
        self.file.flush()
        os.fsync(self.file.fileno())
        # Release the lock on the file.
        unlock_file(self.file)
        self.file.close()
        # Handle exceptions that may have come up during execution, by
        # default any exceptions are raised to the user.
        if (exc_type != None): return False
        else:                  return True        

现在,AtomicOpen可以在with通常使用open语句的块中使用。

警告:如果在Windows上运行并且Python在调用exit之前崩溃,我不确定锁定行为将是什么。

警告:此处提供的锁定仅供参考,并非绝对的。所有可能竞争的进程都必须使用“ AtomicOpen”类。

The other solutions cite a lot of external code bases. If you would prefer to do it yourself, here is some code for a cross-platform solution that uses the respective file locking tools on Linux / DOS systems.

try:
    # Posix based file locking (Linux, Ubuntu, MacOS, etc.)
    #   Only allows locking on writable files, might cause
    #   strange results for reading.
    import fcntl, os
    def lock_file(f):
        if f.writable(): fcntl.lockf(f, fcntl.LOCK_EX)
    def unlock_file(f):
        if f.writable(): fcntl.lockf(f, fcntl.LOCK_UN)
except ModuleNotFoundError:
    # Windows file locking
    import msvcrt, os
    def file_size(f):
        return os.path.getsize( os.path.realpath(f.name) )
    def lock_file(f):
        msvcrt.locking(f.fileno(), msvcrt.LK_RLCK, file_size(f))
    def unlock_file(f):
        msvcrt.locking(f.fileno(), msvcrt.LK_UNLCK, file_size(f))


# Class for ensuring that all file operations are atomic, treat
# initialization like a standard call to 'open' that happens to be atomic.
# This file opener *must* be used in a "with" block.
class AtomicOpen:
    # Open the file with arguments provided by user. Then acquire
    # a lock on that file object (WARNING: Advisory locking).
    def __init__(self, path, *args, **kwargs):
        # Open the file and acquire a lock on the file before operating
        self.file = open(path,*args, **kwargs)
        # Lock the opened file
        lock_file(self.file)

    # Return the opened file object (knowing a lock has been obtained).
    def __enter__(self, *args, **kwargs): return self.file

    # Unlock the file and close the file object.
    def __exit__(self, exc_type=None, exc_value=None, traceback=None):        
        # Flush to make sure all buffered contents are written to file.
        self.file.flush()
        os.fsync(self.file.fileno())
        # Release the lock on the file.
        unlock_file(self.file)
        self.file.close()
        # Handle exceptions that may have come up during execution, by
        # default any exceptions are raised to the user.
        if (exc_type != None): return False
        else:                  return True        

Now, AtomicOpen can be used in a with block where one would normally use an open statement.

WARNINGS:

  • If running on Windows and Python crashes before exit is called, I’m not sure what the lock behavior would be.
  • The locking provided here is advisory, not absolute. All potentially competing processes must use the “AtomicOpen” class.
  • As of (Nov 9th, 2020) this code only locks writable files on Posix systems. At some point after the posting and before this date, it became illegal to use the fcntl.lock on read-only files.

回答 3

我更喜欢lockfile —与平台无关的文件锁定

I prefer lockfile — Platform-independent file locking


回答 4

我一直在寻找几种解决方案,而我的选择是 oslo.concurrency

它功能强大且有据可查。它基于紧固件。

其他解决方案:

I have been looking at several solutions to do that and my choice has been oslo.concurrency

It’s powerful and relatively well documented. It’s based on fasteners.

Other solutions:


回答 5

锁定是特定于平台和设备的,但是通常,您有几种选择:

  1. 使用flock()或同等功能(如果您的操作系统支持)。这是建议性锁定,除非您检查该锁定,否则它将被忽略。
  2. 使用lock-copy-move-unlock方法,在该方法中,您复制文件,写入新数据然后移动它(移动而不是复制-移动是Linux中的原子操作-检查您的操作系统),然后检查是否存在锁定文件的存在。
  3. 将目录用作“锁定”。如果要写入NFS,这是必需的,因为NFS不支持flock()。
  4. 还有可能在进程之间使用共享内存,但是我从未尝试过。这是非常特定于操作系统的。

对于所有这些方法,您都必须使用自旋锁(失败后重试)技术来获取和测试该锁。这确实为误同步留了一个小窗口,但通常足够小,不会成为主要问题。

如果您正在寻找跨平台的解决方案,那么最好通过其他机制登录到另一个系统(第二个最好的方法是上面的NFS技术)。

请注意,sqlite在NFS上受到的限制与普通文件相同,因此您无法在网络共享上写入sqlite数据库并免费获得同步。

Locking is platform and device specific, but generally, you have a few options:

  1. Use flock(), or equivalent (if your os supports it). This is advisory locking, unless you check for the lock, its ignored.
  2. Use a lock-copy-move-unlock methodology, where you copy the file, write the new data, then move it (move, not copy – move is an atomic operation in Linux — check your OS), and you check for the existence of the lock file.
  3. Use a directory as a “lock”. This is necessary if you’re writing to NFS, since NFS doesn’t support flock().
  4. There’s also the possibility of using shared memory between the processes, but I’ve never tried that; it’s very OS-specific.

For all these methods, you’ll have to use a spin-lock (retry-after-failure) technique for acquiring and testing the lock. This does leave a small window for mis-synchronization, but its generally small enough to not be an major issue.

If you’re looking for a solution that is cross platform, then you’re better off logging to another system via some other mechanism (the next best thing is the NFS technique above).

Note that sqlite is subject to the same constraints over NFS that normal files are, so you can’t write to an sqlite database on a network share and get synchronization for free.


回答 6

在操作系统级别协调对单个文件的访问充满了您可能不想解决的各种问题。

最好的选择是有一个单独的进程来协调对该文件的读/写访问。

Coordinating access to a single file at the OS level is fraught with all kinds of issues that you probably don’t want to solve.

Your best bet is have a separate process that coordinates read/write access to that file.


回答 7

锁定文件通常是特定于平台的操作,因此您可能需要考虑在不同操作系统上运行的可能性。例如:

import os

def my_lock(f):
    if os.name == "posix":
        # Unix or OS X specific locking here
    elif os.name == "nt":
        # Windows specific locking here
    else:
        print "Unknown operating system, lock unavailable"

Locking a file is usually a platform-specific operation, so you may need to allow for the possibility of running on different operating systems. For example:

import os

def my_lock(f):
    if os.name == "posix":
        # Unix or OS X specific locking here
    elif os.name == "nt":
        # Windows specific locking here
    else:
        print "Unknown operating system, lock unavailable"

回答 8

我一直在处理这样的情况,即我从同一目录/文件夹中运行同一程序的多个副本,并记录错误。我的方法是在打开日志文件之前将“锁定文件”写入光盘。程序在继续之前检查“锁定文件”的存在,并在“锁定文件”存在的情况下等待其轮到。

这是代码:

def errlogger(error):

    while True:
        if not exists('errloglock'):
            lock = open('errloglock', 'w')
            if exists('errorlog'): log = open('errorlog', 'a')
            else: log = open('errorlog', 'w')
            log.write(str(datetime.utcnow())[0:-7] + ' ' + error + '\n')
            log.close()
            remove('errloglock')
            return
        else:
            check = stat('errloglock')
            if time() - check.st_ctime > 0.01: remove('errloglock')
            print('waiting my turn')

编辑—考虑了以上有关过时锁定的一些评论之后,我编辑了代码,以添加“锁定文件”过时检查。在我的系统上,对该函数定时进行数千次迭代,平均之前为0.002066 …秒:

lock = open('errloglock', 'w')

到之后:

remove('errloglock')

因此,我认为我将以5倍的数量开始,以表明陈旧性并监视问题情况。

另外,当我在计时的时候,我意识到我确实有一些不必要的代码:

lock.close()

我已经紧跟在open语句之后,因此在此编辑中已将其删除。

I have been working on a situation like this where I run multiple copies of the same program from within the same directory/folder and logging errors. My approach was to write a “lock file” to the disc before opening the log file. The program checks for the presence of the “lock file” before proceeding, and waits for its turn if the “lock file” exists.

Here is the code:

def errlogger(error):

    while True:
        if not exists('errloglock'):
            lock = open('errloglock', 'w')
            if exists('errorlog'): log = open('errorlog', 'a')
            else: log = open('errorlog', 'w')
            log.write(str(datetime.utcnow())[0:-7] + ' ' + error + '\n')
            log.close()
            remove('errloglock')
            return
        else:
            check = stat('errloglock')
            if time() - check.st_ctime > 0.01: remove('errloglock')
            print('waiting my turn')

EDIT— After thinking over some of the comments about stale locks above I edited the code to add a check for staleness of the “lock file.” Timing several thousand iterations of this function on my system gave and average of 0.002066… seconds from just before:

lock = open('errloglock', 'w')

to just after:

remove('errloglock')

so I figured I will start with 5 times that amount to indicate staleness and monitor the situation for problems.

Also, as I was working with the timing, I realized that I had a bit of code that was not really necessary:

lock.close()

which I had immediately following the open statement, so I have removed it in this edit.


回答 9

为了补充Evan Fossmark的答案,这是一个如何使用filelock的示例

from filelock import FileLock

lockfile = r"c:\scr.txt"
lock = FileLock(lockfile + ".lock")
with lock:
    file = open(path, "w")
    file.write("123")
    file.close()

with lock:块中的任何代码都是线程安全的,这意味着它将在另一个进程访问该文件之前完成。

To add on to Evan Fossmark’s answer, here’s an example of how to use filelock:

from filelock import FileLock

lockfile = r"c:\scr.txt"
lock = FileLock(lockfile + ".lock")
with lock:
    file = open(path, "w")
    file.write("123")
    file.close()

Any code within the with lock: block is thread-safe, meaning that it will be finished before another process has access to the file.


回答 10

方案是这样的:用户请求一个文件做一些事情。然后,如果用户再次发送相同的请求,它将通知用户第二个请求直到第一个请求完成后才完成。这就是为什么我使用锁定机制来处理此问题。

这是我的工作代码:

from lockfile import LockFile
lock = LockFile(lock_file_path)
status = ""
if not lock.is_locked():
    lock.acquire()
    status = lock.path + ' is locked.'
    print status
else:
    status = lock.path + " is already locked."
    print status

return status

The scenario is like that: The user requests a file to do something. Then, if the user sends the same request again, it informs the user that the second request is not done until the first request finishes. That’s why, I use lock-mechanism to handle this issue.

Here is my working code:

from lockfile import LockFile
lock = LockFile(lock_file_path)
status = ""
if not lock.is_locked():
    lock.acquire()
    status = lock.path + ' is locked.'
    print status
else:
    status = lock.path + " is already locked."
    print status

return status

回答 11

我从grizzled-python 找到了一个简单且可行的实现

简单使用os.open(…,O_EXCL)+ os.close()在Windows上不起作用。

I found a simple and worked(!) implementation from grizzled-python.

Simple use os.open(…, O_EXCL) + os.close() didn’t work on windows.


回答 12

您可能会发现pylocker非常有用。它通常可以用于锁定文件或用于锁定机制,并且可以一次从多个Python进程进行访问。

如果您只是想锁定文件,请按以下步骤操作:

import uuid
from pylocker import Locker

#  create a unique lock pass. This can be any string.
lpass = str(uuid.uuid1())

# create locker instance.
FL = Locker(filePath='myfile.txt', lockPass=lpass, mode='w')

# aquire the lock
with FL as r:
    # get the result
    acquired, code, fd  = r

    # check if aquired.
    if fd is not None:
        print fd
        fd.write("I have succesfuly aquired the lock !")

# no need to release anything or to close the file descriptor, 
# with statement takes care of that. let's print fd and verify that.
print fd

You may find pylocker very useful. It can be used to lock a file or for locking mechanisms in general and can be accessed from multiple Python processes at once.

If you simply want to lock a file here’s how it works:

import uuid
from pylocker import Locker

#  create a unique lock pass. This can be any string.
lpass = str(uuid.uuid1())

# create locker instance.
FL = Locker(filePath='myfile.txt', lockPass=lpass, mode='w')

# aquire the lock
with FL as r:
    # get the result
    acquired, code, fd  = r

    # check if aquired.
    if fd is not None:
        print fd
        fd.write("I have succesfuly aquired the lock !")

# no need to release anything or to close the file descriptor, 
# with statement takes care of that. let's print fd and verify that.
print fd