Python 实用宝典

Question 1

I’m using Python, and would like to insert a string into a text file without deleting or copying the file. How can I do that?

Question 2

Unfortunately there is no way to insert into the middle of a file without re-writing it. As previous posters have indicated, you can append to a file or overwrite part of it using seek but if you want to add stuff at the beginning or the middle, you’ll have to rewrite it.

This is an operating system thing, not a Python thing. It is the same in all languages.

What I usually do is read from the file, make the modifications and write it out to a new file called myfile.txt.tmp or something like that. This is better than reading the whole file into memory because the file may be too large for that. Once the temporary file is completed, I rename it the same as the original file.

This is a good, safe way to do it because if the file write crashes or aborts for any reason, you still have your untouched original file.

Question 3

What’s the difference between file and open in Python? When should I use which one? (Say I’m in 2.5)

Question 4

You should always use open().

As the documentation states:

When opening a file, it’s preferable to use open() instead of invoking this constructor directly. file is more suited to type testing (for example, writing “isinstance(f, file)”).

Also, file() has been removed since Python 3.0.

Question 5

Two reasons: The python philosophy of “There ought to be one way to do it” and file is going away.

file is the actual type (using e.g. file('myfile.txt') is calling its constructor). open is a factory function that will return a file object.

In python 3.0 file is going to move from being a built-in to being implemented by multiple classes in the io library (somewhat similar to Java with buffered readers, etc.)

Question 6

file() is a type, like an int or a list. open() is a function for opening files, and will return a file object.

This is an example of when you should use open:

f = open(filename, 'r')
for line in f:
    process(line)
f.close()

This is an example of when you should use file:

class LoggingFile(file):
    def write(self, data):
        sys.stderr.write("Wrote %d bytes\n" % len(data))
        super(LoggingFile, self).write(data)

As you can see, there’s a good reason for both to exist, and a clear use-case for both.

Question 7

Functionally, the two are the same; open will call file anyway, so currently the difference is a matter of style. The Python docs recommend using open.

When opening a file, it’s preferable to use open() instead of invoking the file constructor directly.

The reason is that in future versions they is not guaranteed to be the same (open will become a factory function, which returns objects of different types depending on the path it’s opening).

Question 8

Only ever use open() for opening files. file() is actually being removed in 3.0, and it’s deprecated at the moment. They’ve had a sort of strange relationship, but file() is going now, so there’s no need to worry anymore.

The following is from the Python 2.6 docs. [bracket stuff] added by me.

When opening a file, it’s preferable to use open() instead of invoking this [file()] constructor directly. file is more suited to type testing (for example, writing isinstance(f, file)

Question 9

According to Mr Van Rossum, although open() is currently an alias for file() you should use open() because this might change in the future.

Question 10

Python recognizes the following as instruction which defines file’s encoding:

# -*- coding: utf-8 -*-

I definitely saw this kind of instructions before (-*- var: value -*-). Where does it come from? What is the full specification, e.g. can the value include spaces, special symbols, newlines, even -*- itself?

My program will be writing plain text files and I’d like to include some metadata in them using this format.

Question 11

This way of specifying the encoding of a Python file comes from PEP 0263 – Defining Python Source Code Encodings.

It is also recognized by GNU Emacs (see Python Language Reference, 2.1.4 Encoding declarations), though I don’t know if it was the first program to use that syntax.

Question 12

# -*- coding: utf-8 -*- is a Python 2 thing. In Python 3+, the default encoding of source files is already UTF-8 and that line is useless.

See: Should I use encoding declaration in Python 3?

pyupgrade is a tool you can run on your code to remove those comments and other no-longer-useful leftovers from Python 2, like having all your classes inherit from object.

Question 13

This is so called file local variables, that are understood by Emacs and set correspondingly. See corresponding section in Emacs manual – you can define them either in header or in footer of file

Question 14

In PyCharm, I’d leave it out. It turns off the UTF-8 indicator at the bottom with a warning that the encoding is hard-coded. Don’t think you need the PyCharm comment mentioned above.

Question 15

How can I import variables from one file to another?

example: file1 has the variables x1 and x2 how to pass them to file2?

How can I import all of the variables from one to another?

Question 16

from file1 import *

will import all objects and methods in file1

Question 17

Import file1 inside file2:

To import all variables from file1 without flooding file2’s namespace, use:

import file1

#now use file1.x1, file2.x2, ... to access those variables

To import all variables from file1 to file2’s namespace( not recommended):

from file1 import *
#now use x1, x2..

From the docs:

While it is valid to use from module import * at module level it is usually a bad idea. For one, this loses an important property Python otherwise has — you can know where each toplevel name is defined by a simple “search” function in your favourite editor. You also open yourself to trouble in the future, if some module grows additional functions or classes.

Question 18

Best to import x1 and x2 explicitly:

from file1 import x1, x2

This allows you to avoid unnecessary namespace conflicts with variables and functions from file1 while working in file2.

But if you really want, you can import all the variables:

from file1 import *

Question 19

Actually this is not really the same to import a variable with:

from file1 import x1
print(x1)

and

import file1
print(file1.x1)

Altough at import time x1 and file1.x1 have the same value, they are not the same variables. For instance, call a function in file1 that modifies x1 and then try to print the variable from the main file: you will not see the modified value.

Question 20

Marc response is correct. Actually, you can print the memory address for the variables print(hex(id(libvar)) and you can see the addresses are different.

# mylib.py
libvar = None
def lib_method():
    global libvar
    print(hex(id(libvar)))

# myapp.py
from mylib import libvar, lib_method
import mylib

lib_method()
print(hex(id(libvar)))
print(hex(id(mylib.libvar)))

Question 21

script1.py

title="Hello world"

script2.py is where we using script1 variable

Method 1:

import script1
print(script1.title)

Method 2:

from script1 import title
print(title)

Question 22

In Python you can access the contents of other files like as if they
are some kind of a library, compared to other languages like java or any oop base languages , This is really cool ;

This makes accessing the contents of the file or import it to to process it or to do anything with it ; And that is the Main reason why Python is highly preferred Language for Data Science and Machine Learning etc. ;

And this is the picture of project structure

Where I am accessing variables from .env file where the API links and Secret keys reside .

General Structure:

from <File-Name> import *

Question 23

first.py:

a=5

second.py:

import first
print(first.a)

The result will be 5.

Question 24

Suppose python code is executed in not known by prior windows directory say ‘main’ , and wherever code is installed when it runs it needs to access to directory ‘main/2091/data.txt’ .

how should I use open(location) function? what should be location ?

Edit :

I found that below simple code will work..does it have any disadvantages ?

    file="\2091\sample.txt"
    path=os.getcwd()+file
    fp=open(path,'r+');

Question 25

With this type of thing you need to be careful what your actual working directory is. For example, you may not run the script from the directory the file is in. In this case, you can’t just use a relative path by itself.

If you are sure the file you want is in a subdirectory beneath where the script is actually located, you can use __file__ to help you out here. __file__ is the full path to where the script you are running is located.

So you can fiddle with something like this:

import os
script_dir = os.path.dirname(__file__) #<-- absolute dir the script is in
rel_path = "2091/data.txt"
abs_file_path = os.path.join(script_dir, rel_path)

Question 26

This code works fine:

import os


def readFile(filename):
    filehandle = open(filename)
    print filehandle.read()
    filehandle.close()



fileDir = os.path.dirname(os.path.realpath('__file__'))
print fileDir

#For accessing the file in the same folder
filename = "same.txt"
readFile(filename)

#For accessing the file in a folder contained in the current folder
filename = os.path.join(fileDir, 'Folder1.1/same.txt')
readFile(filename)

#For accessing the file in the parent folder of the current folder
filename = os.path.join(fileDir, '../same.txt')
readFile(filename)

#For accessing the file inside a sibling folder.
filename = os.path.join(fileDir, '../Folder2/same.txt')
filename = os.path.abspath(os.path.realpath(filename))
print filename
readFile(filename)

Question 27

I created an account just so I could clarify a discrepancy I think I found in Russ’s original response.

For reference, his original answer was:

import os
script_dir = os.path.dirname(__file__)
rel_path = "2091/data.txt"
abs_file_path = os.path.join(script_dir, rel_path)

This is a great answer because it is trying to dynamically creates an absolute system path to the desired file.

Cory Mawhorter noticed that __file__ is a relative path (it is as well on my system) and suggested using os.path.abspath(__file__). os.path.abspath, however, returns the absolute path of your current script (i.e. /path/to/dir/foobar.py)

To use this method (and how I eventually got it working) you have to remove the script name from the end of the path:

import os
script_path = os.path.abspath(__file__) # i.e. /path/to/dir/foobar.py
script_dir = os.path.split(script_path)[0] #i.e. /path/to/dir/
rel_path = "2091/data.txt"
abs_file_path = os.path.join(script_dir, rel_path)

The resulting abs_file_path (in this example) becomes: /path/to/dir/2091/data.txt

Question 28

It depends on what operating system you’re using. If you want a solution that is compatible with both Windows and *nix something like:

from os import path

file_path = path.relpath("2091/data.txt")
with open(file_path) as f:
    <do stuff>

should work fine.

The path module is able to format a path for whatever operating system it’s running on. Also, python handles relative paths just fine, so long as you have correct permissions.

Edit:

As mentioned by kindall in the comments, python can convert between unix-style and windows-style paths anyway, so even simpler code will work:

with open("2091/data/txt") as f:
    <do stuff>

That being said, the path module still has some useful functions.

Question 29

I spend a lot time to discover why my code could not find my file running Python 3 on the Windows system. So I added . before / and everything worked fine:

import os

script_dir = os.path.dirname(__file__)
file_path = os.path.join(script_dir, './output03.txt')
print(file_path)
fptr = open(file_path, 'w')

Question 30

Code:

import os
script_path = os.path.abspath(__file__) 
path_list = script_path.split(os.sep)
script_directory = path_list[0:len(path_list)-1]
rel_path = "main/2091/data.txt"
path = "/".join(script_directory) + "/" + rel_path

Explanation:

Import library:

import os

Use __file__ to attain the current script’s path:

script_path = os.path.abspath(__file__)

Separates the script path into multiple items:

path_list = script_path.split(os.sep)

Remove the last item in the list (the actual script file):

script_directory = path_list[0:len(path_list)-1]

Add the relative file’s path:

rel_path = "main/2091/data.txt

Join the list items, and addition the relative path’s file:

path = "/".join(script_directory) + "/" + rel_path

Now you are set to do whatever you want with the file, such as, for example:

file = open(path)

Question 31

If the file is in your parent folder, eg. follower.txt, you can simply use open('../follower.txt', 'r').read()

Question 32

Try this:

from pathlib import Path

data_folder = Path("/relative/path")
file_to_open = data_folder / "file.pdf"

f = open(file_to_open)

print(f.read())

Python 3.4 introduced a new standard library for dealing with files and paths called pathlib. It works for me!

Question 33

Not sure if this work everywhere.

I’m using ipython in ubuntu.

If you want to read file in current folder’s sub-directory:

/current-folder/sub-directory/data.csv

your script is in current-folder simply try this:

import pandas as pd
path = './sub-directory/data.csv'
pd.read_csv(path)

Question 34

Python just passes the filename you give it to the operating system, which opens it. If your operating system supports relative paths like main/2091/data.txt (hint: it does), then that will work fine.

You may find that the easiest way to answer a question like this is to try it and see what happens.

Question 35

import os
def file_path(relative_path):
    dir = os.path.dirname(os.path.abspath(__file__))
    split_path = relative_path.split("/")
    new_path = os.path.join(dir, *split_path)
    return new_path

with open(file_path("2091/data.txt"), "w") as f:
    f.write("Powerful you have become.")

Question 36

When I was a beginner I found these descriptions a bit intimidating. As at first I would try For Windows

f= open('C:\Users\chidu\Desktop\Skipper New\Special_Note.txt','w+')
print(f)

and this would raise an syntax error. I used get confused alot. Then after some surfing across google. found why the error occurred. Writing this for beginners

It’s because for path to be read in Unicode you simple add a \ when starting file path

f= open('C:\\Users\chidu\Desktop\Skipper New\Special_Note.txt','w+')
print(f)

And now it works just add \ before starting the directory.

Question 37

I’m trying to make a script to list all directory, subdirectory, and files in a given directory.
I tried this:

import sys,os

root = "/home/patate/directory/"
path = os.path.join(root, "targetdirectory")

for r,d,f in os.walk(path):
    for file in f:
        print os.path.join(root,file)

Unfortunatly it doesn’t work properly.
I get all the files, but not their complete paths.

For example if the dir struct would be:

/home/patate/directory/targetdirectory/123/456/789/file.txt

It would print:

/home/patate/directory/targetdirectory/file.txt

What I need is the first result. Any help would be greatly appreciated! Thanks.

Question 38

Use os.path.join to concatenate the directory and file name:

for path, subdirs, files in os.walk(root):
    for name in files:
        print os.path.join(path, name)

Note the usage of path and not root in the concatenation, since using root would be incorrect.

In Python 3.4, the pathlib module was added for easier path manipulations. So the equivalent to os.path.join would be:

pathlib.PurePath(path, name)

The advantage of pathlib is that you can use a variety of useful methods on paths. If you use the concrete Path variant you can also do actual OS calls through them, like changing into a directory, deleting the path, opening the file it points to and much more.

Question 39

Just in case… Getting all files in the directory and subdirectories matching some pattern (*.py for example):

import os
from fnmatch import fnmatch

root = '/some/directory'
pattern = "*.py"

for path, subdirs, files in os.walk(root):
    for name in files:
        if fnmatch(name, pattern):
            print os.path.join(path, name)

Question 40

Here is a one-liner:

import os

[val for sublist in [[os.path.join(i[0], j) for j in i[2]] for i in os.walk('./')] for val in sublist]
# Meta comment to ease selecting text

The outer most val for sublist in ... loop flattens the list to be one dimensional. The j loop collects a list of every file basename and joins it to the current path. Finally, the i loop iterates over all directories and sub directories.

This example uses the hard-coded path ./ in the os.walk(...) call, you can supplement any path string you like.

Note: os.path.expanduser and/or os.path.expandvars can be used for paths strings like ~/

Extending this example:

Its easy to add in file basename tests and directoryname tests.

For Example, testing for *.jpg files:

... for j in i[2] if j.endswith('.jpg')] ...

Additionally, excluding the .git directory:

... for i in os.walk('./') if '.git' not in i[0].split('/')]

Question 41

Couldn’t comment so writing answer here. This is the clearest one-line I have seen:

import os
[os.path.join(path, name) for path, subdirs, files in os.walk(root) for name in files]

Question 42

You can take a look at this sample I made. It uses the os.path.walk function which is deprecated beware.Uses a list to store all the filepaths

root = "Your root directory"
ex = ".txt"
where_to = "Wherever you wanna write your file to"
def fileWalker(ext,dirname,names):
    '''
    checks files in names'''
    pat = "*" + ext[0]
    for f in names:
        if fnmatch.fnmatch(f,pat):
            ext[1].append(os.path.join(dirname,f))


def writeTo(fList):

    with open(where_to,"w") as f:
        for di_r in fList:
            f.write(di_r + "\n")






if __name__ == '__main__':
    li = []
    os.path.walk(root,fileWalker,[ex,li])

    writeTo(li)

Question 43

A bit simpler one-liner:

import os
from itertools import product, chain

chain.from_iterable([[os.sep.join(w) for w in product([i[0]], i[2])] for i in os.walk(dir)])

Question 44

Since every example here is just using walk (with join), i’d like to show a nice example and comparison with listdir:

import os, time

def listFiles1(root): # listdir
    allFiles = []; walk = [root]
    while walk:
        folder = walk.pop(0)+"/"; items = os.listdir(folder) # items = folders + files
        for i in items: i=folder+i; (walk if os.path.isdir(i) else allFiles).append(i)
    return allFiles

def listFiles2(root): # listdir/join (takes ~1.4x as long) (and uses '\\' instead)
    allFiles = []; walk = [root]
    while walk:
        folder = walk.pop(0); items = os.listdir(folder) # items = folders + files
        for i in items: i=os.path.join(folder,i); (walk if os.path.isdir(i) else allFiles).append(i)
    return allFiles

def listFiles3(root): # walk (takes ~1.5x as long)
    allFiles = []
    for folder, folders, files in os.walk(root):
        for file in files: allFiles+=[folder.replace("\\","/")+"/"+file] # folder+"\\"+file still ~1.5x
    return allFiles

def listFiles4(root): # walk/join (takes ~1.6x as long) (and uses '\\' instead)
    allFiles = []
    for folder, folders, files in os.walk(root):
        for file in files: allFiles+=[os.path.join(folder,file)]
    return allFiles


for i in range(100): files = listFiles1("src") # warm up

start = time.time()
for i in range(100): files = listFiles1("src") # listdir
print("Time taken: %.2fs"%(time.time()-start)) # 0.28s

start = time.time()
for i in range(100): files = listFiles2("src") # listdir and join
print("Time taken: %.2fs"%(time.time()-start)) # 0.38s

start = time.time()
for i in range(100): files = listFiles3("src") # walk
print("Time taken: %.2fs"%(time.time()-start)) # 0.42s

start = time.time()
for i in range(100): files = listFiles4("src") # walk and join
print("Time taken: %.2fs"%(time.time()-start)) # 0.47s

So as you can see for yourself, the listdir version is much more efficient. (and that join is slow)

Question 45

So far I have figured out how to import the file, create new files, and randomize the list.

I’m having trouble selecting only 50 items from the list randomly to write to a file?

def randomizer(input,output1='random_1.txt',output2='random_2.txt',output3='random_3.txt',output4='random_total.txt'):

#Input file 
    query=open(input,'r').read().split()
    dir,file=os.path.split(input)

    temp1 = os.path.join(dir,output1)
    temp2 = os.path.join(dir,output2)
    temp3 = os.path.join(dir,output3)
    temp4 = os.path.join(dir,output4)


    out_file4=open(temp4,'w')

    random.shuffle(query)

    for item in query:
        out_file4.write(item+'\n')

So if the total randomization file was

example:

random_total = ['9','2','3','1','5','6','8','7','0','4']

I would want 3 files (out_file1|2|3) with the first random set of 3, second random set of 3, and third random set of 3 (for this example, but the one I want to create should have 50)

random_1 = ['9','2','3']
random_2 = ['1','5','6']
random_3 = ['8','7','0']

So the last ‘4’ will not be included which is fine.

How can I select 50 from the list that I randomized ?

Even better, how could I select 50 at random from the original list ?

Question 46

If the list is in random order, you can just take the first 50.

Otherwise, use

import random
random.sample(the_list, 50)

random.sample help text:

sample(self, population, k) method of random.Random instance
    Chooses k unique random elements from a population sequence.

    Returns a new list containing elements from the population while
    leaving the original population unchanged.  The resulting list is
    in selection order so that all sub-slices will also be valid random
    samples.  This allows raffle winners (the sample) to be partitioned
    into grand prize and second place winners (the subslices).

    Members of the population need not be hashable or unique.  If the
    population contains repeats, then each occurrence is a possible
    selection in the sample.

    To choose a sample in a range of integers, use xrange as an argument.
    This is especially fast and space efficient for sampling from a
    large population:   sample(xrange(10000000), 60)

Question 47

One easy way to select random items is to shuffle then slice.

import random
a = [1,2,3,4,5,6,7,8,9]
random.shuffle(a)
print a[:4] # prints 4 random variables

Question 48

I think random.choice() is a better option.

import numpy as np

mylist = [13,23,14,52,6,23]

np.random.choice(mylist, 3, replace=False)

the function returns an array of 3 randomly chosen values from the list

Question 49

Say your list has 100 elements and you want to pick 50 of them in a random way. Here are the steps to follow:

Import the libraries
Create the seed for random number generator, I have put it at 2
Prepare a list of numbers from which to pick up in a random way
Make the random choices from the numbers list

Code:

from random import seed
from random import choice

seed(2)
numbers = [i for i in range(100)]

print(numbers)

for _ in range(50):
    selection = choice(numbers)
    print(selection)

Question 50

How to read a file in reverse order using python? I want to read a file from last line to first line.

Question 51

for line in reversed(open("filename").readlines()):
    print line.rstrip()

And in Python 3:

for line in reversed(list(open("filename"))):
    print(line.rstrip())

Question 52

A correct, efficient answer written as a generator.

import os

def reverse_readline(filename, buf_size=8192):
    """A generator that returns the lines of a file in reverse order"""
    with open(filename) as fh:
        segment = None
        offset = 0
        fh.seek(0, os.SEEK_END)
        file_size = remaining_size = fh.tell()
        while remaining_size > 0:
            offset = min(file_size, offset + buf_size)
            fh.seek(file_size - offset)
            buffer = fh.read(min(remaining_size, buf_size))
            remaining_size -= buf_size
            lines = buffer.split('\n')
            # The first line of the buffer is probably not a complete line so
            # we'll save it and append it to the last line of the next buffer
            # we read
            if segment is not None:
                # If the previous chunk starts right from the beginning of line
                # do not concat the segment to the last line of new chunk.
                # Instead, yield the segment first 
                if buffer[-1] != '\n':
                    lines[-1] += segment
                else:
                    yield segment
            segment = lines[0]
            for index in range(len(lines) - 1, 0, -1):
                if lines[index]:
                    yield lines[index]
        # Don't yield None if the file was empty
        if segment is not None:
            yield segment

Question 53

How about something like this:

import os


def readlines_reverse(filename):
    with open(filename) as qfile:
        qfile.seek(0, os.SEEK_END)
        position = qfile.tell()
        line = ''
        while position >= 0:
            qfile.seek(position)
            next_char = qfile.read(1)
            if next_char == "\n":
                yield line[::-1]
                line = ''
            else:
                line += next_char
            position -= 1
        yield line[::-1]


if __name__ == '__main__':
    for qline in readlines_reverse(raw_input()):
        print qline

Since the file is read character by character in reverse order, it will work even on very large files, as long as individual lines fit into memory.

Question 54

You can also use python module file_read_backwards.

After installing it, via pip install file_read_backwards (v1.2.1), you can read the entire file backwards (line-wise) in a memory efficient manner via:

#!/usr/bin/env python2.7

from file_read_backwards import FileReadBackwards

with FileReadBackwards("/path/to/file", encoding="utf-8") as frb:
    for l in frb:
         print l

It supports “utf-8″,”latin-1”, and “ascii” encodings.

Support is also available for python3. Further documentation can be found at http://file-read-backwards.readthedocs.io/en/latest/readme.html

Question 55

for line in reversed(open("file").readlines()):
    print line.rstrip()

If you are on linux, you can use tac command.

$ tac file

2 recipes you can find in ActiveState here and here

Question 56

import re

def filerev(somefile, buffer=0x20000):
  somefile.seek(0, os.SEEK_END)
  size = somefile.tell()
  lines = ['']
  rem = size % buffer
  pos = max(0, (size // buffer - 1) * buffer)
  while pos >= 0:
    somefile.seek(pos, os.SEEK_SET)
    data = somefile.read(rem + buffer) + lines[0]
    rem = 0
    lines = re.findall('[^\n]*\n?', data)
    ix = len(lines) - 2
    while ix > 0:
      yield lines[ix]
      ix -= 1
    pos -= buffer
  else:
    yield lines[0]

with open(sys.argv[1], 'r') as f:
  for line in filerev(f):
    sys.stdout.write(line)

Question 57

Accepted answer won’t work for cases with large files that won’t fit in memory (which is not a rare case).

As it was noted by others, @srohde answer looks good, but it has next issues:

openning file looks redundant, when we can pass file object & leave it to user to decide in which encoding it should be read,
even if we refactor to accept file object, it won’t work for all encodings: we can choose file with utf-8 encoding and non-ascii contents like
```
й
```
pass buf_size equal to 1 and will have
```
UnicodeDecodeError: 'utf8' codec can't decode byte 0xb9 in position 0: invalid start byte
```
of course text may be larger but buf_size may be picked up so it’ll lead to obfuscated error like above,
we can’t specify custom line separator,
we can’t choose to keep line separator.

So considering all these concerns I’ve written separate functions:

one which works with byte streams,
second one which works with text streams and delegates its underlying byte stream to the first one and decodes resulting lines.

First of all let’s define next utility functions:

ceil_division for making division with ceiling (in contrast with standard // division with floor, more info can be found in this thread)

def ceil_division(left_number, right_number):
    """
    Divides given numbers with ceiling.
    """
    return -(-left_number // right_number)

split for splitting string by given separator from right end with ability to keep it:

def split(string, separator, keep_separator):
    """
    Splits given string by given separator.
    """
    parts = string.split(separator)
    if keep_separator:
        *parts, last_part = parts
        parts = [part + separator for part in parts]
        if last_part:
            return parts + [last_part]
    return parts

read_batch_from_end to read batch from the right end of binary stream

def read_batch_from_end(byte_stream, size, end_position):
    """
    Reads batch from the end of given byte stream.
    """
    if end_position > size:
        offset = end_position - size
    else:
        offset = 0
        size = end_position
    byte_stream.seek(offset)
    return byte_stream.read(size)

After that we can define function for reading byte stream in reverse order like

import functools
import itertools
import os
from operator import methodcaller, sub


def reverse_binary_stream(byte_stream, batch_size=None,
                          lines_separator=None,
                          keep_lines_separator=True):
    if lines_separator is None:
        lines_separator = (b'\r', b'\n', b'\r\n')
        lines_splitter = methodcaller(str.splitlines.__name__,
                                      keep_lines_separator)
    else:
        lines_splitter = functools.partial(split,
                                           separator=lines_separator,
                                           keep_separator=keep_lines_separator)
    stream_size = byte_stream.seek(0, os.SEEK_END)
    if batch_size is None:
        batch_size = stream_size or 1
    batches_count = ceil_division(stream_size, batch_size)
    remaining_bytes_indicator = itertools.islice(
            itertools.accumulate(itertools.chain([stream_size],
                                                 itertools.repeat(batch_size)),
                                 sub),
            batches_count)
    try:
        remaining_bytes_count = next(remaining_bytes_indicator)
    except StopIteration:
        return

    def read_batch(position):
        result = read_batch_from_end(byte_stream,
                                     size=batch_size,
                                     end_position=position)
        while result.startswith(lines_separator):
            try:
                position = next(remaining_bytes_indicator)
            except StopIteration:
                break
            result = (read_batch_from_end(byte_stream,
                                          size=batch_size,
                                          end_position=position)
                      + result)
        return result

    batch = read_batch(remaining_bytes_count)
    segment, *lines = lines_splitter(batch)
    yield from reverse(lines)
    for remaining_bytes_count in remaining_bytes_indicator:
        batch = read_batch(remaining_bytes_count)
        lines = lines_splitter(batch)
        if batch.endswith(lines_separator):
            yield segment
        else:
            lines[-1] += segment
        segment, *lines = lines
        yield from reverse(lines)
    yield segment

and finally a function for reversing text file can be defined like:

import codecs


def reverse_file(file, batch_size=None, 
                 lines_separator=None,
                 keep_lines_separator=True):
    encoding = file.encoding
    if lines_separator is not None:
        lines_separator = lines_separator.encode(encoding)
    yield from map(functools.partial(codecs.decode,
                                     encoding=encoding),
                   reverse_binary_stream(
                           file.buffer,
                           batch_size=batch_size,
                           lines_separator=lines_separator,
                           keep_lines_separator=keep_lines_separator))

Tests

Preparations

I’ve generated 4 files using fsutil command:

empty.txt with no contents, size 0MB
tiny.txt with size of 1MB
small.txt with size of 10MB
large.txt with size of 50MB

also I’ve refactored @srohde solution to work with file object instead of file path.

Test script

from timeit import Timer

repeats_count = 7
number = 1
create_setup = ('from collections import deque\n'
                'from __main__ import reverse_file, reverse_readline\n'
                'file = open("{}")').format
srohde_solution = ('with file:\n'
                   '    deque(reverse_readline(file,\n'
                   '                           buf_size=8192),'
                   '          maxlen=0)')
azat_ibrakov_solution = ('with file:\n'
                         '    deque(reverse_file(file,\n'
                         '                       lines_separator="\\n",\n'
                         '                       keep_lines_separator=False,\n'
                         '                       batch_size=8192), maxlen=0)')
print('reversing empty file by "srohde"',
      min(Timer(srohde_solution,
                create_setup('empty.txt')).repeat(repeats_count, number)))
print('reversing empty file by "Azat Ibrakov"',
      min(Timer(azat_ibrakov_solution,
                create_setup('empty.txt')).repeat(repeats_count, number)))
print('reversing tiny file (1MB) by "srohde"',
      min(Timer(srohde_solution,
                create_setup('tiny.txt')).repeat(repeats_count, number)))
print('reversing tiny file (1MB) by "Azat Ibrakov"',
      min(Timer(azat_ibrakov_solution,
                create_setup('tiny.txt')).repeat(repeats_count, number)))
print('reversing small file (10MB) by "srohde"',
      min(Timer(srohde_solution,
                create_setup('small.txt')).repeat(repeats_count, number)))
print('reversing small file (10MB) by "Azat Ibrakov"',
      min(Timer(azat_ibrakov_solution,
                create_setup('small.txt')).repeat(repeats_count, number)))
print('reversing large file (50MB) by "srohde"',
      min(Timer(srohde_solution,
                create_setup('large.txt')).repeat(repeats_count, number)))
print('reversing large file (50MB) by "Azat Ibrakov"',
      min(Timer(azat_ibrakov_solution,
                create_setup('large.txt')).repeat(repeats_count, number)))

Note: I’ve used collections.deque class to exhaust generator.

Outputs

For PyPy 3.5 on Windows 10:

reversing empty file by "srohde" 8.31e-05
reversing empty file by "Azat Ibrakov" 0.00016090000000000028
reversing tiny file (1MB) by "srohde" 0.160081
reversing tiny file (1MB) by "Azat Ibrakov" 0.09594989999999998
reversing small file (10MB) by "srohde" 8.8891863
reversing small file (10MB) by "Azat Ibrakov" 5.323388100000001
reversing large file (50MB) by "srohde" 186.5338368
reversing large file (50MB) by "Azat Ibrakov" 99.07450229999998

For CPython 3.5 on Windows 10:

reversing empty file by "srohde" 3.600000000000001e-05
reversing empty file by "Azat Ibrakov" 4.519999999999958e-05
reversing tiny file (1MB) by "srohde" 0.01965560000000001
reversing tiny file (1MB) by "Azat Ibrakov" 0.019207699999999994
reversing small file (10MB) by "srohde" 3.1341862999999996
reversing small file (10MB) by "Azat Ibrakov" 3.0872588000000007
reversing large file (50MB) by "srohde" 82.01206720000002
reversing large file (50MB) by "Azat Ibrakov" 82.16775059999998

So as we can see it performs like original solution, but is more general and free of its disadvantages listed above.

问题：如何修改文本文件？

回答 0

问题：Python-何时使用文件vs打开

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

问题：这是从哪里来的：-*-编码：utf-8-*-

回答 0

回答 1

回答 2

回答 3

问题：从另一个文件导入变量？

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

回答 7

问题：在Python的相对位置打开文件

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

回答 7

回答 8

回答 9

回答 10

回答 11

问题：Python列表目录，子目录和文件

回答 0

回答 1

回答 2

这里是单线：

扩展此示例：

Here is a one-liner:

Extending this example:

回答 3

回答 4

回答 5

回答 6

问题：从列表中随机选择50个项目写入文件

回答 0

回答 1

回答 2

回答 3

问题：如何以相反的顺序读取文件？

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

测验

准备工作

测试脚本

产出

广告

Tests

Preparations

Test script

Outputs

Advertisement

回答 7

回答 8

回答 9

回答 10

回答 11

回答 12

回答 13

回答 14

回答 15

问题：这是从哪里来的：--编码：utf-8--