Python 实用宝典

Question 1

I found, that there is related question, about how to find if at least one item exists in a list:
How to check if one of the following items is in a list?

But what is the best and pythonic way to find whether all items exists in a list?

Searching through the docs I found this solution:

>>> l = ['a', 'b', 'c']
>>> set(['a', 'b']) <= set(l)
True
>>> set(['a', 'x']) <= set(l)
False

Other solution would be this:

>>> l = ['a', 'b', 'c']
>>> all(x in l for x in ['a', 'b'])
True
>>> all(x in l for x in ['a', 'x'])
False

But here you must do more typing.

Is there any other solutions?

Question 2

Operators like <= in Python are generally not overriden to mean something significantly different than “less than or equal to”. It’s unusual for the standard library does this–it smells like legacy API to me.

Use the equivalent and more clearly-named method, set.issubset. Note that you don’t need to convert the argument to a set; it’ll do that for you if needed.

set(['a', 'b']).issubset(['a', 'b', 'c'])

Question 3

I would probably use set in the following manner :

set(l).issuperset(set(['a','b']))

or the other way round :

set(['a','b']).issubset(set(l))

I find it a bit more readable, but it may be over-kill. Sets are particularly useful to compute union/intersection/differences between collections, but it may not be the best option in this situation …

Question 4

I like these two because they seem the most logical, the latter being shorter and probably fastest (shown here using set literal syntax which has been backported to Python 2.7):

all(x in {'a', 'b', 'c'} for x in ['a', 'b'])
#   or
{'a', 'b'}.issubset({'a', 'b', 'c'})

Question 5

What if your lists contain duplicates like this:

v1 = ['s', 'h', 'e', 'e', 'p']
v2 = ['s', 's', 'h']

Sets do not contain duplicates. So, the following line returns True.

set(v2).issubset(v1)

To count for duplicates, you can use the code:

v1 = sorted(v1)
v2 = sorted(v2)


def is_subseq(v2, v1):
    """Check whether v2 is a subsequence of v1."""
    it = iter(v1)
    return all(c in it for c in v2)

So, the following line returns False.

is_subseq(v2, v1)

Question 6

This was what I was searching online but unfortunately found not online but while experimenting on python interpreter.

>>> case  = "caseCamel"
>>> label = "Case Camel"
>>> list  = ["apple", "banana"]
>>>
>>> (case or label) in list
False
>>> list = ["apple", "caseCamel"]
>>> (case or label) in list
True
>>> (case and label) in list
False
>>> list = ["case", "caseCamel", "Case Camel"]
>>> (case and label) in list
True
>>>

and if you have a looong list of variables held in a sublist variable

>>>
>>> list  = ["case", "caseCamel", "Case Camel"]
>>> label = "Case Camel"
>>> case  = "caseCamel"
>>>
>>> sublist = ["unique banana", "very unique banana"]
>>>
>>> # example for if any (at least one) item contained in superset (or statement)
...
>>> next((True for item in sublist if next((True for x in list if x == item), False)), False)
False
>>>
>>> sublist[0] = label
>>>
>>> next((True for item in sublist if next((True for x in list if x == item), False)), False)
True
>>>
>>> # example for whether a subset (all items) contained in superset (and statement)
...
>>> # a bit of demorgan's law
...
>>> next((False for item in sublist if item not in list), True)
False
>>>
>>> sublist[1] = case
>>>
>>> next((False for item in sublist if item not in list), True)
True
>>>
>>> next((True for item in sublist if next((True for x in list if x == item), False)), False)
True
>>>
>>>

Question 7

An example of how to do this using a lambda expression would be:

issublist = lambda x, y: 0 in [_ in x for _ in y]

Question 8

Not OP’s case, but – for anyone who wants to assert intersection in dicts and ended up here due to poor googling (e.g. me) – you need to work with dict.items:

>>> a = {'key': 'value'}
>>> b = {'key': 'value', 'extra_key': 'extra_value'}
>>> all(item in a.items() for item in b.items())
True
>>> all(item in b.items() for item in a.items())
False

That’s because dict.items returns tuples of key/value pairs, and much like any object in Python, they’re interchangeably comparable

Question 9

Finally I migrated my development env from runserver to gunicorn/nginx.

It’d be convenient to replicate the autoreload feature of runserver to gunicorn, so the server automatically restarts when source changes. Otherwise I have to restart the server manually with kill -HUP.

Any way to avoid the manual restart?

Question 10

While this is old question you need to know that ever since version 19.0 gunicorn has had the --reload option. So now no third party tools are needed.

Question 11

One option would be to use the –max-requests to limit each spawned process to serving only one request by adding --max-requests 1 to the startup options. Every newly spawned process should see your code changes and in a development environment the extra startup time per request should be negligible.

Question 12

Bryan Helmig came up with this and I modified it to use run_gunicorn instead of launching gunicorn directly, to make it possible to just cut and paste these 3 commands into a shell in your django project root folder (with your virtualenv activated):

pip install watchdog -U
watchmedo shell-command --patterns="*.py;*.html;*.css;*.js" --recursive --command='echo "${watch_src_path}" && kill -HUP `cat gunicorn.pid`' . &
python manage.py run_gunicorn 127.0.0.1:80 --pid=gunicorn.pid

Question 13

I use git push to deploy to production and set up git hooks to run a script. The advantage of this approach is you can also do your migration and package installation at the same time. https://mikeeverhart.net/2013/01/using-git-to-deploy-code/

mkdir -p /home/git/project_name.git
cd /home/git/project_name.git
git init --bare

Then create a script /home/git/project_name.git/hooks/post-receive.

#!/bin/bash
GIT_WORK_TREE=/path/to/project git checkout -f
source /path/to/virtualenv/activate
pip install -r /path/to/project/requirements.txt
python /path/to/project/manage.py migrate
sudo supervisorctl restart project_name

Make sure to chmod u+x post-receive, and add user to sudoers. Allow it to run sudo supervisorctl without password. https://www.cyberciti.biz/faq/linux-unix-running-sudo-command-without-a-password/

From my local / development server, I set up git remote that allows me to push to the production server

git remote add production ssh://user_name@production-server/home/git/project_name.git

# initial push
git push production +master:refs/heads/master

# subsequent push
git push production master

As a bonus, you will get to see all the prompts as the script is running. So you will see if there is any issue with the migration/package installation/supervisor restart.

Question 14

Given a list of numbers, how does one find differences between every (i)-th elements and its (i+1)-th?

Is it better to use a lambda expression or maybe a list comprehension?

For example:

Given a list t=[1,3,6,...], the goal is to find a list v=[2,3,...] because 3-1=2, 6-3=3, etc.

Question 15

>>> t
[1, 3, 6]
>>> [j-i for i, j in zip(t[:-1], t[1:])]  # or use itertools.izip in py2k
[2, 3]

Question 16

The other answers are correct but if you’re doing numerical work, you might want to consider numpy. Using numpy, the answer is:

v = numpy.diff(t)

Question 17

If you don’t want to use numpy nor zip, you can use the following solution:

>>> t = [1, 3, 6]
>>> v = [t[i+1]-t[i] for i in range(len(t)-1)]
>>> v
[2, 3]

Question 18

You can use itertools.tee and zip to efficiently build the result:

from itertools import tee
# python2 only:
#from itertools import izip as zip

def differences(seq):
    iterable, copied = tee(seq)
    next(copied)
    for x, y in zip(iterable, copied):
        yield y - x

Or using itertools.islice instead:

from itertools import islice

def differences(seq):
    nexts = islice(seq, 1, None)
    for x, y in zip(seq, nexts):
        yield y - x

You can also avoid using the itertools module:

def differences(seq):
    iterable = iter(seq)
    prev = next(iterable)
    for element in iterable:
        yield element - prev
        prev = element

All these solution work in constant space if you don’t need to store all the results and support infinite iterables.

Here are some micro-benchmarks of the solutions:

In [12]: L = range(10**6)

In [13]: from collections import deque
In [15]: %timeit deque(differences_tee(L), maxlen=0)
10 loops, best of 3: 122 ms per loop

In [16]: %timeit deque(differences_islice(L), maxlen=0)
10 loops, best of 3: 127 ms per loop

In [17]: %timeit deque(differences_no_it(L), maxlen=0)
10 loops, best of 3: 89.9 ms per loop

And the other proposed solutions:

In [18]: %timeit [x[1] - x[0] for x in zip(L[1:], L)]
10 loops, best of 3: 163 ms per loop

In [19]: %timeit [L[i+1]-L[i] for i in range(len(L)-1)]
1 loops, best of 3: 395 ms per loop

In [20]: import numpy as np

In [21]: %timeit np.diff(L)
1 loops, best of 3: 479 ms per loop

In [35]: %%timeit
    ...: res = []
    ...: for i in range(len(L) - 1):
    ...:     res.append(L[i+1] - L[i])
    ...: 
1 loops, best of 3: 234 ms per loop

Note that:

zip(L[1:], L) is equivalent to zip(L[1:], L[:-1]) since zip already terminates on the shortest input, however it avoids a whole copy of L.
Accessing the single elements by index is very slow because every index access is a method call in python
numpy.diff is slow because it has to first convert the list to a ndarray. Obviously if you start with an ndarray it will be much faster:
```
In [22]: arr = np.array(L)

In [23]: %timeit np.diff(arr)
100 loops, best of 3: 3.02 ms per loop
```

Question 19

Using the := walrus operator available in Python 3.8+:

>>> t = [1, 3, 6]
>>> prev = t[0]; [-prev + (prev := x) for x in t[1:]]
[2, 3]

Question 20

I would suggest using

v = np.diff(t)

this is simple and easy to read.

But if you want v to have the same length as t then

v = np.diff([t[0]] + t) # for python 3.x

or

v = np.diff(t + [t[-1]])

FYI: this will only work for lists.

for numpy arrays

v = np.diff(np.append(t[0], t))

Question 21

A functional approach:

>>> import operator
>>> a = [1,3,5,7,11,13,17,21]
>>> map(operator.sub, a[1:], a[:-1])
[2, 2, 2, 4, 2, 4, 4]

Using generator:

>>> import operator, itertools
>>> g1,g2 = itertools.tee((x*x for x in xrange(5)),2)
>>> list(itertools.imap(operator.sub, itertools.islice(g1,1,None), g2))
[1, 3, 5, 7]

Using indices:

>>> [a[i+1]-a[i] for i in xrange(len(a)-1)]
[2, 2, 2, 4, 2, 4, 4]

Question 22

Ok. I think I found the proper solution:

v = [x[1]-x[0] for x in zip(t[1:],t[:-1])]

Question 23

Solution with periodic boundaries

Sometimes with numerical integration you will want to difference a list with periodic boundary conditions (so the first element calculates the difference to the last. In this case the numpy.roll function is helpful:

v-np.roll(v,1)

Solutions with zero prepended

Another numpy solution (just for completeness) is to use

numpy.ediff1d(v)

This works as numpy.diff, but only on a vector (it flattens the input array). It offers the ability to prepend or append numbers to the resulting vector. This is useful when handling accumulated fields that is often the case fluxes in meteorological variables (e.g. rain, latent heat etc), as you want a resulting list of the same length as the input variable, with the first entry untouched.

Then you would write

np.ediff1d(v,to_begin=v[0])

Of course, you can also do this with the np.diff command, in this case though you need to prepend zero to the series with the prepend keyword:

np.diff(v,prepend=0.0)

All the above solutions return a vector that is the same length as the input.

Question 24

My way

>>>v = [1,2,3,4,5]
>>>[v[i] - v[i-1] for i, value in enumerate(v[1:], 1)]
[1, 1, 1, 1]

Question 25

What is the best way to split a string like "HELLO there HOW are YOU" by upper case words (in Python)?

So I’d end up with an array like such: results = ['HELLO there', 'HOW are', 'YOU']

EDIT:

I have tried:

p = re.compile("\b[A-Z]{2,}\b")
print p.split(page_text)

It doesn’t seem to work, though.

Question 26

I suggest

l = re.compile("(?<!^)\s+(?=[A-Z])(?!.\s)").split(s)

Check this demo.

Question 27

You could use a lookahead:

re.split(r'[ ](?=[A-Z]+\b)', input)

This will split at every space that is followed by a string of upper-case letters which end in a word-boundary.

Note that the square brackets are only for readability and could as well be omitted.

If it is enough that the first letter of a word is upper case (so if you would want to split in front of Hello as well) it gets even easier:

re.split(r'[ ](?=[A-Z])', input)

Now this splits at every space followed by any upper-case letter.

Question 28

Your question contains the string literal "\b[A-Z]{2,}\b", but that \b will mean backspace, because there is no r-modifier.

Try: r"\b[A-Z]{2,}\b".

Question 29

I am trying to create a nice column list in python for use with commandline admin tools which I create.

Basicly, I want a list like:

[['a', 'b', 'c'], ['aaaaaaaaaa', 'b', 'c'], ['a', 'bbbbbbbbbb', 'c']]

To turn into:

a            b            c
aaaaaaaaaa   b            c
a            bbbbbbbbbb   c

Using plain tabs wont do the trick here because I don’t know the longest data in each row.

This is the same behavior as ‘column -t’ in Linux..

$ echo -e "a b c\naaaaaaaaaa b c\na bbbbbbbbbb c"
a b c
aaaaaaaaaa b c
a bbbbbbbbbb c

$ echo -e "a b c\naaaaaaaaaa b c\na bbbbbbbbbb c" | column -t
a           b           c
aaaaaaaaaa  b           c
a           bbbbbbbbbb  c

I have looked around for various python libraries to do this but can’t find anything useful.

Question 30

data = [['a', 'b', 'c'], ['aaaaaaaaaa', 'b', 'c'], ['a', 'bbbbbbbbbb', 'c']]

col_width = max(len(word) for row in data for word in row) + 2  # padding
for row in data:
    print "".join(word.ljust(col_width) for word in row)

a            b            c            
aaaaaaaaaa   b            c            
a            bbbbbbbbbb   c

What this does is calculate the longest data entry to determine the column width, then use .ljust() to add the necessary padding when printing out each column.

Question 31

Since Python 2.6+, you can use a format string in the following way to set the columns to a minimum of 20 characters and align text to right.

table_data = [
    ['a', 'b', 'c'],
    ['aaaaaaaaaa', 'b', 'c'], 
    ['a', 'bbbbbbbbbb', 'c']
]
for row in table_data:
    print("{: >20} {: >20} {: >20}".format(*row))

Output:

               a                    b                    c
      aaaaaaaaaa                    b                    c
               a           bbbbbbbbbb                    c

Question 32

I came here with the same requirements but @lvc and @Preet’s answers seems more inline with what column -t produces in that columns have different widths:

>>> rows =  [   ['a',           'b',            'c',    'd']
...         ,   ['aaaaaaaaaa',  'b',            'c',    'd']
...         ,   ['a',           'bbbbbbbbbb',   'c',    'd']
...         ]
...

>>> widths = [max(map(len, col)) for col in zip(*rows)]
>>> for row in rows:
...     print "  ".join((val.ljust(width) for val, width in zip(row, widths)))
...
a           b           c  d
aaaaaaaaaa  b           c  d
a           bbbbbbbbbb  c  d

Question 33

This is a little late to the party, and a shameless plug for a package I wrote, but you can also check out the Columnar package.

It takes a list of lists of input and a list of headers and outputs a table-formatted string. This snippet creates a docker-esque table:

from columnar import columnar

headers = ['name', 'id', 'host', 'notes']

data = [
    ['busybox', 'c3c37d5d-38d2-409f-8d02-600fd9d51239', 'linuxnode-1-292735', 'Test server.'],
    ['alpine-python', '6bb77855-0fda-45a9-b553-e19e1a795f1e', 'linuxnode-2-249253', 'The one that runs python.'],
    ['redis', 'afb648ba-ac97-4fb2-8953-9a5b5f39663e', 'linuxnode-3-3416918', 'For queues and stuff.'],
    ['app-server', 'b866cd0f-bf80-40c7-84e3-c40891ec68f9', 'linuxnode-4-295918', 'A popular destination.'],
    ['nginx', '76fea0f0-aa53-4911-b7e4-fae28c2e469b', 'linuxnode-5-292735', 'Traffic Cop'],
]

table = columnar(data, headers, no_borders=True)
print(table)

Or you can get a little fancier with colors and borders.

To read more about the column-sizing algorithm and see the rest of the API you can check out the link above or see the Columnar GitHub Repo

Question 34

You have to do this with 2 passes:

get the maximum width of each column.
formatting the columns using our knowledge of max width from the first pass using str.ljust() and str.rjust()

Question 35

Transposing the columns like that is a job for zip:

>>> a = [['a', 'b', 'c'], ['aaaaaaaaaa', 'b', 'c'], ['a', 'bbbbbbbbbb', 'c']]
>>> list(zip(*a))
[('a', 'aaaaaaaaaa', 'a'), ('b', 'b', 'bbbbbbbbbb'), ('c', 'c', 'c')]

To find the required length of each column, you can use max:

>>> trans_a = zip(*a)
>>> [max(len(c) for c in b) for b in trans_a]
[10, 10, 1]

Which you can use, with suitable padding, to construct strings to pass to print:

>>> col_lenghts = [max(len(c) for c in b) for b in trans_a]
>>> padding = ' ' # You might want more
>>> padding.join(s.ljust(l) for s,l in zip(a[0], col_lenghts))
'a          b          c'

Question 36

To get fancier tables like

---------------------------------------------------
| First Name | Last Name        | Age | Position  |
---------------------------------------------------
| John       | Smith            | 24  | Software  |
|            |                  |     | Engineer  |
---------------------------------------------------
| Mary       | Brohowski        | 23  | Sales     |
|            |                  |     | Manager   |
---------------------------------------------------
| Aristidis  | Papageorgopoulos | 28  | Senior    |
|            |                  |     | Reseacher |
---------------------------------------------------

you can use this Python recipe:

'''
From http://code.activestate.com/recipes/267662-table-indentation/
PSF License
'''
import cStringIO,operator

def indent(rows, hasHeader=False, headerChar='-', delim=' | ', justify='left',
           separateRows=False, prefix='', postfix='', wrapfunc=lambda x:x):
    """Indents a table by column.
       - rows: A sequence of sequences of items, one sequence per row.
       - hasHeader: True if the first row consists of the columns' names.
       - headerChar: Character to be used for the row separator line
         (if hasHeader==True or separateRows==True).
       - delim: The column delimiter.
       - justify: Determines how are data justified in their column. 
         Valid values are 'left','right' and 'center'.
       - separateRows: True if rows are to be separated by a line
         of 'headerChar's.
       - prefix: A string prepended to each printed row.
       - postfix: A string appended to each printed row.
       - wrapfunc: A function f(text) for wrapping text; each element in
         the table is first wrapped by this function."""
    # closure for breaking logical rows to physical, using wrapfunc
    def rowWrapper(row):
        newRows = [wrapfunc(item).split('\n') for item in row]
        return [[substr or '' for substr in item] for item in map(None,*newRows)]
    # break each logical row into one or more physical ones
    logicalRows = [rowWrapper(row) for row in rows]
    # columns of physical rows
    columns = map(None,*reduce(operator.add,logicalRows))
    # get the maximum of each column by the string length of its items
    maxWidths = [max([len(str(item)) for item in column]) for column in columns]
    rowSeparator = headerChar * (len(prefix) + len(postfix) + sum(maxWidths) + \
                                 len(delim)*(len(maxWidths)-1))
    # select the appropriate justify method
    justify = {'center':str.center, 'right':str.rjust, 'left':str.ljust}[justify.lower()]
    output=cStringIO.StringIO()
    if separateRows: print >> output, rowSeparator
    for physicalRows in logicalRows:
        for row in physicalRows:
            print >> output, \
                prefix \
                + delim.join([justify(str(item),width) for (item,width) in zip(row,maxWidths)]) \
                + postfix
        if separateRows or hasHeader: print >> output, rowSeparator; hasHeader=False
    return output.getvalue()

# written by Mike Brown
# http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/148061
def wrap_onspace(text, width):
    """
    A word-wrap function that preserves existing line breaks
    and most spaces in the text. Expects that existing line
    breaks are posix newlines (\n).
    """
    return reduce(lambda line, word, width=width: '%s%s%s' %
                  (line,
                   ' \n'[(len(line[line.rfind('\n')+1:])
                         + len(word.split('\n',1)[0]
                              ) >= width)],
                   word),
                  text.split(' ')
                 )

import re
def wrap_onspace_strict(text, width):
    """Similar to wrap_onspace, but enforces the width constraint:
       words longer than width are split."""
    wordRegex = re.compile(r'\S{'+str(width)+r',}')
    return wrap_onspace(wordRegex.sub(lambda m: wrap_always(m.group(),width),text),width)

import math
def wrap_always(text, width):
    """A simple word-wrap function that wraps text on exactly width characters.
       It doesn't split the text in words."""
    return '\n'.join([ text[width*i:width*(i+1)] \
                       for i in xrange(int(math.ceil(1.*len(text)/width))) ])

if __name__ == '__main__':
    labels = ('First Name', 'Last Name', 'Age', 'Position')
    data = \
    '''John,Smith,24,Software Engineer
       Mary,Brohowski,23,Sales Manager
       Aristidis,Papageorgopoulos,28,Senior Reseacher'''
    rows = [row.strip().split(',')  for row in data.splitlines()]

    print 'Without wrapping function\n'
    print indent([labels]+rows, hasHeader=True)
    # test indent with different wrapping functions
    width = 10
    for wrapper in (wrap_always,wrap_onspace,wrap_onspace_strict):
        print 'Wrapping function: %s(x,width=%d)\n' % (wrapper.__name__,width)
        print indent([labels]+rows, hasHeader=True, separateRows=True,
                     prefix='| ', postfix=' |',
                     wrapfunc=lambda x: wrapper(x,width))

    # output:
    #
    #Without wrapping function
    #
    #First Name | Last Name        | Age | Position         
    #-------------------------------------------------------
    #John       | Smith            | 24  | Software Engineer
    #Mary       | Brohowski        | 23  | Sales Manager    
    #Aristidis  | Papageorgopoulos | 28  | Senior Reseacher 
    #
    #Wrapping function: wrap_always(x,width=10)
    #
    #----------------------------------------------
    #| First Name | Last Name  | Age | Position   |
    #----------------------------------------------
    #| John       | Smith      | 24  | Software E |
    #|            |            |     | ngineer    |
    #----------------------------------------------
    #| Mary       | Brohowski  | 23  | Sales Mana |
    #|            |            |     | ger        |
    #----------------------------------------------
    #| Aristidis  | Papageorgo | 28  | Senior Res |
    #|            | poulos     |     | eacher     |
    #----------------------------------------------
    #
    #Wrapping function: wrap_onspace(x,width=10)
    #
    #---------------------------------------------------
    #| First Name | Last Name        | Age | Position  |
    #---------------------------------------------------
    #| John       | Smith            | 24  | Software  |
    #|            |                  |     | Engineer  |
    #---------------------------------------------------
    #| Mary       | Brohowski        | 23  | Sales     |
    #|            |                  |     | Manager   |
    #---------------------------------------------------
    #| Aristidis  | Papageorgopoulos | 28  | Senior    |
    #|            |                  |     | Reseacher |
    #---------------------------------------------------
    #
    #Wrapping function: wrap_onspace_strict(x,width=10)
    #
    #---------------------------------------------
    #| First Name | Last Name  | Age | Position  |
    #---------------------------------------------
    #| John       | Smith      | 24  | Software  |
    #|            |            |     | Engineer  |
    #---------------------------------------------
    #| Mary       | Brohowski  | 23  | Sales     |
    #|            |            |     | Manager   |
    #---------------------------------------------
    #| Aristidis  | Papageorgo | 28  | Senior    |
    #|            | poulos     |     | Reseacher |
    #---------------------------------------------

The Python recipe page contains a few improvements on it.

Question 37

pandas based solution with creating dataframe:

import pandas as pd
l = [['a', 'b', 'c'], ['aaaaaaaaaa', 'b', 'c'], ['a', 'bbbbbbbbbb', 'c']]
df = pd.DataFrame(l)

print(df)
            0           1  2
0           a           b  c
1  aaaaaaaaaa           b  c
2           a  bbbbbbbbbb  c

To remove index and header values to create output what you want you could use to_string method:

result = df.to_string(index=False, header=False)

print(result)
          a           b  c
 aaaaaaaaaa           b  c
          a  bbbbbbbbbb  c

Question 38

Scolp is a new library that lets you pretty print streaming columnar data easily while auto-adjusting column width.

(Disclaimer: I am the author)

Question 39

This sets independent, best-fit column widths based on the max-metric used in other answers.

data = [['a', 'b', 'c'], ['aaaaaaaaaa', 'b', 'c'], ['a', 'bbbbbbbbbb', 'c']]
padding = 2
col_widths = [max(len(w) for w in [r[cn] for r in data]) + padding for cn in range(len(data[0]))]
format_string = "{{:{}}}{{:{}}}{{:{}}}".format(*col_widths)
for row in data:
    print(format_string.format(*row))

Question 40

For lazy people

that are using Python 3.* and Pandas/Geopandas; universal simple in-class approach (for ‘normal’ script just remove self):

Function colorize:

    def colorize(self,s,color):
        s = color+str(s)+"\033[0m"
        return s

Header:

print('{0:<23} {1:>24} {2:>26} {3:>26} {4:>11} {5:>11}'.format('Road name','Classification','Function','Form of road','Length','Distance') )

and then data from Pandas/Geopandas dataframe:

            for index, row in clipped.iterrows():
                rdName      = self.colorize(row['name1'],"\033[32m")
                rdClass     = self.colorize(row['roadClassification'],"\033[93m")
                rdFunction  = self.colorize(row['roadFunction'],"\033[33m")
                rdForm      = self.colorize(row['formOfWay'],"\033[94m")
                rdLength    = self.colorize(row['length'],"\033[97m")
                rdDistance  = self.colorize(row['distance'],"\033[96m")
                print('{0:<30} {1:>35} {2:>35} {3:>35} {4:>20} {5:>20}'.format(rdName,rdClass,rdFunction,rdForm,rdLength,rdDistance) )

Meaning of {0:<30} {1:>35} {2:>35} {3:>35} {4:>20} {5:>20}:

0, 1, 2, 3, 4, 5 -> columns, there are 6 in total in this case

30, 35, 20 -> width of column (note that you’ll have to add length of \033[96m – this for Python is a string as well), just experiment :)

>, < -> justify: right, left (there is = for filling with zeros as well)

If you want to distinct e.g. max value, you’ll have to switch to special Pandas style function, but suppose that’s far enough to present data on terminal window.

Result:

Question 41

A slight variation on a previous answer (I don’t have enough rep to comment on it). The format library lets you specify the width and alignment of an element but not where it starts, ie, you can say “be 20 columns wide” but not “start in column 20”. Which leads to this issue:

table_data = [
    ['a', 'b', 'c'],
    ['aaaaaaaaaa', 'b', 'c'], 
    ['a', 'bbbbbbbbbb', 'c']
]

print("first row: {: >20} {: >20} {: >20}".format(*table_data[0]))
print("second row: {: >20} {: >20} {: >20}".format(*table_data[1]))
print("third row: {: >20} {: >20} {: >20}".format(*table_data[2]))

Output

first row:                    a                    b                    c
second row:           aaaaaaaaaa                    b                    c
third row:                    a           bbbbbbbbbb                    c

The answer of course is to format the literal strings as well, which combines slightly weirdly with the format:

table_data = [
    ['a', 'b', 'c'],
    ['aaaaaaaaaa', 'b', 'c'], 
    ['a', 'bbbbbbbbbb', 'c']
]

print(f"{'first row:': <20} {table_data[0][0]: >20} {table_data[0][1]: >20} {table_data[0][2]: >20}")
print("{: <20} {: >20} {: >20} {: >20}".format(*['second row:', *table_data[1]]))
print("{: <20} {: >20} {: >20} {: >20}".format(*['third row:', *table_data[1]]))

Output

first row:                              a                    b                    c
second row:                    aaaaaaaaaa                    b                    c
third row:                     aaaaaaaaaa                    b                    c

Question 42

I found this answer super-helpful and elegant, originally from here:

matrix = [["A", "B"], ["C", "D"]]

print('\n'.join(['\t'.join([str(cell) for cell in row]) for row in matrix]))

Output

A   B
C   D

Question 43

Here is a variation of the Shawn Chin’s answer. The width is fixed per column, not over all columns. There is also a border below the first row and between the columns. (icontract library is used to enforce the contracts.)

@icontract.pre(
    lambda table: not table or all(len(row) == len(table[0]) for row in table))
@icontract.post(lambda table, result: result == "" if not table else True)
@icontract.post(lambda result: not result.endswith("\n"))
def format_table(table: List[List[str]]) -> str:
    """
    Format the table as equal-spaced columns.

    :param table: rows of cells
    :return: table as string
    """
    cols = len(table[0])

    col_widths = [max(len(row[i]) for row in table) for i in range(cols)]

    lines = []  # type: List[str]
    for i, row in enumerate(table):
        parts = []  # type: List[str]

        for cell, width in zip(row, col_widths):
            parts.append(cell.ljust(width))

        line = " | ".join(parts)
        lines.append(line)

        if i == 0:
            border = []  # type: List[str]

            for width in col_widths:
                border.append("-" * width)

            lines.append("-+-".join(border))

    result = "\n".join(lines)

    return result

Here is an example:

>>> table = [['column 0', 'another column 1'], ['00', '01'], ['10', '11']]
>>> result = packagery._format_table(table=table)
>>> print(result)
column 0 | another column 1
---------+-----------------
00       | 01              
10       | 11

Question 44

updated @Franck Dernoncourt fancy recipe to be python 3 and PEP8 compliant

import io
import math
import operator
import re
import functools

from itertools import zip_longest


def indent(
    rows,
    has_header=False,
    header_char="-",
    delim=" | ",
    justify="left",
    separate_rows=False,
    prefix="",
    postfix="",
    wrapfunc=lambda x: x,
):
    """Indents a table by column.
       - rows: A sequence of sequences of items, one sequence per row.
       - hasHeader: True if the first row consists of the columns' names.
       - headerChar: Character to be used for the row separator line
         (if hasHeader==True or separateRows==True).
       - delim: The column delimiter.
       - justify: Determines how are data justified in their column.
         Valid values are 'left','right' and 'center'.
       - separateRows: True if rows are to be separated by a line
         of 'headerChar's.
       - prefix: A string prepended to each printed row.
       - postfix: A string appended to each printed row.
       - wrapfunc: A function f(text) for wrapping text; each element in
         the table is first wrapped by this function."""

    # closure for breaking logical rows to physical, using wrapfunc
    def row_wrapper(row):
        new_rows = [wrapfunc(item).split("\n") for item in row]
        return [[substr or "" for substr in item] for item in zip_longest(*new_rows)]

    # break each logical row into one or more physical ones
    logical_rows = [row_wrapper(row) for row in rows]
    # columns of physical rows
    columns = zip_longest(*functools.reduce(operator.add, logical_rows))
    # get the maximum of each column by the string length of its items
    max_widths = [max([len(str(item)) for item in column]) for column in columns]
    row_separator = header_char * (
        len(prefix) + len(postfix) + sum(max_widths) + len(delim) * (len(max_widths) - 1)
    )
    # select the appropriate justify method
    justify = {"center": str.center, "right": str.rjust, "left": str.ljust}[
        justify.lower()
    ]
    output = io.StringIO()
    if separate_rows:
        print(output, row_separator)
    for physicalRows in logical_rows:
        for row in physicalRows:
            print( output, prefix + delim.join(
                [justify(str(item), width) for (item, width) in zip(row, max_widths)]
            ) + postfix)
        if separate_rows or has_header:
            print(output, row_separator)
            has_header = False
    return output.getvalue()


# written by Mike Brown
# http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/148061
def wrap_onspace(text, width):
    """
    A word-wrap function that preserves existing line breaks
    and most spaces in the text. Expects that existing line
    breaks are posix newlines (\n).
    """
    return functools.reduce(
        lambda line, word, i_width=width: "%s%s%s"
        % (
            line,
            " \n"[
                (
                    len(line[line.rfind("\n") + 1 :]) + len(word.split("\n", 1)[0])
                    >= i_width
                )
            ],
            word,
        ),
        text.split(" "),
    )


def wrap_onspace_strict(text, i_width):
    """Similar to wrap_onspace, but enforces the width constraint:
       words longer than width are split."""
    word_regex = re.compile(r"\S{" + str(i_width) + r",}")
    return wrap_onspace(
        word_regex.sub(lambda m: wrap_always(m.group(), i_width), text), i_width
    )


def wrap_always(text, width):
    """A simple word-wrap function that wraps text on exactly width characters.
       It doesn't split the text in words."""
    return "\n".join(
        [
            text[width * i : width * (i + 1)]
            for i in range(int(math.ceil(1.0 * len(text) / width)))
        ]
    )


if __name__ == "__main__":
    labels = ("First Name", "Last Name", "Age", "Position")
    data = """John,Smith,24,Software Engineer
           Mary,Brohowski,23,Sales Manager
           Aristidis,Papageorgopoulos,28,Senior Reseacher"""
    rows = [row.strip().split(",") for row in data.splitlines()]

    print("Without wrapping function\n")
    print(indent([labels] + rows, has_header=True))

    # test indent with different wrapping functions
    width = 10
    for wrapper in (wrap_always, wrap_onspace, wrap_onspace_strict):
        print("Wrapping function: %s(x,width=%d)\n" % (wrapper.__name__, width))

        print(
            indent(
                [labels] + rows,
                has_header=True,
                separate_rows=True,
                prefix="| ",
                postfix=" |",
                wrapfunc=lambda x: wrapper(x, width),
            )
        )

    # output:
    #
    # Without wrapping function
    #
    # First Name | Last Name        | Age | Position
    # -------------------------------------------------------
    # John       | Smith            | 24  | Software Engineer
    # Mary       | Brohowski        | 23  | Sales Manager
    # Aristidis  | Papageorgopoulos | 28  | Senior Reseacher
    #
    # Wrapping function: wrap_always(x,width=10)
    #
    # ----------------------------------------------
    # | First Name | Last Name  | Age | Position   |
    # ----------------------------------------------
    # | John       | Smith      | 24  | Software E |
    # |            |            |     | ngineer    |
    # ----------------------------------------------
    # | Mary       | Brohowski  | 23  | Sales Mana |
    # |            |            |     | ger        |
    # ----------------------------------------------
    # | Aristidis  | Papageorgo | 28  | Senior Res |
    # |            | poulos     |     | eacher     |
    # ----------------------------------------------
    #
    # Wrapping function: wrap_onspace(x,width=10)
    #
    # ---------------------------------------------------
    # | First Name | Last Name        | Age | Position  |
    # ---------------------------------------------------
    # | John       | Smith            | 24  | Software  |
    # |            |                  |     | Engineer  |
    # ---------------------------------------------------
    # | Mary       | Brohowski        | 23  | Sales     |
    # |            |                  |     | Manager   |
    # ---------------------------------------------------
    # | Aristidis  | Papageorgopoulos | 28  | Senior    |
    # |            |                  |     | Reseacher |
    # ---------------------------------------------------
    #
    # Wrapping function: wrap_onspace_strict(x,width=10)
    #
    # ---------------------------------------------
    # | First Name | Last Name  | Age | Position  |
    # ---------------------------------------------
    # | John       | Smith      | 24  | Software  |
    # |            |            |     | Engineer  |
    # ---------------------------------------------
    # | Mary       | Brohowski  | 23  | Sales     |
    # |            |            |     | Manager   |
    # ---------------------------------------------
    # | Aristidis  | Papageorgo | 28  | Senior    |
    # |            | poulos     |     | Reseacher |
    # ---------------------------------------------

Question 45

I realize this question is old but I didn’t understand Antak’s answer and didn’t want to use a library so I rolled my own solution.

Solution assumes records is a 2D array, records are all the same length, and that fields are all strings.

def stringifyRecords(records):
    column_widths = [0] * len(records[0])
    for record in records:
        for i, field in enumerate(record):
            width = len(field)
            if width > column_widths[i]: column_widths[i] = width

    s = ""
    for record in records:
        for column_width, field in zip(column_widths, record):
            s += field.ljust(column_width+1)
        s += "\n"

    return s

Question 46

I am asking Python to print the minimum number from a column of CSV data, but the top row is the column number, and I don’t want Python to take the top row into account. How can I make sure Python ignores the first line?

This is the code so far:

import csv

with open('all16.csv', 'rb') as inf:
    incsv = csv.reader(inf)
    column = 1                
    datatype = float          
    data = (datatype(column) for row in incsv)   
    least_value = min(data)

print least_value

Could you also explain what you are doing, not just give the code? I am very very new to Python and would like to make sure I understand everything.

Question 47

You could use an instance of the csv module’s Sniffer class to deduce the format of a CSV file and detect whether a header row is present along with the built-in next() function to skip over the first row only when necessary:

import csv

with open('all16.csv', 'r', newline='') as file:
    has_header = csv.Sniffer().has_header(file.read(1024))
    file.seek(0)  # Rewind.
    reader = csv.reader(file)
    if has_header:
        next(reader)  # Skip header row.
    column = 1
    datatype = float
    data = (datatype(row[column]) for row in reader)
    least_value = min(data)

print(least_value)

Since datatype and column are hardcoded in your example, it would be slightly faster to process the row like this:

    data = (float(row[1]) for row in reader)

Note: the code above is for Python 3.x. For Python 2.x use the following line to open the file instead of what is shown:

with open('all16.csv', 'rb') as file:

Question 48

To skip the first line just call:

next(inf)

Files in Python are iterators over lines.

Question 49

In a similar use case I had to skip annoying lines before the line with my actual column names. This solution worked nicely. Read the file first, then pass the list to csv.DictReader.

with open('all16.csv') as tmp:
    # Skip first line (if any)
    next(tmp, None)

    # {line_num: row}
    data = dict(enumerate(csv.DictReader(tmp)))

Question 50

Borrowed from python cookbook,
A more concise template code might look like this:

import csv
with open('stocks.csv') as f:
    f_csv = csv.reader(f) 
    headers = next(f_csv) 
    for row in f_csv:
        # Process row ...

Question 51

You would normally use next(incsv) which advances the iterator one row, so you skip the header. The other (say you wanted to skip 30 rows) would be:

from itertools import islice
for row in islice(incsv, 30, None):
    # process

Question 52

use csv.DictReader instead of csv.Reader. If the fieldnames parameter is omitted, the values in the first row of the csvfile will be used as field names. you would then be able to access field values using row[“1”] etc

Question 53

The new ‘pandas’ package might be more relevant than ‘csv’. The code below will read a CSV file, by default interpreting the first line as the column header and find the minimum across columns.

import pandas as pd

data = pd.read_csv('all16.csv')
data.min()

Question 54

Well, my mini wrapper library would do the job as well.

>>> import pyexcel as pe
>>> data = pe.load('all16.csv', name_columns_by_row=0)
>>> min(data.column[1])

Meanwhile, if you know what header column index one is, for example “Column 1”, you can do this instead:

>>> min(data.column["Column 1"])

Question 55

For me the easiest way to go is to use range.

import csv

with open('files/filename.csv') as I:
    reader = csv.reader(I)
    fulllist = list(reader)

# Starting with data skipping header
for item in range(1, len(fulllist)): 
    # Print each row using "item" as the index value
    print (fulllist[item])

Question 56

Because this is related to something I was doing, I’ll share here.

What if we’re not sure if there’s a header and you also don’t feel like importing sniffer and other things?

If your task is basic, such as printing or appending to a list or array, you could just use an if statement:

# Let's say there's 4 columns
with open('file.csv') as csvfile:
     csvreader = csv.reader(csvfile)
# read first line
     first_line = next(csvreader)
# My headers were just text. You can use any suitable conditional here
     if len(first_line) == 4:
          array.append(first_line)
# Now we'll just iterate over everything else as usual:
     for row in csvreader:
          array.append(row)

Question 57

The documentation for the Python 3 CSV module provides this example:

with open('example.csv', newline='') as csvfile:
    dialect = csv.Sniffer().sniff(csvfile.read(1024))
    csvfile.seek(0)
    reader = csv.reader(csvfile, dialect)
    # ... process CSV file contents here ...

The Sniffer will try to auto-detect many things about the CSV file. You need to explicitly call its has_header() method to determine whether the file has a header line. If it does, then skip the first row when iterating the CSV rows. You can do it like this:

if sniffer.has_header():
    for header_row in reader:
        break
for data_row in reader:
    # do something with the row

Question 58

I would use tail to get rid of the unwanted first line:

tail -n +2 $INFIL | whatever_script.py

Question 59

just add [1:]

example below:

data = pd.read_csv("/Users/xyz/Desktop/xyxData/xyz.csv", sep=',', header=None)**[1:]**

that works for me in iPython

Question 60

Python 3.X

Handles UTF8 BOM + HEADER

It was quite frustrating that the csv module could not easily get the header, there is also a bug with the UTF-8 BOM (first char in file). This works for me using only the csv module:

import csv

def read_csv(self, csv_path, delimiter):
    with open(csv_path, newline='', encoding='utf-8') as f:
        # https://bugs.python.org/issue7185
        # Remove UTF8 BOM.
        txt = f.read()[1:]

    # Remove header line.
    header = txt.splitlines()[:1]
    lines = txt.splitlines()[1:]

    # Convert to list.
    csv_rows = list(csv.reader(lines, delimiter=delimiter))

    for row in csv_rows:
        value = row[INDEX_HERE]

Question 61

I would convert csvreader to list, then pop the first element

import csv        

with open(fileName, 'r') as csvfile:
        csvreader = csv.reader(csvfile)
        data = list(csvreader)               # Convert to list
        data.pop(0)                          # Removes the first row

        for row in data:
            print(row)

Question 62

Python 2.x

csvreader.next()

Return the next row of the reader’s iterable object as a list, parsed according to the current dialect.

csv_data = csv.reader(open('sample.csv'))
csv_data.next() # skip first row
for row in csv_data:
    print(row) # should print second row

Python 3.x

csvreader.__next__()

Return the next row of the reader’s iterable object as a list (if the object was returned from reader()) or a dict (if it is a DictReader instance), parsed according to the current dialect. Usually you should call this as next(reader).

csv_data = csv.reader(open('sample.csv'))
csv_data.__next__() # skip first row
for row in csv_data:
    print(row) # should print second row

Question 63

In Python, when given the URL for a text file, what is the simplest way to access the contents off the text file and print the contents of the file out locally line-by-line without saving a local copy of the text file?

TargetURL=http://www.myhost.com/SomeFile.txt
#read the file
#print first line
#print second line
#etc

Question 64

Edit 09/2016: In Python 3 and up use urllib.request instead of urllib2

Actually the simplest way is:

import urllib2  # the lib that handles the url stuff

data = urllib2.urlopen(target_url) # it's a file like object and works just like a file
for line in data: # files are iterable
    print line

You don’t even need “readlines”, as Will suggested. You could even shorten it to: ^*

import urllib2

for line in urllib2.urlopen(target_url):
    print line

But remember in Python, readability matters.

However, this is the simplest way but not the safe way because most of the time with network programming, you don’t know if the amount of data to expect will be respected. So you’d generally better read a fixed and reasonable amount of data, something you know to be enough for the data you expect but will prevent your script from been flooded:

import urllib2

data = urllib2.urlopen("http://www.google.com").read(20000) # read only 20 000 chars
data = data.split("\n") # then split it into lines

for line in data:
    print line

^{* Second example in Python 3:}

import urllib.request  # the lib that handles the url stuff

for line in urllib.request.urlopen(target_url):
    print(line.decode('utf-8')) #utf-8 or iso8859-1 or whatever the page encoding scheme is

问题：如何检查以下所有项目是否都在列表中？

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

问题：更改源代码时自动加载gunicorn

回答 0

回答 1

回答 2

回答 3

问题：查找列表元素之间的差异

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

回答 7

回答 8

回答 9

问题：基于正则表达式的Python拆分字符串

回答 0

回答 1

回答 2

问题：在python中创建漂亮的列输出

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

回答 7

回答 8

回答 9

回答 10

对于懒惰的人

结果：

For lazy people

Result:

回答 11

回答 12

回答 13

回答 14

回答 15

问题：处理CSV数据时如何忽略第一行数据？

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

回答 7

回答 8

回答 9

回答 10

回答 11

回答 12

回答 13

回答 14

回答 15

问题：给定一个文本文件的URL，最简单的读取文本文件内容的方法是什么？

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

回答 7

回答 8

问题：将字符串插入列表中而不会拆分为字符

回答 0

回答 1

回答 2

`eval` 是邪恶的

使用以下方法评估算术表达式 `ast`

`eval` is evil

Evaluate arithmetic expression using `ast`