Python 实用宝典

Question 1

I have a list in python and I want to convert it to an array to be able to use ravel() function.

Question 2

Use numpy.asarray:

import numpy as np
myarray = np.asarray(mylist)

Question 3

create an int array and a list

from array import array
listA = list(range(0,50))
for item in listA:
    print(item)
arrayA = array("i", listA)
for item in arrayA:
    print(item)

Question 4

I wanted a way to do this without using an extra module. First turn list to string, then append to an array:

dataset_list = ''.join(input_list)
dataset_array = []
for item in dataset_list.split(';'): # comma, or other
    dataset_array.append(item)

Question 5

If all you want is calling ravel on your (nested, I s’pose?) list, you can do that directly, numpy will do the casting for you:

L = [[1,None,3],["The", "quick", object]]
np.ravel(L)
# array([1, None, 3, 'The', 'quick', <class 'object'>], dtype=object)

Also worth mentioning that you needn’t go through numpy at all.

Question 6

Use the following code:

import numpy as np

myArray=np.array([1,2,4])  #func used to convert [1,2,3] list into an array
print(myArray)

Question 7

if variable b has a list then you can simply do the below:

create a new variable “a” as: a=[] then assign the list to “a” as: a=b

now “a” has all the components of list “b” in array.

so you have successfully converted list to array.

Question 8

1. Print a-n: a b c d e f g h i j k l m n

2. Every second in a-n: a c e g i k m

3. Append a-n to index of urls{hello.com/, hej.com/, …, hallo.com/}: hello.com/a hej.com/b … hallo.com/n

Question 9

>>> import string
>>> string.ascii_lowercase[:14]
'abcdefghijklmn'
>>> string.ascii_lowercase[:14:2]
'acegikm'

To do the urls, you could use something like this

[i + j for i, j in zip(list_of_urls, string.ascii_lowercase[:14])]

Question 10

Assuming this is a homework ;-) – no need to summon libraries etc – it probably expect you to use range() with chr/ord, like so:

for i in range(ord('a'), ord('n')+1):
    print chr(i),

For the rest, just play a bit more with the range()

Question 11

Hints:

import string
print string.ascii_lowercase

and

for i in xrange(0, 10, 2):
    print i

and

"hello{0}, world!".format('z')

Question 12

for one in range(97,110):
    print chr(one)

Question 13

Get a list with the desired values

small_letters = map(chr, range(ord('a'), ord('z')+1))
big_letters = map(chr, range(ord('A'), ord('Z')+1))
digits = map(chr, range(ord('0'), ord('9')+1))

or

import string
string.letters
string.uppercase
string.digits

This solution uses the ASCII table. ord gets the ascii value from a character and chr vice versa.

Apply what you know about lists

>>> small_letters = map(chr, range(ord('a'), ord('z')+1))

>>> an = small_letters[0:(ord('n')-ord('a')+1)]
>>> print(" ".join(an))
a b c d e f g h i j k l m n

>>> print(" ".join(small_letters[0::2]))
a c e g i k m o q s u w y

>>> s = small_letters[0:(ord('n')-ord('a')+1):2]
>>> print(" ".join(s))
a c e g i k m

>>> urls = ["hello.com/", "hej.com/", "hallo.com/"]
>>> print([x + y for x, y in zip(urls, an)])
['hello.com/a', 'hej.com/b', 'hallo.com/c']

Question 14

import string
print list(string.ascii_lowercase)
# ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']

Question 15

import string
print list(string.ascii_lowercase)
# ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']

and

for c in list(string.ascii_lowercase)[:5]:
    ...operation with the first 5 characters

Question 16

myList = [chr(chNum) for chNum in list(range(ord('a'),ord('z')+1))]
print(myList)

Output

['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']

Question 17

#1)
print " ".join(map(chr, range(ord('a'),ord('n')+1)))

#2)
print " ".join(map(chr, range(ord('a'),ord('n')+1,2)))

#3)
urls = ["hello.com/", "hej.com/", "hallo.com/"]
an = map(chr, range(ord('a'),ord('n')+1))
print [ x + y for x,y in zip(urls, an)]

Question 18

The answer to this question is simple, just make a list called ABC like so:

ABC = ['abcdefghijklmnopqrstuvwxyz']

And whenever you need to refer to it, just do:

print ABC[0:9] #prints abcdefghij
print ABC       #prints abcdefghijklmnopqrstuvwxyz
for x in range(0,25):
    if x % 2 == 0:
        print ABC[x] #prints acegikmoqsuwy (all odd numbered letters)

Also try this to break ur device :D

##Try this and call it AlphabetSoup.py:

ABC = ['abcdefghijklmnopqrstuvwxyz']


try:
    while True:
        for a in ABC:
            for b in ABC:
                for c in ABC:
                    for d in ABC:
                        for e in ABC:
                            for f in ABC:
                                print a, b, c, d, e, f, '    ',
except KeyboardInterrupt:
    pass

Question 19

Try:

strng = ""
for i in range(97,123):
    strng = strng + chr(i)
print(strng)

Question 20

This is your 2nd question: string.lowercase[ord('a')-97:ord('n')-97:2] because 97==ord('a') — if you want to learn a bit you should figure out the rest yourself ;-)

Question 21

list(string.ascii_lowercase)

['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']

Question 22

I hope this helps:

import string

alphas = list(string.ascii_letters[:26])
for chr in alphas:
 print(chr)

Question 23

About gnibbler’s answer.

Zip -function, full explanation, returns a list of tuples, where the i-th tuple contains the i-th element from each of the argument sequences or iterables. [...] construct is called list comprehension, very cool feature!

Question 24

Another way to do it

  import string
  pass

  aalist = list(string.ascii_lowercase)
  aaurls = ['alpha.com','bravo.com','chrly.com','delta.com',]
  iilen  =  aaurls.__len__()
  pass

  ans01 = "".join( (aalist[0:14]) )
  ans02 = "".join( (aalist[0:14:2]) )
  ans03 = "".join( "{vurl}/{vl}\n".format(vl=vjj[1],vurl=aaurls[vjj[0] % iilen]) for vjj in enumerate(aalist[0:14]) )
  pass

  print(ans01)
  print(ans02)
  print(ans03)
  pass

Result

abcdefghijklmn
acegikm
alpha.com/a
bravo.com/b
chrly.com/c
delta.com/d
alpha.com/e
bravo.com/f
chrly.com/g
delta.com/h
alpha.com/i
bravo.com/j
chrly.com/k
delta.com/l
alpha.com/m
bravo.com/n

How this differs from the other replies

iterate over an arbitrary number of base urls
cycle through the urls and do not stop until we run out of letters
use enumerate in conjunction with list comprehension and str.format

Question 25

I often use python to process directories of data. Recently, I have noticed that the default order of the lists has changed to something almost nonsensical. For example, if I am in a current directory containing the following subdirectories: run01, run02, … run19, run20, and then I generate a list from the following command:

dir = os.listdir(os.getcwd())

then I usually get a list in this order:

dir = ['run01', 'run18', 'run14', 'run13', 'run12', 'run11', 'run08', ... ]

and so on. The order used to be alphanumeric. But this new order has remained with me for a while now.

What is determining the (displayed) order of these lists?

Question 26

I think the order has to do with the way the files are indexed on your FileSystem. If you really want to make it adhere to some order you can always sort the list after getting the files.

Question 27

You can use the builtin sorted function to sort the strings however you want. Based on what you describe,

sorted(os.listdir(whatever_directory))

Alternatively, you can use the .sort method of a list:

lst = os.listdir(whatever_directory)
lst.sort()

I think should do the trick.

Note that the order that os.listdir gets the filenames is probably completely dependent on your filesystem.

Question 28

Per the documentation:

os.listdir(path)

Return a list containing the names of the entries in the directory given by path. The list is in arbitrary order. It does not include the special entries ‘.’ and ‘..’ even if they are present in the directory.

Order cannot be relied upon and is an artifact of the filesystem.

To sort the result, use sorted(os.listdir(path)).

Question 29

Python for whatever reason does not come with a built-in way to have natural sorting (meaning 1, 2, 10 instead of 1, 10, 2), so you have to write it yourself:

import re
def sorted_alphanumeric(data):
    convert = lambda text: int(text) if text.isdigit() else text.lower()
    alphanum_key = lambda key: [ convert(c) for c in re.split('([0-9]+)', key) ] 
    return sorted(data, key=alphanum_key)

You can now use this function to sort a list:

dirlist = sorted_alphanumeric(os.listdir(...))

PROBLEMS: In case you use the above function to sort strings (for example folder names) and want them sorted like Windows Explorer does, it will not work properly in some edge cases.
This sorting function will return incorrect results on Windows, if you have folder names with certain ‘special’ characters in them. For example this function will sort 1, !1, !a, a, whereas Windows Explorer would sort !1, 1, !a, a.

So if you want to sort exactly like Windows Explorer does in Python you have to use the Windows built-in function StrCmpLogicalW via ctypes (this of course won’t work on Unix):

from ctypes import wintypes, windll
from functools import cmp_to_key
def winsort(data):
    _StrCmpLogicalW = windll.Shlwapi.StrCmpLogicalW
    _StrCmpLogicalW.argtypes = [wintypes.LPWSTR, wintypes.LPWSTR]
    _StrCmpLogicalW.restype  = wintypes.INT

    cmp_fnc = lambda psz1, psz2: _StrCmpLogicalW(psz1, psz2)
    return sorted(data, key=cmp_to_key(cmp_fnc))

This function is slightly slower than sorted_alphanumeric().

Bonus: winsort can also sort full paths on Windows.

Alternatively, especially if you use Unix, you can use the natsort library (pip install natsort) to sort by full paths in a correct way (meaning subfolders at the correct position).

You can use it like this to sort full paths:

from natsort import natsorted, ns
dirlist = natsorted(dirlist, alg=ns.PATH | ns.IGNORECASE)

Don’t use it for normal sorting of just folder names (or strings in general), as it’s quite a bit slower than then sorted_alphanumeric() function above.
natsorted library will give you incorrect results if you expect Windows Explorer sorting, so use winsort() for that.

Question 30

I think by default the order is determined with the ASCII value. The solution to this problem is this

dir = sorted(os.listdir(os.getcwd()), key=len)

Question 31

It’s probably just the order that C’s readdir() returns. Try running this C program:

#include <dirent.h>
#include <stdio.h>
int main(void)
{   DIR *dirp;
    struct dirent* de;
    dirp = opendir(".");
    while(de = readdir(dirp)) // Yes, one '='.
        printf("%s\n", de->d_name);
    closedir(dirp);
    return 0;
}

The build line should be something like gcc -o foo foo.c.

P.S. Just ran this and your Python code, and they both gave me sorted output, so I can’t reproduce what you’re seeing.

Question 32

aaa = ['row_163.pkl', 'row_394.pkl', 'row_679.pkl', 'row_202.pkl', 'row_1449.pkl', 'row_247.pkl', 'row_1353.pkl', 'row_749.pkl', 'row_1293.pkl', 'row_1304.pkl', 'row_78.pkl', 'row_532.pkl', 'row_9.pkl', 'row_1435.pkl']                                                                                                                                                                                                                                                                                                 
sorted(aaa, key=lambda x: int(os.path.splitext(x.split('_')[1])[0]))

As In case of mine requirement I have the case like row_163.pkl here os.path.splitext('row_163.pkl') will break it into ('row_163', '.pkl') so need to split it based on ‘_’ also.

but in case of your requirement you can do something like

sorted(aa, key = lambda x: (int(re.sub('\D','',x)),x))

where

aa = ['run01', 'run08', 'run11', 'run12', 'run13', 'run14', 'run18']

and also for directory retrieving you can do sorted(os.listdir(path))

and for the case of like 'run01.txt' or 'run01.csv' you can do like this

sorted(files, key=lambda x : int(os.path.splitext(x)[0]))

Question 33

I found “sort” does not always do what I expected. eg, I have a directory as below, and the “sort” give me a very strange result:

>>> os.listdir(pathon)
['2', '3', '4', '5', '403', '404', '407', '408', '410', '411', '412', '413', '414', '415', '416', '472']
>>> sorted([ f for f in os.listdir(pathon)])
['2', '3', '4', '403', '404', '407', '408', '410', '411', '412', '413', '414', '415', '416', '472', '5']

It seems it compares the first character first, if that is the biggest, it would be the last one.

Question 34

From the documentation:

The list is in arbitrary order, and does not include the special entries ‘.’ and ‘..’ even if they are present in the directory.

This means that the order is probably OS/filesystem dependent, has no particularly meaningful order, and is therefore not guaranteed to be anything in particular. As many answers mentioned: if preferred, the retrieved list can be sorted.

Cheers :)

Question 35

Elliot’s answer solves it perfectly but because it is a comment, it goes unnoticed so with the aim of helping someone, I am reiterating it as a solution.

Use natsort library:

Install the library with the following command for Ubuntu and other Debian versions

Python 2

sudo pip install natsort

Python 3

sudo pip3 install natsort

Details of how to use this library is found here

Question 36

In [6]: os.listdir?

Type:       builtin_function_or_method
String Form:<built-in function listdir>
Docstring:
listdir(path) -> list_of_strings
Return a list containing the names of the entries in the directory.
path: path of directory to list
The list is in **arbitrary order**.  It does not include the special
entries '.' and '..' even if they are present in the directory.

Question 37

The proposed combination of os.listdir and sorted commands generates the same result as ls -l command under Linux. The following example verifies this assumption:

user@user-PC:/tmp/test$ touch 3a 4a 5a b c d1 d2 d3 k l p0 p1 p3 q 410a 409a 408a 407a
user@user-PC:/tmp/test$ ls -l
total 0
-rw-rw-r-- 1 user user 0 Feb  15 10:31 3a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 407a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 408a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 409a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 410a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 4a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 5a
-rw-rw-r-- 1 user user 0 Feb  15 10:31 b
-rw-rw-r-- 1 user user 0 Feb  15 10:31 c
-rw-rw-r-- 1 user user 0 Feb  15 10:31 d1
-rw-rw-r-- 1 user user 0 Feb  15 10:31 d2
-rw-rw-r-- 1 user user 0 Feb  15 10:31 d3
-rw-rw-r-- 1 user user 0 Feb  15 10:31 k
-rw-rw-r-- 1 user user 0 Feb  15 10:31 l
-rw-rw-r-- 1 user user 0 Feb  15 10:31 p0
-rw-rw-r-- 1 user user 0 Feb  15 10:31 p1
-rw-rw-r-- 1 user user 0 Feb  15 10:31 p3
-rw-rw-r-- 1 user user 0 Feb  15 10:31 q

user@user-PC:/tmp/test$ python
Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.listdir( './' )
['d3', 'k', 'p1', 'b', '410a', '5a', 'l', 'p0', '407a', '409a', '408a', 'd2', '4a', 'p3', '3a', 'q', 'c', 'd1']
>>> sorted( os.listdir( './' ) )
['3a', '407a', '408a', '409a', '410a', '4a', '5a', 'b', 'c', 'd1', 'd2', 'd3', 'k', 'l', 'p0', 'p1', 'p3', 'q']
>>> exit()
user@user-PC:/tmp/test$

So, for someone who wants to reproduce the result of the well-known ls -l command in their python code, sorted( os.listdir( DIR ) ) works pretty well.

Question 38

I am using BeautifulSoup and parsing some HTMLs.

I’m getting a certain data from each HTML (using for loop) and adding that data to a certain list.

The problem is, some of the HTMLs have different format (and they don’t have the data that I want in them).

So, I was trying to use exception handling and add value null to the list (I should do this since the sequence of data is important.)

For instance, I have a code like:

soup = BeautifulSoup(links)
dlist = soup.findAll('dd', 'title')
# I'm trying to find content between <dd class='title'> and </dd>
gotdata = dlist[1]
# and what i want is the 2nd content of those
newlist.append(gotdata)
# and I add that to a newlist

and some of the links don’t have any <dd class='title'>, so what I want to do is add string null to the list instead.

The error appears:

list index out of range.

What I have done tried is to add some lines like this:

if not dlist[1]:  
   newlist.append('null')
   continue

But it doesn’t work out. It still shows error:

list index out of range.

What should I do about this? Should I use exception handling? or is there any easier way?

Any suggestions? Any help would be really great!

Question 39

Handling the exception is the way to go:

try:
    gotdata = dlist[1]
except IndexError:
    gotdata = 'null'

Of course you could also check the len() of dlist; but handling the exception is more intuitive.

Question 40

You have two options; either handle the exception or test the length:

if len(dlist) > 1:
    newlist.append(dlist[1])
    continue

or

try:
    newlist.append(dlist[1])
except IndexError:
    pass
continue

Use the first if there often is no second item, the second if there sometimes is no second item.

Question 41

A ternary will suffice. change:

gotdata = dlist[1]

to

gotdata = dlist[1] if len(dlist) > 1 else 'null'

this is a shorter way of expressing

if len(dlist) > 1:
    gotdata = dlist[1]
else: 
    gotdata = 'null'

Question 42

Taking reference of ThiefMaster♦ sometimes we get an error with value given as ‘\n’ or null and perform for that required to handle ValueError:

Handling the exception is the way to go

try:
    gotdata = dlist[1]
except (IndexError, ValueError):
    gotdata = 'null'

Question 43

for i in range (1, len(list))
    try:
        print (list[i])

    except ValueError:
        print("Error Value.")
    except indexError:
        print("Erorr index")
    except :
        print('error ')

Question 44

For anyone interested in a shorter way:

gotdata = len(dlist)>1 and dlist[1] or 'null'

But for best performance, I suggest using False instead of 'null', then a one line test will suffice:

gotdata = len(dlist)>1 and dlist[1]

Question 45

I would like do something like that.

list_of_urls = ['http://www.google.fr/', 'http://www.google.fr/', 
                'http://www.google.cn/', 'http://www.google.com/', 
                'http://www.google.fr/', 'http://www.google.fr/', 
                'http://www.google.fr/', 'http://www.google.com/', 
                'http://www.google.fr/', 'http://www.google.com/', 
                'http://www.google.cn/']

urls = [{'url': 'http://www.google.fr/', 'nbr': 1}]

for url in list_of_urls:
    if url in [f['url'] for f in urls]:
         urls[??]['nbr'] += 1
    else:
         urls.append({'url': url, 'nbr': 1})

How can I do ? I don’t know if I should take the tuple to edit it or figure out the tuple indices?

Any help ?

Question 46

That is a very strange way to organize things. If you stored in a dictionary, this is easy:

# This example should work in any version of Python.
# urls_d will contain URL keys, with counts as values, like: {'http://www.google.fr/' : 1 }
urls_d = {}
for url in list_of_urls:
    if not url in urls_d:
        urls_d[url] = 1
    else:
        urls_d[url] += 1

This code for updating a dictionary of counts is a common “pattern” in Python. It is so common that there is a special data structure, defaultdict, created just to make this even easier:

from collections import defaultdict  # available in Python 2.5 and newer

urls_d = defaultdict(int)
for url in list_of_urls:
    urls_d[url] += 1

If you access the defaultdict using a key, and the key is not already in the defaultdict, the key is automatically added with a default value. The defaultdict takes the callable you passed in, and calls it to get the default value. In this case, we passed in class int; when Python calls int() it returns a zero value. So, the first time you reference a URL, its count is initialized to zero, and then you add one to the count.

But a dictionary full of counts is also a common pattern, so Python provides a ready-to-use class: containers.Counter You just create a Counter instance by calling the class, passing in any iterable; it builds a dictionary where the keys are values from the iterable, and the values are counts of how many times the key appeared in the iterable. The above example then becomes:

from collections import Counter  # available in Python 2.7 and newer

urls_d = Counter(list_of_urls)

If you really need to do it the way you showed, the easiest and fastest way would be to use any one of these three examples, and then build the one you need.

from collections import defaultdict  # available in Python 2.5 and newer

urls_d = defaultdict(int)
for url in list_of_urls:
    urls_d[url] += 1

urls = [{"url": key, "nbr": value} for key, value in urls_d.items()]

If you are using Python 2.7 or newer you can do it in a one-liner:

from collections import Counter

urls = [{"url": key, "nbr": value} for key, value in Counter(list_of_urls).items()]

Question 47

Using the default works, but so does:

urls[url] = urls.get(url, 0) + 1

using .get, you can get a default return if it doesn’t exist. By default it’s None, but in the case I sent you, it would be 0.

Question 48

Use defaultdict:

from collections import defaultdict

urls = defaultdict(int)

for url in list_of_urls:
    urls[url] += 1

Question 49

This always works fine for me:

for url in list_of_urls:
    urls.setdefault(url, 0)
    urls[url] += 1

Question 50

To do it exactly your way? You could use the for…else structure

for url in list_of_urls:
    for url_dict in urls:
        if url_dict['url'] == url:
            url_dict['nbr'] += 1
            break
    else:
        urls.append(dict(url=url, nbr=1))

But it is quite inelegant. Do you really have to store the visited urls as a LIST? If you sort it as a dict, indexed by url string, for example, it would be way cleaner:

urls = {'http://www.google.fr/': dict(url='http://www.google.fr/', nbr=1)}

for url in list_of_urls:
    if url in urls:
        urls[url]['nbr'] += 1
    else:
        urls[url] = dict(url=url, nbr=1)

A few things to note in that second example:

see how using a dict for urls removes the need for going through the whole urls list when testing for one single url. This approach will be faster.
Using dict( ) instead of braces makes your code shorter
using list_of_urls, urls and url as variable names make the code quite hard to parse. It’s better to find something clearer, such as urls_to_visit, urls_already_visited and current_url. I know, it’s longer. But it’s clearer.

And of course I’m assuming that dict(url='http://www.google.fr', nbr=1) is a simplification of your own data structure, because otherwise, urls could simply be:

urls = {'http://www.google.fr':1}

for url in list_of_urls:
    if url in urls:
        urls[url] += 1
    else:
        urls[url] = 1

Which can get very elegant with the defaultdict stance:

urls = collections.defaultdict(int)
for url in list_of_urls:
    urls[url] += 1

Question 51

Except for the first time, each time a word is seen the if statement’s test fails. If you are counting a large number of words, many will probably occur multiple times. In a situation where the initialization of a value is only going to occur once and the augmentation of that value will occur many times it is cheaper to use a try statement:

urls_d = {}
for url in list_of_urls:
    try:
        urls_d[url] += 1
    except KeyError:
        urls_d[url] = 1

you can read more about this: https://wiki.python.org/moin/PythonSpeed/PerformanceTips

Question 52

A tuple takes less memory space in Python:

>>> a = (1,2,3)
>>> a.__sizeof__()
48

whereas lists takes more memory space:

>>> b = [1,2,3]
>>> b.__sizeof__()
64

What happens internally on the Python memory management?

Question 53

I assume you’re using CPython and with 64bits (I got the same results on my CPython 2.7 64-bit). There could be differences in other Python implementations or if you have a 32bit Python.

Regardless of the implementation, lists are variable-sized while tuples are fixed-size.

So tuples can store the elements directly inside the struct, lists on the other hand need a layer of indirection (it stores a pointer to the elements). This layer of indirection is a pointer, on 64bit systems that’s 64bit, hence 8bytes.

But there’s another thing that lists do: They over-allocate. Otherwise list.append would be an O(n) operation always – to make it amortized O(1) (much faster!!!) it over-allocates. But now it has to keep track of the allocated size and the filled size (tuples only need to store one size, because allocated and filled size are always identical). That means each list has to store another “size” which on 64bit systems is a 64bit integer, again 8 bytes.

So lists need at least 16 bytes more memory than tuples. Why did I say “at least”? Because of the over-allocation. Over-allocation means it allocates more space than needed. However, the amount of over-allocation depends on “how” you create the list and the append/deletion history:

>>> l = [1,2,3]
>>> l.__sizeof__()
64
>>> l.append(4)  # triggers re-allocation (with over-allocation), because the original list is full
>>> l.__sizeof__()
96

>>> l = []
>>> l.__sizeof__()
40
>>> l.append(1)  # re-allocation with over-allocation
>>> l.__sizeof__()
72
>>> l.append(2)  # no re-alloc
>>> l.append(3)  # no re-alloc
>>> l.__sizeof__()
72
>>> l.append(4)  # still has room, so no over-allocation needed (yet)
>>> l.__sizeof__()
72

Images

I decided to create some images to accompany the explanation above. Maybe these are helpful

This is how it (schematically) is stored in memory in your example. I highlighted the differences with red (free-hand) cycles:

That’s actually just an approximation because int objects are also Python objects and CPython even reuses small integers, so a probably more accurate representation (although not as readable) of the objects in memory would be:

Useful links:

Note that __sizeof__ doesn’t really return the “correct” size! It only returns the size of the stored values. However when you use sys.getsizeof the result is different:

>>> import sys
>>> l = [1,2,3]
>>> t = (1, 2, 3)
>>> sys.getsizeof(l)
88
>>> sys.getsizeof(t)
72

There are 24 “extra” bytes. These are real, that’s the garbage collector overhead that isn’t accounted for in the __sizeof__ method. That’s because you’re generally not supposed to use magic methods directly – use the functions that know how to handle them, in this case: sys.getsizeof (which actually adds the GC overhead to the value returned from __sizeof__).

Question 54

I’ll take a deeper dive into the CPython codebase so we can see how the sizes are actually calculated. In your specific example, no over-allocations have been performed, so I won’t touch on that.

I’m going to use 64-bit values here, as you are.

The size for lists is calculated from the following function, list_sizeof:

static PyObject *
list_sizeof(PyListObject *self)
{
    Py_ssize_t res;

    res = _PyObject_SIZE(Py_TYPE(self)) + self->allocated * sizeof(void*);
    return PyInt_FromSsize_t(res);
}

Here Py_TYPE(self) is a macro that grabs the ob_type of self (returning PyList_Type) while _PyObject_SIZE is another macro that grabs tp_basicsize from that type. tp_basicsize is calculated as sizeof(PyListObject) where PyListObject is the instance struct.

The PyListObject structure has three fields:

PyObject_VAR_HEAD     # 24 bytes 
PyObject **ob_item;   #  8 bytes
Py_ssize_t allocated; #  8 bytes

these have comments (which I trimmed) explaining what they are, follow the link above to read them. PyObject_VAR_HEAD expands into three 8 byte fields (ob_refcount, ob_type and ob_size) so a 24 byte contribution.

So for now res is:

sizeof(PyListObject) + self->allocated * sizeof(void*)

or:

40 + self->allocated * sizeof(void*)

If the list instance has elements that are allocated. the second part calculates their contribution. self->allocated, as it’s name implies, holds the number of allocated elements.

Without any elements, the size of lists is calculated to be:

>>> [].__sizeof__()
40

i.e the size of the instance struct.

tuple objects don’t define a tuple_sizeof function. Instead, they use object_sizeof to calculate their size:

static PyObject *
object_sizeof(PyObject *self, PyObject *args)
{
    Py_ssize_t res, isize;

    res = 0;
    isize = self->ob_type->tp_itemsize;
    if (isize > 0)
        res = Py_SIZE(self) * isize;
    res += self->ob_type->tp_basicsize;

    return PyInt_FromSsize_t(res);
}

This, as for lists, grabs the tp_basicsize and, if the object has a non-zero tp_itemsize (meaning it has variable-length instances), it multiplies the number of items in the tuple (which it gets via Py_SIZE) with tp_itemsize.

tp_basicsize again uses sizeof(PyTupleObject) where the PyTupleObject struct contains:

PyObject_VAR_HEAD       # 24 bytes 
PyObject *ob_item[1];   # 8  bytes

So, without any elements (that is, Py_SIZE returns 0) the size of empty tuples is equal to sizeof(PyTupleObject):

>>> ().__sizeof__()
24

huh? Well, here’s an oddity which I haven’t found an explanation for, the tp_basicsize of tuples is actually calculated as follows:

sizeof(PyTupleObject) - sizeof(PyObject *)

why an additional 8 bytes is removed from tp_basicsize is something I haven’t been able to find out. (See MSeifert’s comment for a possible explanation)

But, this is basically the difference in your specific example. lists also keep around a number of allocated elements which helps determine when to over-allocate again.

Now, when additional elements are added, lists do indeed perform this over-allocation in order to achieve O(1) appends. This results in greater sizes as MSeifert’s covers nicely in his answer.

Question 55

MSeifert answer covers it broadly; to keep it simple you can think of:

tuple is immutable. Once it set, you can’t change it. So you know in advance how much memory you need to allocate for that object.

list is mutable. You can add or remove items to or from it. It has to know the size of it (for internal impl.). It resizes as needed.

There are no free meals – these capabilities comes with a cost. Hence the overhead in memory for lists.

Question 56

The size of the tuple is prefixed, meaning at tuple initialization the interpreter allocate enough space for the contained data, and that’s the end of it, giving it’s immutable (can’t be modified), whereas a list is a mutable object hence implying dynamic allocation of memory, so to avoid allocating space each time you append or modify the list ( allocate enough space to contain the changed data and copy the data to it), it allocates additional space for future append, modifications, … that pretty much sums it up.

Question 57

Suppose I have a list of tuples and I want to convert to multiple lists.

For example, the list of tuples is

[(1,2),(3,4),(5,6),]

Is there any built-in function in Python that convert it to:

[1,3,5],[2,4,6]

This can be a simple program. But I am just curious about the existence of such built-in function in Python.

Question 58

The built-in function zip() will almost do what you want:

>>> zip(*[(1, 2), (3, 4), (5, 6)])
[(1, 3, 5), (2, 4, 6)]

The only difference is that you get tuples instead of lists. You can convert them to lists using

map(list, zip(*[(1, 2), (3, 4), (5, 6)]))

Question 59

From the python docs:

zip() in conjunction with the * operator can be used to unzip a list:

Specific example:

>>> zip((1,3,5),(2,4,6))
[(1, 2), (3, 4), (5, 6)]
>>> zip(*[(1, 2), (3, 4), (5, 6)])
[(1, 3, 5), (2, 4, 6)]

Or, if you really want lists:

>>> map(list, zip(*[(1, 2), (3, 4), (5, 6)]))
[[1, 3, 5], [2, 4, 6]]

Question 60

Use:

a = [(1,2),(3,4),(5,6),]    
b = zip(*a)
>>> [(1, 3, 5), (2, 4, 6)]

Question 61

franklsf95 goes for performance in his answer and opts for list.append(), but they are not optimal.

Adding list comprehensions, I ended up with the following:

def t1(zs):
    xs, ys = zip(*zs)
    return xs, ys

def t2(zs):
    xs, ys = [], []
    for x, y in zs:
        xs.append(x)
        ys.append(y)
    return xs, ys

def t3(zs):
    xs, ys = [x for x, y in zs], [y for x, y in zs]
    return xs, ys

if __name__ == '__main__':
    from timeit import timeit
    setup_string='''\
N = 2000000
xs = list(range(1, N))
ys = list(range(N+1, N*2))
zs = list(zip(xs, ys))
from __main__ import t1, t2, t3
'''
    print(f'zip:\t\t{timeit('t1(zs)', setup=setup_string, number=1000)}')
    print(f'append:\t\t{timeit('t2(zs)', setup=setup_string, number=1000)}')
    print(f'list comp:\t{timeit('t3(zs)', setup=setup_string, number=1000)}')

This gave the result:

zip:            122.11585397789766
append:         356.44876132614047
list comp:      144.637765085659

So if you are after performance, you should probably use zip() although list comprehensions are not too far behind. The performance of append is actually pretty poor in comparison.

Question 62

Despite *zip being more Pythonic, the following code has much better performance:

xs, ys = [], []
for x, y in zs:
    xs.append(x)
    ys.append(y)

Also, when the original list zs is empty, *zip will raise, but this code can properly handle.

I just ran a quick experiment, and here is the result:

Using *zip:     1.54701614s
Using append:   0.52687597s

Running it multiple times, append is 3x – 4x faster than zip! The test script is here:

#!/usr/bin/env python3
import time

N = 2000000
xs = list(range(1, N))
ys = list(range(N+1, N*2))
zs = list(zip(xs, ys))

t1 = time.time()

xs_, ys_ = zip(*zs)
print(len(xs_), len(ys_))

t2 = time.time()

xs_, ys_ = [], []
for x, y in zs:
    xs_.append(x)
    ys_.append(y)
print(len(xs_), len(ys_))

t3 = time.time()

print('Using *zip:\t{:.8f}s'.format(t2 - t1))
print('Using append:\t{:.8f}s'.format(t3 - t2))

My Python Version:

Python 3.6.3 (default, Oct 24 2017, 12:18:40)
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

Question 63

In addition to Claudiu’s answer, you can use:

>>>a, b = map(list, zip(*[(1, 2), (3, 4), (5, 6)]))
>>>a
[1,3,5]
>>>b
[2,4,6]

Edited according to @Peyman mohseni kiasari

Question 64

Adding to Claudiu’s and Claudiu’s answer and since map needs to be imported from itertools in python 3, you also use a list comprehension like:

[[*x] for x in zip(*[(1,2),(3,4),(5,6)])]
>>> [[1, 3, 5], [2, 4, 6]]

问题：列表到数组的转换以使用ravel（）函数

回答 0

回答 1

创建一个int数组和一个列表

create an int array and a list

回答 2

回答 3

回答 4

回答 5

问题：Python：如何打印范围az？

回答 0

回答 1

回答 2

回答 3

回答 4

获取具有所需值的列表

应用您对列表的了解

Get a list with the desired values

Apply what you know about lists

回答 5

回答 6

回答 7

回答 8

回答 9

回答 10

回答 11

回答 12

回答 13

回答 14

回答 15

另一种方式

结果

这与其他回复有何不同

Another way to do it

Result

How this differs from the other replies

问题：来自os.listdir（）的非字母数字列表顺序

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

回答 7

回答 8

回答 9

回答 10

回答 11

问题：我想exceptions处理“列表索引超出范围”。

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

问题：Python：字典列表，如果存在，则增加一个字典值，如果不增加新字典

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

问题：为什么元组在内存中的空间比列表少？

回答 0

图片

Images

回答 1

回答 2

回答 3

问题：如何将元组列表转换为多个列表？

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

问题：Python Pandas将列表插入单元格

编辑

使用`at`/ 设置单元格值`iat`

如果得到了`ValueError: setting an array element with a sequence`怎么办？

Setting Cell Values with `at`/`iat`

What if I get `ValueError: setting an array element with a sequence`?